Suse Linuxenterprize server 15虚拟机如何安装kubernetes？

Question

我们正在尝试在 SUSE enterprize linux server v15 中安装 kubernetes。我们发现没有办法使用 kubeadm 安装 k8s。 SUSE 提供容器即服务平台（CaasP）来安装 k8s。

我们所拥有的只是很少的虚拟机和 suse 订阅。我们可以在其中安装 CaasP 吗？ 我们找不到任何文档可以在虚拟机中安装它。

是否有任何文档可以在虚拟机中逐步安装 CaasP？

Answer 1

SLES 上的 Kubeadm

可以使用kubeadm 在 SUSE Linux Enterprise Server 15 上安装 Kubernetes。
您可以在下面找到分步示例。

该示例在以下云 VM 映像上进行了测试：

GCP：

• SUSE Linux Enterprise Server 15 SP1 x86_x64

AWS：

• openSUSE-Leap-15.2-v20200710-HVM-x86_64-548f7b74-f1d6-437e-b650-f6315f6d8aa3-ami-0f5745b812a5b7654.4 - ami-023643495f15f104b
• suse-sles-15-sp1-v20200615-hvm-ssd-x86_64 - ami-0044ae6906d786f4b

天蓝色：

• SUSE Enterprise Linux 15 SP1 +补丁

因此，它很有可能与其他图像一起使用，只需进行一些更改。

它还在 Vagrant 框 trueability/sles-15-sp1 上进行了测试，由于订阅密钥已过期，因此需要执行一些额外的步骤。我使用了OSS repositories 并忽略了过期错误：

# add OSS repository for software installation 
$ zypper addrepo http://download.opensuse.org/distribution/leap/15.2/repo/oss/ public

# add repository for installing newer Docker version

$ zypper addrepo https://download.opensuse.org/repositories/Virtualization:containers/openSUSE_Leap_15.0/Virtualization:containers.repo virt

# install symbols required by Docker:

$ zypper install libseccomp-devel

# turn off all swap partitions. Comment appropriate /etc/fstab entry as well.

$ swapoff -a

# Rest of the steps is similar except additional argument during cluster initialization. 

# This box is using btrfs for /var/lib/docker and kubeadm complains about it. 
# I've just asked kubeadm to ignore that fact. 
# Even with btrfs it can start and run pods, but there might be some problems with Persistent Volumes usage, 
# so consider using additional xfs or ext4 partition for /var/lib/docker

$ kubeadm init --pod-network-cidr=10.244.0.0/16 --ignore-preflight-errors=all

云虚拟机：

Cloud SLES 15 SP1 映像将 xfs 用于其 / 文件系统，并且不使用开箱即用的交换，并且 kubeadm 无错误地通过所有飞行前检查。

# become root

$ sudo -s

# install docker

$ zypper refresh
$ zypper install docker

# configure sysctl for Kubernetes

$ cat <<EOF >> /etc/sysctl.conf
net.ipv4.ip_forward=1
net.ipv4.conf.all.forwarding=1
net.bridge.bridge-nf-call-iptables=1
EOF

# add Google repository for installing Kubernetes packages
#$ zypper addrepo --type yum --gpgcheck-strict --refresh https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64 google-k8s

#or

$ cat <<EOF > /etc/zypp/repos.d/google-k8s.repo
[google-k8s]
name=google-k8s
enabled=1
autorefresh=1
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
type=rpm-md
gpgcheck=1
repo_gpgcheck=1
pkg_gpgcheck=1
EOF

# import Google repository keys

$ rpm --import https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
$ rpm --import https://packages.cloud.google.com/yum/doc/yum-key.gpg
$ rpm -q gpg-pubkey --qf '%{name}-%{version}-%{release} --> %{summary}\n'

# the following repository was needed only for GCP image
# other images was able successfully install conntrack-tools using existing repository

$ zypper addrepo https://download.opensuse.org/repositories/security:netfilter/SLE_12/security:netfilter.repo conntrack
$ zypper refresh conntrack

# conntrack presence is checked during kubeadm pre-flight checks 
# but zypper unable to find appropriate dependency for kubelet, 
# so let's install it manually

$ zypper install conntrack-tools

# refresh Google repository cache and check if we see several versions of Kubernetes packages to choose from

$ zypper refresh google-k8s
$ zypper packages --repo google-k8s

# install latest available kubelet package
# ignore conntrack dependency and install kubelet (Solution 2 in my case)

$ zypper install kubelet

# install kubeadm package. kubectl and cri-tools are installed as kubeadm dependency

$ zypper install kubeadm

# force docker to use systemd cgroup driver and overlay2 storage driver. 
# Check the links in the end of the answer for details. 
# BTW, kubelet would work even with default content of the file.

$ cat > /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}
EOF

# Not sure if it's necessary it was taken from the Kubernetes documentation

$ mkdir -p /etc/systemd/system/docker.service.d

# lets start and enable docker and kubelet services

$ systemctl start docker.service
$ systemctl enable docker.service
$ systemctl enable kubelet.service

# apply configured earlier sysctl settings. 
# net.bridge.bridge-nf-call-iptables becomes available after successfully starting
# Docker service 

$ sysctl -p

# Now it's time to initialize Kubernetes master node. 
# Ignore pre-flight checks for Vagrant box.

$ kubeadm init --pod-network-cidr=10.244.0.0/16

# prepare kubectl configuration to connect the cluster

$  mkdir -p $HOME/.kube
$  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
$  sudo chown $(id -u):$(id -g) $HOME/.kube/config

# Check if api-server responds to our requests. 
# At this moment it's fine to see master node in NotReady state.

$ kubectl get nodes

# Deploy Flannel network addon

$ kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

# remove taint from the master node. 
# It allows master node to run application pods. 
# At least one worker node is required if this step is skipped.

$ kubectl taint nodes --all node-role.kubernetes.io/master-

# run test pod to check if everything works fine

$ kubectl run nginx1 --image=nginx

# after some time... ~ 3-5 minutes

# check the pods' state 

$ kubectl get pods -A -o wide
NAMESPACE     NAME                                READY   STATUS    RESTARTS   AGE     IP           NODE        NOMINATED NODE   READINESS GATES
default       nginx1                              1/1     Running   0          74s     10.244.0.4   suse-test   <none>           <none>
kube-system   coredns-66bff467f8-vc2x4            1/1     Running   0          2m26s   10.244.0.2   suse-test   <none>           <none>
kube-system   coredns-66bff467f8-w4jvq            1/1     Running   0          2m26s   10.244.0.3   suse-test   <none>           <none>
kube-system   etcd-suse-test                      1/1     Running   0          2m41s   10.4.0.4     suse-test   <none>           <none>
kube-system   kube-apiserver-suse-test            1/1     Running   0          2m41s   10.4.0.4     suse-test   <none>           <none>
kube-system   kube-controller-manager-suse-test   1/1     Running   0          2m41s   10.4.0.4     suse-test   <none>           <none>
kube-system   kube-flannel-ds-amd64-mbfxp         1/1     Running   0          2m12s   10.4.0.4     suse-test   <none>           <none>
kube-system   kube-proxy-cw5xm                    1/1     Running   0          2m26s   10.4.0.4     suse-test   <none>           <none>
kube-system   kube-scheduler-suse-test            1/1     Running   0          2m41s   10.4.0.4     suse-test   <none>           <none>

# check if the test pod is working fine

# curl 10.244.0.4
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
...skipped...

# basic Kubernetes installation is done

附加材料：

• Container runtimes（Kubernetes 文档页面）

• Giving error saying “unsupported graph driver: btrfs” in SLES when try to Kubeadm init
  （如果您有btrfs / 分区，您可以为/var/lib/docker 挂载额外的xfs 或ext4 分区以使用overlay2 Docker 存储驱动程序）

• OverlayFS support for SLES 12（我希望 SLES 15 也支持它。）

• Docker Compatibility Matrix

    OS Distribution (x86_64): SLES 15
    Enterprise Engine: 19.03.x
    UCP: 3.2.x
    DTR: 2.7.x
    Storage Driver: overlay2,btrfs
    Orchestration: Swarm mode, Kubernetes
    DTR Storage Backend: NFSv4, NFSv3, Amazon S3, S3 Compliant Alternatives, 
                Azure Storage (Blob), Google Cloud Storage, OpenStack Swift, 
                Local Filesystem
  

• SUSE: Docker Open Source Engine Guide（非常有用的书）

• Use the OverlayFS storage driver

• Install Linux Kernel 4.12 in openSUSE（如果您想将 AUFS 支持添加到 Linux 内核）

关于 SUSE CaaSP 的资料

• SUSE CaaS Platform Setup - Nothing to Everything in 1 Hour （视频很老但是很有用）
• CaaSP download page, 60 days free trial