kubeadm init 在初始化 Kubernetes 集群时失败 [关闭]

【问题标题】：kubeadm init failing while initializing a Kubernetes cluster [closed]kubeadm init 在初始化 Kubernetes 集群时失败 [关闭]
【发布时间】：2021-08-08 21:56:47
【问题描述】：

来自 kubeadm init 的执行日志

  • VM 操作系统版本 - RHEL 8.2 (4.18.0-193.el8.x86_64)
  • Docker 版本 - 19.03.8
  • Kubernetes 版本 - 1.18.0
  • 硬件配置 - 4 个 CPU 和 16 GB RAM（90% 可用）
$$kubeadm init --kubernetes-version="v1.18.0" --pod-network-cidr="10.244.0.0/16"

W0519 21:08:48.180818  913499 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[init] Using Kubernetes version: v1.18.0
[preflight] Running pre-flight checks
        [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
        [WARNING FileExisting-socat]: socat not found in system path
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [host422 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 10.180.40.75]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [host422 localhost] and IPs [10.180.40.75 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [host422 localhost] and IPs [10.180.40.75 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
W0519 21:08:50.681218  913499 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[control-plane] Creating static Pod manifest for "kube-scheduler"
W0519 21:08:50.681948  913499 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
        Unfortunately, an error has occurred:
                timed out waiting for the condition

        This error is likely caused by:
                - The kubelet is not running
                - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

        If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
                - 'systemctl status kubelet'
                - 'journalctl -xeu kubelet'

        Additionally, a control plane component may have crashed or exited when started by the container runtime.
        To troubleshoot, list all containers using your preferred container runtimes CLI.

        Here is one example how you may list all Kubernetes containers running in docker:
                - 'docker ps -a | grep kube | grep -v pause'
                Once you have found the failing container, you can inspect its logs with:
                - 'docker logs CONTAINERID'

error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster

kubelet 状态（可以看到 Setting node annotation to enable volume controller attach/detach 不断打印在日志中）

 kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: active (running) since Wed 2021-05-19 21:08:48 IST; 17min ago
     Docs: https://kubernetes.io/docs/
 Main PID: 913672 (kubelet)
    Tasks: 18 (limit: 101228)
   Memory: 33.7M
   CGroup: /system.slice/kubelet.service
           └─913672 /home0/kubernetes/kubernetes/server/bin/kubelet --root-dir=/home0/kubernetes/workdir

May 19 21:24:49 InBlrbnc422 kubelet[913672]: I0519 21:24:49.379623  913672 kubelet_node_status.go:294] Setting node annotation to enable volume controller attach/detach
May 19 21:24:59 InBlrbnc422 kubelet[913672]: I0519 21:24:59.425035  913672 kubelet_node_status.go:294] Setting node annotation to enable volume controller attach/detach

(1) 执行swapoff -a
(2) 尝试将 docker 和 kubelet 的 cgroup 驱动程序更新为 systemd，但不知何故 kubelet 没有接受更改。希望 kubeadm init 应该能够使用驱动程序版本 cgroupfs 运行。
请让我知道我还需要检查什么。

更新堆栈

couldn't initialize a Kubernetes cluster
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/init.runWaitControlPlanePhase
        /workspace/anago-v1.18.0-rc.1.21+8be33caaf953ac/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/init/waitcontrolplane.go:114
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run.func1
        /workspace/anago-v1.18.0-rc.1.21+8be33caaf953ac/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:234
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).visitAll
        /workspace/anago-v1.18.0-rc.1.21+8be33caaf953ac/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:422
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run
        /workspace/anago-v1.18.0-rc.1.21+8be33caaf953ac/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:207
k8s.io/kubernetes/cmd/kubeadm/app/cmd.NewCmdInit.func1
        /workspace/anago-v1.18.0-rc.1.21+8be33caaf953ac/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/init.go:147
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).execute
        /workspace/anago-v1.18.0-rc.1.21+8be33caaf953ac/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:826
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).ExecuteC
        /workspace/anago-v1.18.0-rc.1.21+8be33caaf953ac/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:914
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).Execute
        /workspace/anago-v1.18.0-rc.1.21+8be33caaf953ac/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:864
k8s.io/kubernetes/cmd/kubeadm/app.Run
        /workspace/anago-v1.18.0-rc.1.21+8be33caaf953ac/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/kubeadm.go:50
main.main
        _output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/kubeadm.go:25
runtime.main
        /usr/local/go/src/runtime/proc.go:203
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1357
error execution phase wait-control-plane
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run.func1
        /workspace/anago-v1.18.0-rc.1.21+8be33caaf953ac/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:235
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).visitAll
        /workspace/anago-v1.18.0-rc.1.21+8be33caaf953ac/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:422
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run
        /workspace/anago-v1.18.0-rc.1.21+8be33caaf953ac/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:207
k8s.io/kubernetes/cmd/kubeadm/app/cmd.NewCmdInit.func1
        /workspace/anago-v1.18.0-rc.1.21+8be33caaf953ac/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/init.go:147
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).execute
        /workspace/anago-v1.18.0-rc.1.21+8be33caaf953ac/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:826
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).ExecuteC
        /workspace/anago-v1.18.0-rc.1.21+8be33caaf953ac/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:914
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).Execute
        /workspace/anago-v1.18.0-rc.1.21+8be33caaf953ac/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:864
k8s.io/kubernetes/cmd/kubeadm/app.Run
        /workspace/anago-v1.18.0-rc.1.21+8be33caaf953ac/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/kubeadm.go:50
main.main
        _output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/kubeadm.go:25
runtime.main
        /usr/local/go/src/runtime/proc.go:203
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1357

【问题讨论】：

    标签： kubernetes kubeadm kubelet


    【解决方案1】：

    这个问题通常是由于您的虚拟机或程序包配置错误造成的。尝试执行这些步骤，它应该适合你（所有命令都需要以 root 身份运行）：

    首先，通过运行 reset 命令重置您的 kubeadm 集群并刷新您的 iptables（以避免任何网络问题）：

    kubeadm reset -f
iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X

    其次，你需要将你的 Docker cgroup 驱动更改为 systemd（默认为 kubernetes kubelet 推荐 CRI conf）然后重启 docker 服务：

    cat > /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}
EOF

systemctl daemon-reload
systemctl restart docker

    最后你需要关闭 swapoff 并重启并启用 kubelet 服务

    swapoff -a
systemctl start kubelet

    我确实使用完全相同的软件包安装了 kubernetes，但我使用的是 Kubernetes v1.21.0，它对我来说很好，如果不适合你，你应该升级那个版本。

    【讨论】：

    • kubeadm reset , swap -a and iptables were already flushed 并尝试过但没有成功。同样正如我在关于 cgroupdriver 更改的原始帖子中所解释的那样，docker 更改正在得到反映，但不知何故 kubelet(KUBELET_CGROUP_ARGS=--cgroup-driver=systemd under 10-kubeadm.conf) 没有将 systemd 作为驱动程序并始终显示cgroupfs 作为默认驱动程序。
    • cgroupDriver 需要在 kubelet 配置文件中更改 /var/lib/kubelet/config.yaml 链接：kubernetes.io/docs/tasks/administer-cluster/kubeadm/…。但是我认为这不是阻塞问题！验证 kubelet 状态是否为自动重启并启用
    • 现在 cgroupDriver 上的 WARNING 已解决（确实在 ExecStart 下的 kubelet.service 中传递了相同的内容）。剩下的警告是[WARNING FileExisting-socat]: socat not found in system path。使用 kubeadm init 期间收到的跟踪更新原始帖子。任何指针将不胜感激。
    • 其实“socat”是sockets网络的kubeadm依赖，所以它必须与kubeadm一起安装，如果不存在，验证它是否存在于你的安装包列表中（sudo yum list installed | grep socat）存在它太奇怪了，在这种情况下，我建议将 kubeadm 升级到 v1.21.0（如果你真的需要这个版本的 kubeadm，你可以单独安装它：sudo yum install socat-1.7.3.3-2.el8）然后重新启动一切（reset kubeadm stop kubelet run dockersystem prune delete all k8s conf 和 cni conf flush iptables 并闭上你的眼睛并点击繁荣命令“kubeadm init”;）
    • 这是一个比较指南（与您的发行版本相同）：redaix.com/2020/06/how-to-install-and-configure-kubernetes.html
    猜你喜欢
    • 1970-01-01
    • 2018-09-17
    • 2019-08-27
    • 2020-03-16
    • 2020-11-03
    • 2018-09-17
    • 2019-02-02
    • 2019-05-19
    • 2019-05-23
    相关资源
    最近更新 更多
    热门标签
    Java Python linux javascript Mysql C# Docker 算法 前端 SpringBoot Redis Vue spring 设计模式 .net core .net kubernetes c++ 数据库 数据结构 大数据 js 机器学习 微服务 Android Go 程序员 面试 JVM ASP.net core 云原生 人工智能 后端 PHP git CSS golang k8s Nginx Django mybatis 深度学习 多线程 React 架构 devops 爬虫 云计算 Spring Boot LeetCode