kubernetes 中的 CrashLoopBackoff 错误中的 kube-controller-manager 和 kube-scheduler

【问题标题】：kube-controller-manager & kube-scheduler in CrashLoopBackoff Error in kuberneteskubernetes 中的 CrashLoopBackoff 错误中的 kube-controller-manager 和 kube-scheduler
【发布时间】：2019-09-30 12:40:36
【问题描述】：

我在我的 k8s 中使用 calico 作为 CNI，我正在尝试在 3 个服务器中部署一个主集群。我用的是kubeadm，关注官方setup guide。但是出现了一些错误，kube-controller-manager 和kube-scheduler 进入 CrashLoopBackOff 错误，无法正常运行。

我在每台服务器上都试过kubeadm reset，还重启了服务器，降级了docker。

我使用kubeadm init --apiserver-advertise-address=192.168.213.128 --pod-network-cidr=192.168.0.0/16来初始化master，然后运行kubectl apply -f https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/rbac-kdd.yaml和kubectl apply -f https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml来启动calico。

[root@k8s-master ~]# docker info
Containers: 20
 Running: 18
 Paused: 0
 Stopped: 2
Images: 10
Server Version: 18.09.6
Storage Driver: overlay2
 Backing Filesystem: xfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: systemd
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: bb71b10fd8f58240ca47fbb579b9d1028eea7c84
runc version: 2b18fe1d885ee5083ef9f0838fee39b62d653e30
init version: fec3683
Security Options:
 seccomp
  Profile: default
Kernel Version: 3.10.0-957.12.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 972.6MiB
Name: k8s-master
ID: RN6I:PP52:4WTU:UP7E:T3LF:MXVZ:EDBX:RSII:BIRW:36O2:CYJ3:FRV2
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Registry Mirrors:
 https://i70c3eqq.mirror.aliyuncs.com/
 https://docker.mirrors.ustc.edu.cn/
Live Restore Enabled: false
Product License: Community Engine

[root@k8s-master ~]# kubectl version
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.1", GitCommit:"b7394102d6ef778017f2ca4046abbaa23b88c290", GitTreeState:"clean", BuildDate:"2019-04-08T17:11:31Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.1", GitCommit:"b7394102d6ef778017f2ca4046abbaa23b88c290", GitTreeState:"clean", BuildDate:"2019-04-08T17:02:58Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}
[root@k8s-master ~]# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.1", GitCommit:"b7394102d6ef778017f2ca4046abbaa23b88c290", GitTreeState:"clean", BuildDate:"2019-04-08T17:08:49Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}
[root@k8s-master ~]# kubelet --version
Kubernetes v1.14.1

[root@k8s-master ~]# kubectl get no -A
NAME         STATUS   ROLES    AGE   VERSION
k8s-master   Ready    master   49m   v1.14.1

[root@k8s-master ~]# kubectl get pods -A
NAMESPACE     NAME                                 READY   STATUS             RESTARTS   AGE
kube-system   calico-node-xmc5t                    2/2     Running            0          27m
kube-system   coredns-6765558d84-945mt             1/1     Running            0          28m
kube-system   coredns-6765558d84-xz7lw             1/1     Running            0          28m
kube-system   coredns-fb8b8dccf-z87sl              1/1     Running            0          31m
kube-system   etcd-k8s-master                      1/1     Running            0          30m
kube-system   kube-apiserver-k8s-master            1/1     Running            0          29m
kube-system   kube-controller-manager-k8s-master   0/1     CrashLoopBackOff   8          30m
kube-system   kube-proxy-wp7n9                     1/1     Running            0          31m
kube-system   kube-scheduler-k8s-master            1/1     Running            7          29m


[root@k8s-master ~]# kubectl logs -n kube-system kube-controller-manager-k8s-master
I0513 13:49:51.836448       1 serving.go:319] Generated self-signed cert in-memory
I0513 13:49:52.988794       1 controllermanager.go:155] Version: v1.14.1
I0513 13:49:53.003873       1 secure_serving.go:116] Serving securely on 127.0.0.1:10257
I0513 13:49:53.005146       1 deprecated_insecure_serving.go:51] Serving insecurely on [::]:10252
I0513 13:49:53.008661       1 leaderelection.go:217] attempting to acquire leader lease  kube-system/kube-controller-manager...
I0513 13:50:12.687383       1 leaderelection.go:227] successfully acquired lease kube-system/kube-controller-manager
I0513 13:50:12.700344       1 event.go:209] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"kube-system", Name:"kube-controller-manager", UID:"39adc911-7582-11e9-a70e-000c2908c796", APIVersion:"v1", ResourceVersion:"1706", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' k8s-master_fbfa0502-7585-11e9-9939-000c2908c796 became leader
I0513 13:50:13.131264       1 plugins.go:103] No cloud provider specified.
I0513 13:50:13.166088       1 controller_utils.go:1027] Waiting for caches to sync for tokens controller
I0513 13:50:13.368381       1 controllermanager.go:497] Started "podgc"
I0513 13:50:13.368666       1 gc_controller.go:76] Starting GC controller
I0513 13:50:13.368697       1 controller_utils.go:1027] Waiting for caches to sync for GC controller
I0513 13:50:13.368717       1 controller_utils.go:1034] Caches are synced for tokens controller
I0513 13:50:13.453276       1 controllermanager.go:497] Started "attachdetach"
I0513 13:50:13.453534       1 attach_detach_controller.go:323] Starting attach detach controller
I0513 13:50:13.453545       1 controller_utils.go:1027] Waiting for caches to sync for attach detach controller
I0513 13:50:13.461756       1 controllermanager.go:497] Started "clusterrole-aggregation"
I0513 13:50:13.461833       1 clusterroleaggregation_controller.go:148] Starting ClusterRoleAggregator
I0513 13:50:13.461849       1 controller_utils.go:1027] Waiting for caches to sync for ClusterRoleAggregator controller
I0513 13:50:13.517257       1 controllermanager.go:497] Started "endpoint"
I0513 13:50:13.525394       1 endpoints_controller.go:166] Starting endpoint controller
I0513 13:50:13.525425       1 controller_utils.go:1027] Waiting for caches to sync for endpoint controller
I0513 13:50:14.151371       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for rolebindings.rbac.authorization.k8s.io
I0513 13:50:14.151463       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for leases.coordination.k8s.io
I0513 13:50:14.151489       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for limitranges
I0513 13:50:14.163632       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for ingresses.extensions
I0513 13:50:14.163695       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for daemonsets.apps
I0513 13:50:14.163721       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for ingresses.networking.k8s.io
I0513 13:50:14.163742       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for poddisruptionbudgets.policy
W0513 13:50:14.163757       1 shared_informer.go:311] resyncPeriod 67689210101997 is smaller than resyncCheckPeriod 86008177281797 and the informer has already started. Changing it to 86008177281797
I0513 13:50:14.163840       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for networkpolicies.networking.k8s.io
W0513 13:50:14.163848       1 shared_informer.go:311] resyncPeriod 64017623179979 is smaller than resyncCheckPeriod 86008177281797 and the informer has already started. Changing it to 86008177281797
I0513 13:50:14.163867       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for serviceaccounts
I0513 13:50:14.163885       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for deployments.extensions
I0513 13:50:14.163911       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for daemonsets.extensions
I0513 13:50:14.163925       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for controllerrevisions.apps
I0513 13:50:14.163942       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for roles.rbac.authorization.k8s.io
I0513 13:50:14.163965       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for podtemplates
I0513 13:50:14.163994       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for cronjobs.batch
I0513 13:50:14.164004       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for endpoints
I0513 13:50:14.164019       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for replicasets.extensions
I0513 13:50:14.164030       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for replicasets.apps
I0513 13:50:14.164039       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for deployments.apps
I0513 13:50:14.164054       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for jobs.batch
I0513 13:50:14.164079       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for statefulsets.apps
I0513 13:50:14.164097       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for events.events.k8s.io
I0513 13:50:14.164115       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for horizontalpodautoscalers.autoscaling
E0513 13:50:14.164139       1 resource_quota_controller.go:171] initial monitor sync has error: [couldn't start monitor for resource "extensions/v1beta1, Resource=networkpolicies": unable to monitor quota for resource "extensions/v1beta1, Resource=networkpolicies", couldn't start monitor for resource "crd.projectcalico.org/v1, Resource=networkpolicies": unable to monitor quota for resource "crd.projectcalico.org/v1, Resource=networkpolicies"]
I0513 13:50:14.164154       1 controllermanager.go:497] Started "resourcequota"
I0513 13:50:14.171002       1 resource_quota_controller.go:276] Starting resource quota controller
I0513 13:50:14.171096       1 controller_utils.go:1027] Waiting for caches to sync for resource quota controller
I0513 13:50:14.171138       1 resource_quota_monitor.go:301] QuotaMonitor running
I0513 13:50:15.776814       1 controllermanager.go:497] Started "job"
I0513 13:50:15.771658       1 job_controller.go:143] Starting job controller
I0513 13:50:15.807719       1 controller_utils.go:1027] Waiting for caches to sync for job controller
I0513 13:50:23.065972       1 controllermanager.go:497] Started "csrcleaner"
I0513 13:50:23.047495       1 cleaner.go:81] Starting CSR cleaner controller
I0513 13:50:25.019036       1 event.go:209] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"kube-system", Name:"kube-controller-manager", UID:"39adc911-7582-11e9-a70e-000c2908c796", APIVersion:"v1", ResourceVersion:"1706", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' k8s-master_fbfa0502-7585-11e9-9939-000c2908c796 stopped leading
I0513 13:50:25.125784       1 leaderelection.go:263] failed to renew lease kube-system/kube-controller-manager: failed to tryAcquireOrRenew context deadline exceeded
F0513 13:50:25.189307       1 controllermanager.go:260] leaderelection lost


[root@k8s-master ~]# kubectl logs -n kube-system kube-scheduler-k8s-master
I0513 14:16:04.350818       1 serving.go:319] Generated self-signed cert in-memory
W0513 14:16:06.203477       1 authentication.go:387] failed to read in-cluster kubeconfig for delegated authentication: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
W0513 14:16:06.215933       1 authentication.go:249] No authentication-kubeconfig provided in order to lookup client-ca-file in configmap/extension-apiserver-authentication in kube-system, so client certificate authentication won't work.
W0513 14:16:06.215947       1 authentication.go:252] No authentication-kubeconfig provided in order to lookup requestheader-client-ca-file in configmap/extension-apiserver-authentication in kube-system, so request-header client certificate authentication won't work.
W0513 14:16:06.218951       1 authorization.go:177] failed to read in-cluster kubeconfig for delegated authorization: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
W0513 14:16:06.218983       1 authorization.go:146] No authorization-kubeconfig provided, so SubjectAccessReview of authorization tokens won't work.
I0513 14:16:06.961417       1 server.go:142] Version: v1.14.1
I0513 14:16:06.974064       1 defaults.go:87] TaintNodesByCondition is enabled, PodToleratesNodeTaints predicate is mandatory
W0513 14:16:06.997875       1 authorization.go:47] Authorization is disabled
W0513 14:16:06.997889       1 authentication.go:55] Authentication is disabled
I0513 14:16:06.997908       1 deprecated_insecure_serving.go:49] Serving healthz insecurely on [::]:10251
I0513 14:16:06.998196       1 secure_serving.go:116] Serving securely on 127.0.0.1:10259
I0513 14:16:08.872649       1 controller_utils.go:1027] Waiting for caches to sync for scheduler controller
I0513 14:16:08.973148       1 controller_utils.go:1034] Caches are synced for scheduler controller
I0513 14:16:09.003227       1 leaderelection.go:217] attempting to acquire leader lease  kube-system/kube-scheduler...
I0513 14:16:25.814160       1 leaderelection.go:227] successfully acquired lease kube-system/kube-scheduler

kube-controller-manager 和 kube-scheduler 进入 CrashLoopBackoff 的原因是什么？以及如何让kube-controller-manager 和kube-scheduler 运行良好？

【问题讨论】：

    标签： kubernetes kubectl kubeadm kube-controller-manager kube-scheduler


    【解决方案1】：

    我已经重现了您在云 VM 上列出的步骤，并设法使其正常工作。

    有一些想法可能会有所帮助：

    1. 确保满足所有列出的先决条件here

    2. 按照here 的指南安装最新版本的 Docker （选择您使用的正确操作系统）

    3. 使用以下命令安装 kubeadm：

       apt-get update && apt-get install -y apt-transport-https curl
 curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
 cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
 deb https://apt.kubernetes.io/ kubernetes-xenial main
 EOF
 apt-get update
 apt-get install -y kubelet kubeadm kubectl
 apt-mark hold kubelet kubeadm kubectl

    4. 通过执行以下命令确保您获得了最新版本的 kubeadm：apt-get update &amp;&amp; apt-get upgrade

    5. 确保在 kubeadm init 旁边使用正确的参数

    6. 别忘了跑：

      • mkdir -p $HOME/.kube

      • sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

      • sudo chown $(id -u):$(id -g) $HOME/.kube/config

    kubeadm init 完成后（这些命令也是 kubeadm init 输出的一部分）。

    1. 最后应用您在问题中列出的 .yaml 文件。

    请注意，按照上述步骤，您将在 v1.14.3 中拥有 kubectl version、kubelet --version 和 kubectl get no -A，而不是像您展示的那样在 v1.14.1 中，这可能是这种情况。

    希望对你有帮助。

    【讨论】：

      猜你喜欢
      • 2021-04-05
      • 1970-01-01
      • 2019-08-11
      • 1970-01-01
      • 2020-12-02
      • 2020-07-01
      • 2018-12-21
      • 2019-01-31
      • 1970-01-01
      相关资源
      最近更新 更多
      热门标签
      Java Python linux javascript Mysql C# Docker 算法 前端 SpringBoot Redis Vue spring 设计模式 .net core .net kubernetes c++ 数据库 数据结构 大数据 js 机器学习 微服务 Android Go 程序员 面试 JVM ASP.net core 云原生 人工智能 后端 PHP git CSS golang k8s Nginx Django mybatis 深度学习 多线程 React 架构 devops 爬虫 云计算 Spring Boot LeetCode