【发布时间】：2022-12-04 02:53:54
【问题描述】：
我使用 Calico for CNI 创建了一个单节点 Kubernetes 集群。但是，所有kube-system pod 不断重启：
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-58dbc876ff-nbsvm 0/1 CrashLoopBackOff 3 (12s ago) 5m30s
calico-node-bz82h 1/1 Running 2 (42s ago) 5m30s
coredns-dd9cb97b6-52g5h 1/1 Running 2 (2m16s ago) 17m
coredns-dd9cb97b6-fl9vw 1/1 Running 2 (2m16s ago) 17m
etcd-ai-research-linux 1/1 Running 165 (46s ago) 9h
kube-apiserver-ai-research-linux 1/1 Running 171 (3m58s ago) 9h
kube-controller-manager-ai-research-linux 0/1 CrashLoopBackOff 181 (8s ago) 9h
kube-proxy-6drtb 1/1 Running 108 (85s ago) 9h
kube-scheduler-ai-research-linux 1/1 Running 181 (74s ago) 9h
并且经常，我无法使用kubectl 命令进行连接。虽然做了一个systemctl restart kubelet有时暂时修复它：
admin@ai-research-linux:~$ kubectl get node
The connection to the server 10.10.26.12:6443 was refused - did you specify the right host or port?
admin@ai-research-linux:~$ systemctl restart kubelet
admin@ai-research-linux:~$ kubectl get node
NAME STATUS ROLES AGE VERSION
ai-research-linux Ready control-plane 10h v1.25.4
查看每个 pod 的Events，每个人都有这个问题Pod sandbox changed, it will be killed and re-created：
kube-controller-manager-ai-research-linux:
Normal Killing 3m19s kubelet Stopping container kube-controller-manager
Normal SandboxChanged 3m18s kubelet Pod sandbox changed, it will be killed and re-created.
Warning BackOff 3m9s (x5 over 3m18s) kubelet Back-off restarting failed container
Normal Pulled 2m54s (x2 over 3m24s) kubelet Container image "registry.k8s.io/kube-controller-manager:v1.25.4" already present on machine
Normal Created 2m54s (x2 over 3m24s) kubelet Created container kube-controller-manager
Normal Started 2m54s (x2 over 3m23s) kubelet Started container kube-controller-manager
kube-proxy-6drtb:
Warning FailedMount 6m34s (x8 over 7m40s) kubelet MountVolume.SetUp failed for volume "kube-api-access-86wkc" : failed to fetch token: Post "https://10.10.26.12:6443/api/v1/namespaces/kube-system/serviceaccounts/kube-proxy/token": dial tcp 10.10.26.12:6443: connect: connection refused
Warning FailedMount 3m13s (x8 over 4m19s) kubelet MountVolume.SetUp failed for volume "kube-api-access-86wkc" : failed to fetch token: Post "https://10.10.26.12:6443/api/v1/namespaces/kube-system/serviceaccounts/kube-proxy/token": dial tcp 10.10.26.12:6443: connect: connection refused
Warning FailedMount 2m17s kubelet Unable to attach or mount volumes: unmounted volumes=[kube-api-access-86wkc], unattached volumes=[xtables-lock lib-modules kube-api-access-86wkc kube-proxy]: timed out waiting for the condition
Normal Killing 2m2s kubelet Stopping container kube-proxy
Normal SandboxChanged 2m1s kubelet Pod sandbox changed, it will be killed and re-created.
Warning BackOff 119s (x3 over 2m1s) kubelet Back-off restarting failed container
Normal Pulled 104s (x2 over 2m2s) kubelet Container image "registry.k8s.io/kube-proxy:v1.25.4" already present on machine
Normal Created 104s (x2 over 2m2s) kubelet Created container kube-proxy
Normal Started 104s (x2 over 2m2s) kubelet Started container kube-proxy
kube-scheduler-ai-research-linux:
Normal Killing 6m29s kubelet Stopping container kube-scheduler
Normal SandboxChanged 6m28s kubelet Pod sandbox changed, it will be killed and re-created.
Warning BackOff 6m20s (x5 over 6m28s) kubelet Back-off restarting failed container
Normal Pulled 6m8s (x2 over 8m3s) kubelet Container image "registry.k8s.io/kube-scheduler:v1.25.4" already present on machine
Normal Created 6m8s (x2 over 8m3s) kubelet Created container kube-scheduler
Normal Started 6m8s (x2 over 8m3s) kubelet Started container kube-scheduler
kube-apiserver-ai-research-linux:
Normal SandboxChanged 10m (x2 over 11m) kubelet Pod sandbox changed, it will be killed and re-created.
Warning BackOff 10m (x4 over 10m) kubelet Back-off restarting failed container
Normal Pulled 10m (x2 over 11m) kubelet Container image "registry.k8s.io/kube-apiserver:v1.25.4" already present on machine
Normal Created 10m (x2 over 11m) kubelet Created container kube-apiserver
Normal Started 10m (x2 over 11m) kubelet Started container kube-apiserver
Normal Killing 9m56s (x3 over 11m) kubelet Stopping container kube-apiserver
Warning Unhealthy 9m49s kubelet Liveness probe failed: Get "https://10.10.26.12:6443/livez": dial tcp 10.10.26.12:6443: connect: connection refused
Warning Unhealthy 5m48s (x42 over 9m55s) kubelet Readiness probe failed: Get "https://10.10.26.12:6443/readyz": dial tcp 10.10.26.12:6443: connect: connection refused
etcd-ai-research-linux:
Normal Killing 2m10s kubelet Stopping container etcd
Normal SandboxChanged 2m9s kubelet Pod sandbox changed, it will be killed and re-created.
Normal Pulled 2m9s kubelet Container image "registry.k8s.io/etcd:3.5.5-0" already present on machine
Normal Created 2m9s kubelet Created container etcd
Normal Started 2m9s kubelet Started container etcd
coredns-dd9cb97b6-fl9vw:
Normal Killing 6m15s kubelet Stopping container coredns
Normal SandboxChanged 6m9s kubelet Pod sandbox changed, it will be killed and re-created.
Normal Pulled 6m8s kubelet Container image "registry.k8s.io/coredns/coredns:v1.9.3" already present on machine
Normal Created 6m8s kubelet Created container coredns
Normal Started 6m8s kubelet Started container coredns
印花布节点-bz82h:
Warning Unhealthy 7m44s (x2 over 7m45s) kubelet Readiness probe failed: calico/node is not ready: BIRD is not ready: Error querying BIRD: unable to connect to BIRDv4 socket: dial unix /var/run/calico/bird.ctl: connect: connection refused
Warning BackOff 7m41s kubelet Back-off restarting failed container
Normal SandboxChanged 7m41s (x2 over 7m49s) kubelet Pod sandbox changed, it will be killed and re-created.
Normal Started 7m40s (x2 over 7m49s) kubelet Started container upgrade-ipam
Normal Created 7m40s (x2 over 7m49s) kubelet Created container upgrade-ipam
Normal Pulled 7m40s (x2 over 7m49s) kubelet Container image "docker.io/calico/cni:v3.24.1" already present on machine
Normal Started 7m39s (x2 over 7m48s) kubelet Started container install-cni
Normal Created 7m39s (x2 over 7m48s) kubelet Created container install-cni
Normal Pulled 7m39s (x2 over 7m48s) kubelet Container image "docker.io/calico/cni:v3.24.1" already present on machine
Normal Killing 2m5s (x5 over 7m50s) kubelet Stopping container calico-node
印花布-kube-控制器-58dbc876ff-nbsvm：
Warning Failed 8m55s kubelet Error: services have not yet been read at least once, cannot construct envvars
Normal Created 8m30s (x2 over 8m54s) kubelet Created container calico-kube-controllers
Normal Started 8m30s (x2 over 8m54s) kubelet Started container calico-kube-controllers
Normal Killing 8m29s (x2 over 8m52s) kubelet Stopping container calico-kube-controllers
Warning Unhealthy 8m29s kubelet Readiness probe errored: rpc error: code = NotFound desc = failed to exec in container: failed to load task: no running task found: task e02dcaaf40a14398790b5ed3b6219279fa28464d71daad01f3297e80d3749fe4 not found: not found
Normal SandboxChanged 8m28s (x2 over 8m51s) kubelet Pod sandbox changed, it will be killed and re-created.
Normal Pulled 7m47s (x4 over 8m55s) kubelet Container image "docker.io/calico/kube-controllers:v3.24.1" already present on machine
Warning BackOff 3m41s (x32 over 8m50s) kubelet Back-off restarting failed container
对于拒绝连接的问题，我尝试了重启kubelet、containerd、关闭swap...只有重启kubelet似乎可以暂时解决问题。
对于 Calico BIRD 连接问题，我尝试了这个解决方案：Solution
【问题讨论】：
标签： kubernetes networking kubelet calico project-calico