豆荚死了。容器在 K8s 1.13 中启动答案

【问题标题】：Pod is Dead. Container is up in K8s 1.13豆荚死了。容器在 K8s 1.13 中启动
【发布时间】：2020-01-04 01:02:55
【问题描述】：

有没有人遇到过 pod 从一个节点（即节点 A）被逐出，然后立即被调度到另一个节点（即节点 B）的情况。但是 Node A 的 kubelet 一直抱怨无法删除 pod 的容器。容器已启动且未被杀死。

Aug 30 20:29:36 staging-node-4 kubelet[2173]: I0830 20:29:36.358238    2173 kubelet_pods.go:1073] Killing unwanted pod "thanos-compactor-0"
Aug 30 20:29:36 staging-node-4 kubelet[2173]: I0830 20:29:36.362581    2173 kuberuntime_container.go:559] Killing container "docker://b22287cd406c3fe9eff4ff2df1792c6f84b5b92d001359f05ea73f8788715609" with 30 second grace period
Aug 30 20:29:36 staging-node-4 kubelet[2173]: E0830 20:29:36.363416    2173 kuberuntime_container.go:71] Can't make a ref to pod "thanos-compactor-0_thanos(23096418-c7b2-11e9-9be7-005056bc2883)", container thanos: selfLink was empty, can't make reference
Aug 30 20:29:38 staging-node-4 kubelet[2173]: I0830 20:29:38.362191    2173 kubelet_pods.go:1073] Killing unwanted pod "thanos-compactor-0"
Aug 30 20:29:38 staging-node-4 kubelet[2173]: I0830 20:29:38.366884    2173 kuberuntime_container.go:559] Killing container "docker://b22287cd406c3fe9eff4ff2df1792c6f84b5b92d001359f05ea73f8788715609" with 30 second grace period
Aug 30 20:29:38 staging-node-4 kubelet[2173]: E0830 20:29:38.367901    2173 kuberuntime_container.go:71] Can't make a ref to pod "thanos-compactor-0_thanos(23096418-c7b2-11e9-9be7-005056bc2883)", container thanos: selfLink was empty, can't make reference
Aug 30 20:29:38 staging-node-4 kubelet[2173]: I0830 20:29:38.997510    2173 setters.go:72] Using node IP: "172.17.21.146"
Aug 30 20:29:40 staging-node-4 kubelet[2173]: I0830 20:29:40.364388    2173 kubelet_pods.go:1073] Killing unwanted pod "thanos-compactor-0"
Aug 30 20:29:40 staging-node-4 kubelet[2173]: I0830 20:29:40.368226    2173 kuberuntime_container.go:559] Killing container "docker://b22287cd406c3fe9eff4ff2df1792c6f84b5b92d001359f05ea73f8788715609" with 30 second grace period
Aug 30 20:29:40 staging-node-4 kubelet[2173]: E0830 20:29:40.369235    2173 kuberuntime_container.go:71] Can't make a ref to pod "thanos-compactor-0_thanos(23096418-c7b2-11e9-9be7-005056bc2883)", container thanos: selfLink was empty, can't make reference

b22287cd 是没有被删除的docker containerid。

【问题讨论】：

标签： docker kubernetes kubelet

【解决方案1】：

几周前我遇到了这个问题。 Pod 被逐出节点，有时“卡”在终止状态。

通过运行 kubectl -n <NAMESPACE> get pod 检查您的 pod 状态

使用kubectl delete pod <PODNAME> --grace-period=0 --force --namespace <NAMESPACE> 解决了问题。

【讨论】：

就我而言，它们并未处于终止状态。事实上，pod 不再作为 statefulset 存在。它只是启动和运行的容器