Google GKE 工作负载突然脱机显示错误：ErrImagePull

Question

我有一个 GKE 集群，直到最近它一直运行良好。现在我看到一大堆 Kubernetes 工作负载显示为脱机，并带有以下错误消息：

 Type     Reason          Age                    From                                                          Message
  ----     ------          ----                   ----                                                          -------
  Normal   Scheduled       6m23s                  default-scheduler                                               Warning  Failed          5m39s (x3 over 6m22s)  kubelet, gke-platsol-bots-staging-default-pool-f489f2f3-rjrq  Error: ErrImagePull
  Normal   BackOff         5m2s (x7 over 6m21s)   kubelet, gke-platsol-bots-staging-default-pool-f489f2f3-rjrq  Back-off pulling image "us.gcr.io/project/poc-app-bot@sha256:b99b5fb1b77407ade49d9bf42a94919e90422fee26c1a46ec6247370bd96c4d8"
  Normal   Pulling         4m49s (x4 over 6m22s)  kubelet, gke-platsol-bots-staging-default-pool-f489f2f3-rjrq  pulling image "us.gcr.io/project/poc-app-bot@sha256:b99b5fb1b77407ade49d9bf42a94919e90422fee26c1a46ec6247370bd96c4d8"
  Warning  Failed          81s (x22 over 6m21s)   kubelet, gke-platsol-bots-staging-default-pool-f489f2f3-rjrq  Error: ImagePullBackOff

不确定是什么改变导致了这个问题。

这是 kubectl 的输出

Name:               project-5dddbd66b5-vpw8q
Namespace:          default
Priority:           0
PriorityClassName:  <none>
Node:               gke-platsol-bots-staging-default-pool-f489f2f3-rjrq/10.x.x.x
Start Time:         Wed, 18 Sep 2019 16:48:23 +0100
Labels:             app=bot
                    pod-template-hash=5dddbd66b5
Annotations:        kubernetes.io/limit-ranger: LimitRanger plugin set: cpu request for container project
Status:             Pending
IP:                 10.20.1.9
Controlled By:      ReplicaSet/bot-5dddbd66b5
Containers:
  project:
    Container ID:
    Image:          us.gcr.io/project/project@sha256:b99b5fb1b77407ade49d9bf42a94919e90422fee26c1a46ec6247370bd96c4d8
    Image ID:
    Port:           8080/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:      100m
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  default-token-99cns:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-99cns
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason  Age                     From                                                          Message
  ----     ------  ----                    ----                                                          -------
  Warning  Failed  4m38s (x793 over 3h4m)  kubelet, gke-platsol-bots-staging-default-pool-f489f2f3-rjrq  Error: ImagePullBackOff

以下是我的 YAML 部署定义中的内容。我没有定义秘密，因为不需要从 Google Container Registry 中提取图像，

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      annotations:
        deployment.kubernetes.io/revision: "3"
        kubectl.kubernetes.io/last-applied-configuration: |
<redacted annotations>
      creationTimestamp: 2019-06-06T08:37:01Z
      generation: 3
      labels:
        app: project
      name: bot
      namespace: default
      resourceVersion: "68945490"
      selfLink: /apis/apps/v1/namespaces/default/deployments/bot
      uid: 412ce711-8836-11e9-905f-42010a8e016c
     image: us.gcr.io/project/app-bot@sha256:b99b5fb1b77407ade49d9bf42a94919e90422fee26c1a46ec6247370bd96c4d8
        imagePullPolicy: IfNotPresent

好的，所以我按照本指南在从 GCR https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/ 提取图像时使用“秘密”修补服务帐户

我通过 SSH 连接到单个节点，并且可以成功为一个应用程序拉取映像，

vinay@cloudshell:~ (project-id)$ docker pull us.gcr.io/project-id/project2-bot@sha256:9817462c743a93bb9206e4b8685
5322f731a768dca18e26b8bfc39b0cc886d31
sha256:9817462c743a93bb9206e4b86855322f731a768dca18e26b8bfc39b0cc886d31: Pulling from project-id/project2-bot
092586df9206: Pull complete
ef599477fae0: Pull complete
4530c6472b5d: Pull complete
d34d61487075: Pull complete
272f46008219: Pull complete
12ff6ccfe7a6: Pull complete
f26b99e1adb1: Pull complete
bb50901cd579: Pull complete
64a286652062: Pull complete
283785ced197: Pull complete
ed5a2062edd6: Pull complete
Digest: sha256:9817462c743a93bb9206e4b86855322f731a768dca18e26b8bfc39b0cc886d31
Status: Downloaded newer image for us.gcr.io/project-id/project2-bot@sha256:9817462c743a93bb9206e4b86855322f731a768dca18e26b8
bfc39b0cc886d31
us.gcr.io/project-id/project2-bot@sha256:9817462c743a93bb9206e4b86855322f731a768dca18e26b8bfc39b0cc886d31

但是这个应用程序似乎抛出了一个错误，

vinay@cloudshell:~ (project-id)$ docker pull us.gcr.io/project-id/project1-plug@sha256:c53ac1c536a1187ce940f9221730cc0eae3103f4313033659e2162a70bc66c59
    sha256:c53ac1c536a1187ce940f9221730cc0eae3103f4313033659e2162a70bc66c59: Pulling from project-id/project1-plug
    a4d8138d0f6b: Pulling fs layer
    dbdc36973392: Pulling fs layer
    f59d6d019dd5: Pulling fs layer
    aaef3e026258: Waiting
    5e86b04a4500: Waiting
    1a6643a2873a: Waiting
    2ad1e30fc17c: Waiting
    ddb5baaf3393: Waiting
    0a7edc889b3c: Waiting
    31a1f16c256b: Waiting
    172a500f7b4d: Waiting
    error pulling image configuration: unknown blob

Answer 1

ErrImagePull 很可能是最常见的，幸运的是它易于调试和诊断。发生这种情况时，您会看到 ErrImagePull 作为状态消息，表明 Kubernetes 无法检索您在清单中指定的映像（可能该映像已从寄存器中删除）。

您可以使用kubectl describe [pod] 命令立即获得有关此错误发生原因的更多详细信息。这并不完全是错误情况，因为 Kubernetes 在技术上处于等待状态，希望映像可用