【发布时间】：2020-03-01 19:30:33
【问题描述】：
我部署了一个带有 1 个 pod 和 2 个容器的 helm chart (statefulSet)，其中一个容器附加了一个 PV (readwriteonce)。升级时，需要 30 分钟（7 次失败尝试）才能再次上升（因此服务关闭 30 分钟）
一些上下文：
- PV 使用默认的 GKE 类
- 是一个 GKE 区域，每个区域中有一个节点
- 即使没有强制执行，Pod 也会在同一个节点中再次星标（所以我可以看到不是节点转移）
- 我在 azure AKS 中遇到过类似问题，它也失败了 7 次，但速度更快，因此停机时间最短，并且涉及节点传输
yaml文件的相关部分：
volumeMounts:
- mountPath: /app/data
name: prod-data
volumeClaimTemplates:
- metadata:
creationTimestamp: null
name: prod-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 500Gi
storageClassName: standard
volumeMode: Filesystem
错误信息：
Unable to mount volumes for pod "foo" timeout expired waiting for volumes to attach or mount for pod "foo". list of unmounted volumes=[foo] list of unattached volumes [foo default-token-foo]
附加上下文，这是触发StatefulSet升级后发生的情况：
还没有改变
Name: prod-data-prod-0
Namespace: prod
StorageClass: standard
Status: Bound
Volume: pvc-16f49d12-f644-11e9-952a-4201ac100008
Labels: app=prod
release=prod
Annotations: pv.kubernetes.io/bind-completed: yes
pv.kubernetes.io/bound-by-controller: yes
volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/gce-pd
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 500Gi
Access Modes: RWO
VolumeMode: Filesystem
Mounted By: prod-0
Events: <none>
然后是第一个错误
Unable to mount volumes for pod "prod-0_prod(89fb0cf5-0008-11ea-b349-4201ac100009)": timeout expired waiting for volumes to attach or mount for pod "prod"/"prod-0". list of unmounted volumes=[prod-data]. list of unattached volumes=[prod-data default-token-4624v]
还是一样的描述
Name: prod-data-prod-0
Namespace: prod
StorageClass: standard
Status: Bound
Volume: pvc-16f49d12-f644-11e9-952a-4201ac100008
Labels: app=prod
release=prod
Annotations: pv.kubernetes.io/bind-completed: yes
pv.kubernetes.io/bound-by-controller: yes
volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/gce-pd
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 500Gi
Access Modes: RWO
VolumeMode: Filesystem
Mounted By: prod-0
Events: <none>
第二次挂载失败后，这是 pod 描述
Conditions:
Type Status
Initialized False
Ready False
ContainersReady False
PodScheduled True
Volumes:
vlapi-prod-data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: prod-data-prod-0
ReadOnly: false
default-token-4624v:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-4624v
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
FailedMount nr 3 pvc 描述没有变化 pod 描述的事件
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 8m44s default-scheduler Successfully assigned prod/prod-0 to gke-vlgke-a-default-pool-312c60b0-p8lb
Warning FailedMount 2m8s (x3 over 6m41s) kubelet, gke-vlgke-a-default-pool-312c60b0-p8lb Unable to mount volumes for pod "prod-0_prod(89fb0cf5-0008-11ea-b349-4201ac100009)": timeout expired waiting for volumes to attach or mount for pod "prod"/"prod-0". list of unmounted volumes=[prod-data]. list of unattached volumes=[prod-data default-token-4624v]
警告 FailedMount 48s (x4 over 7m38s) 警告 FailedMount 13s (x5 over 9m17s)
Name: pvc-16f49d12-f644-11e9-952a-4201ac100008
Labels: failure-domain.beta.kubernetes.io/region=europe-west1
failure-domain.beta.kubernetes.io/zone=europe-west1-d
Annotations: kubernetes.io/createdby: gce-pd-dynamic-provisioner
pv.kubernetes.io/bound-by-controller: yes
pv.kubernetes.io/provisioned-by: kubernetes.io/gce-pd
Finalizers: [kubernetes.io/pv-protection]
StorageClass: standard
Status: Bound
Claim: prod/prod-data-prod-0
Reclaim Policy: Retain
Access Modes: RWO
VolumeMode: Filesystem
Capacity: 500Gi
Node Affinity:
Required Terms:
Term 0: failure-domain.beta.kubernetes.io/zone in [europe-west1-d]
failure-domain.beta.kubernetes.io/region in [europe-west1]
Message:
Source:
Type: GCEPersistentDisk (a Persistent Disk resource in Google Compute Engine)
PDName: gke-vlgke-a-0d42343f-d-pvc-16f49d12-f644-11e9-952a-4201ac100008
FSType: ext4
Partition: 0
ReadOnly: false
FailedMount 47s（x6 超过 12m） FailedMount 11s（x7 超过 13m） FailedMount 33s（x8 超过 16m） FailedMount 9s（x9 超过 18m） FailedMount 0s (x10 over 20m) FailedMount 超时之间约 2m
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 24m default-scheduler Successfully assigned prod/prod-0 to gke-vlgke-a-default-pool-312c60b0-p8lb
Warning FailedMount 2m4s (x10 over 22m) kubelet, gke-vlgke-a-default-pool-312c60b0-p8lb Unable to mount volumes for pod "prod-0_prod(89fb0cf5-0008-11ea-b349-4201ac100009)": timeout expired waiting for volumes to attach or mount for pod "prod"/"prod-0". list of unmounted volumes=[prod-data]. list of unattached volumes=[prod-data default-token-4624v]
Normal Pulling 11s kubelet, gke-gke-default-pool-312c60b0-p8lb Pulling image "gcr.io/foo-251818/`foo:2019-11-05"
第 11 次尝试挂载成功 PVC 描述没有变化
【问题讨论】：
-
你在升级什么？状态集？节点？集群版本？这个过程中PVC和PV的状态如何？
-
statefulset 正在升级（即：新图像）。 PVC状态是什么意思？当 statefulset 启动时，按预期安装，当 satefulset 完成获取新图像并尝试安装 PVC 时，我在帖子中收到错误。
-
触发升级后，检查pvc（kubectl get，kubectl describe）
-
@PatrickW 我在整个错误过程中为问题添加了更多信息。谢谢！
-
这个问题最初出现在一个卷“foo”上，你可以添加 foo 的配置是不是 persistentVolumeClaim？