kubernetes aws-efs-csi-driver 和权限

Question

我正在使用 bitnami/etcd 图表，它可以通过EFS mounted pvc 创建快照。

但是，在配置 aws-efs-csi-driver 并将 PVC 安装到任何 非 root pod（用户/gid 为 1001）

后，我得到了权限错误

我正在使用 helm chart https://kubernetes-sigs.github.io/aws-efs-csi-driver/ 2.2.0 版

图表的值：

# you can obtain the fileSystemId with
# aws efs describe-file-systems --query "FileSystems[*].FileSystemId"
storageClasses:
  - name: efs
    parameters:
      fileSystemId: fs-exxxxxxx
      directoryPerms: "777"
      gidRangeStart: "1000"
      gidRangeEnd: "2000"
      basePath: "/snapshots"

# enable it after the following issue is resolved
# https://github.com/bitnami/charts/issues/7769
# node:
#   nodeSelector:
#     etcd: "true"

然后我手动创建了 PV

apiVersion: v1
kind: PersistentVolume
metadata:
  name: etcd-snapshotter-pv
  annotations:
    argocd.argoproj.io/sync-wave: "60"
spec:
  capacity:
    storage: 32Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: efs
  csi:
    driver: efs.csi.aws.com
    volumeHandle: fs-exxxxxxx

然后，如果我将该 EFS PVC 安装在非 rood pod 中，我会收到以下错误

➜ klo etcd-snapshotter-001-ph8w9                          
etcd 23:18:38.76 DEBUG ==> Using endpoint etcd-snapshotter-001-ph8w9:2379
{"level":"warn","ts":1633994320.7789018,"logger":"client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0005ea380/#initially=[etcd-snapshotter-001-ph8w9:2379]","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing dial tcp 10.120.2.206:2379: connect: connection refused\""}
etcd-snapshotter-001-ph8w9:2379 is unhealthy: failed to commit proposal: context deadline exceeded
Error: unhealthy cluster
etcd 23:18:40.78 WARN  ==> etcd endpoint etcd-snapshotter-001-ph8w9:2379 not healthy. Trying a different endpoint
etcd 23:18:40.78 DEBUG ==> Using endpoint etcd-2.etcd-headless.etcd.svc.cluster.local:2379
etcd-2.etcd-headless.etcd.svc.cluster.local:2379 is healthy: successfully committed proposal: took = 1.6312ms
etcd 23:18:40.87 INFO  ==> Snapshotting the keyspace
Error: could not open /snapshots/db-2021-10-11_23-18.part (open /snapshots/db-2021-10-11_23-18.part: permission denied)

因此，我必须生成一个新的“根”pod，进入 pod 并手动调整权限

apiVersion: v1
kind: Pod
metadata:
  name: perm
spec:
  securityContext:
    runAsUser: 0
    runAsGroup: 0
    fsGroup: 0    
  containers:
  - name: app1
    image: busybox
    command: ["/bin/sh"]
    args: ["-c", "sleep 3000"]
    volumeMounts:
    - name: persistent-storage
      mountPath: /snapshots
    securityContext:
      runAsUser: 0
      runAsGroup: 0
  volumes:
  - name: persistent-storage
    persistentVolumeClaim:
      claimName: etcd-snapshotter
  nodeSelector:
    etcd: "true"

k apply -f setup.yaml
k exec -ti perm -- ash
cd /snapshots
/snapshots # chown -R 1001.1001 .
/snapshots # chmod -R 777 .
/snapshots # exit

➜ k create job --from=cronjob/etcd-snapshotter etcd-snapshotter-001
job.batch/etcd-snapshotter-001 created

➜ klo etcd-snapshotter-001-bmv79                          
etcd 23:31:10.22 DEBUG ==> Using endpoint etcd-1.etcd-headless.etcd.svc.cluster.local:2379
etcd-1.etcd-headless.etcd.svc.cluster.local:2379 is healthy: successfully committed proposal: took = 2.258532ms
etcd 23:31:10.32 INFO  ==> Snapshotting the keyspace
{"level":"info","ts":1633995070.4244702,"caller":"snapshot/v3_snapshot.go:68","msg":"created temporary db file","path":"/snapshots/db-2021-10-11_23-31.part"}
{"level":"info","ts":1633995070.4907935,"logger":"client","caller":"v3/maintenance.go:211","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":1633995070.4908395,"caller":"snapshot/v3_snapshot.go:76","msg":"fetching snapshot","endpoint":"etcd-1.etcd-headless.etcd.svc.cluster.local:2379"}
{"level":"info","ts":1633995070.4965465,"logger":"client","caller":"v3/maintenance.go:219","msg":"completed snapshot read; closing"}
{"level":"info","ts":1633995070.544217,"caller":"snapshot/v3_snapshot.go:91","msg":"fetched snapshot","endpoint":"etcd-1.etcd-headless.etcd.svc.cluster.local:2379","size":"320 kB","took":"now"}
{"level":"info","ts":1633995070.5507936,"caller":"snapshot/v3_snapshot.go:100","msg":"saved","path":"/snapshots/db-2021-10-11_23-31"}
Snapshot saved at /snapshots/db-2021-10-11_23-31

➜ k exec -ti perm -- ls -la /snapshots                             
total 924
drwxrwxrwx    2 1001     1001          6144 Oct 11 23:31 .
drwxr-xr-x    1 root     root            46 Oct 11 23:25 ..
-rw-------    1 1001     root        319520 Oct 11 23:31 db-2021-10-11_23-31

有没有办法自动化这个？

我在存储类中有这个设置

gidRangeStart: "1000"
gidRangeEnd: "2000"

但它没有效果。

PVC 定义为：

➜ kg pvc etcd-snapshotter -o yaml                
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: efs.csi.aws.com
  name: etcd-snapshotter
  namespace: etcd
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 32Gi
  storageClassName: efs
  volumeMode: Filesystem
  volumeName: etcd-snapshotter-pv

Answer 1

默认情况下，StorageClass 字段 provisioningMode 未设置，请将其设置为 provisioningMode: "efs-ap" 以启用与接入点的动态配置。