当 heketi 端点与 pv 和 PVC 不在同一个命名空间时，glusterfs 如何创建卷答案

【问题标题】：How does glusterfs create volume when the heketi endpoint is not in the same namespace as pv and PVC当 heketi 端点与 pv 和 PVC 不在同一个命名空间时，glusterfs 如何创建卷
【发布时间】：2020-03-09 18:57:02
【问题描述】：

我有两个命名空间“runsdata”和“监控”。 heketi pod 和 glusterfs 的 daemonSet pod 都在“runsdata”命名空间下。现在我想在 'monitoring' 命名空间下创建 Prometheus 监视器。因为我需要存储来存储我的 Prometheus 数据。所以我创建了PVC（在'monitoring'ns下）和pv，并在PVC yaml中声明了storageclass来创建相应的卷，以便为Prometheus提供存储。但是当我创建与 pv 绑定的 pvc 并应用 prometheus-server.yaml 时。我得到了错误：

  Warning  FailedMount       18m (x3 over 43m)     kubelet, 172.16.5.151  Unable to attach or mount volumes: unmounted volumes=[prometheus-data-volume], unattached volumes=[prometheus-rules-volume prometheus-token-vcrr2 prometheus-data-volume prometheus-conf-volume]: timed out waiting for the condition
  Warning  FailedMount       13m (x5 over 50m)     kubelet, 172.16.5.151  Unable to attach or mount volumes: unmounted volumes=[prometheus-data-volume], unattached volumes=[prometheus-token-vcrr2 prometheus-data-volume prometheus-conf-volume prometheus-rules-volume]: timed out waiting for the condition
  Warning  FailedMount       3m58s (x35 over 59m)  kubelet, 172.16.5.151  MountVolume.NewMounter initialization failed for volume "data-prometheus-pv" : endpoints "heketi-storage-endpoints" not found

从上面的日志不难知道，storageClass找不到heketi端点来创建volume。因为 heketi 端点位于“runsdata”下。我该如何解决这个问题？

其他信息： 1. pv与pvc

apiVersion: v1
kind: PersistentVolume
metadata:
  name: data-prometheus-pv
  labels:
    pv: data-prometheus-pv
    release: stable
spec:
  capacity:
    storage: 2Gi
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Recycle
  storageClassName: runsdata-static-class
  glusterfs:
    endpoints: "heketi-storage-endpoints"
    path: "runsdata-glusterfs-static-class"
    readOnly: true

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-prometheus-claim
  namespace: monitoring
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 2Gi
  storageClassName: runsdata-static-class
  selector:
    matchLabels:
      pv: data-prometheus-pv
      release: stable

[root@localhost online-prometheus]# kubectl get pv
    NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                              STORAGECLASS            REASON   AGE
    data-config-pv                             1Gi        RWX            Retain           Bound    runsdata/data-config-claim         runsdata-static-class            5d22h
    data-mongo-pv                              1Gi        RWX            Retain           Bound    runsdata/data-mongo-claim          runsdata-static-class            4d4h
    data-prometheus-pv                         2Gi        RWX            Recycle          Bound    monitoring/data-prometheus-claim   runsdata-static-class            151m
    data-static-pv                             1Gi        RWX            Retain           Bound    runsdata/data-static-claim         runsdata-static-class            7d15h
    pvc-02f5ce74-db7c-40ba-b0e1-ac3bf3ba1b37   3Gi        RWX            Delete           Bound    runsdata/data-test-claim           runsdata-static-class            3d5h
    pvc-085ec0f1-6429-4612-9f71-309b94a94463   1Gi        RWX            Delete           Bound    runsdata/data-file-claim           runsdata-static-class            3d17h
    [root@localhost online-prometheus]# kubectl get pvc -n monitoring
    NAME                    STATUS   VOLUME               CAPACITY   ACCESS MODES   STORAGECLASS            AGE
    data-prometheus-claim   Bound    data-prometheus-pv   2Gi        RWX            runsdata-static-class   151m
    [root@localhost online-prometheus]#

heketi 和 glusterfs

[root@localhost online-prometheus]# kubectl get pods -n runsdata|egrep "heketi|gluster"
glusterfs-5btbl                               1/1     Running   1          11d
glusterfs-7gmbh                               1/1     Running   3          11d
glusterfs-rmx7k                               1/1     Running   7          11d
heketi-78ccdb6fd-97tkv                        1/1     Running   2          10d
[root@localhost online-prometheus]#

存储类定义

---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: runsdata-static-class
provisioner: kubernetes.io/glusterfs
allowVolumeExpansion: true
reclaimPolicy: Delete
parameters:
  resturl: "http://10.10.11.181:8080"
  volumetype: "replicate:3"
  restauthenabled: "true"
  restuser: "admin"
  restuserkey: "runsdata-gf-admin"
  #secretNamespace: "runsdata"
  #secretName: "heketi-secret"

【问题讨论】：

这个 IP 和端口 10.10.11.181:8080 是什么？ Heketi pod？
是的，我已经解决了，谢谢

标签： kubernetes namespaces kubernetes-pod glusterfs

【解决方案1】：

解决方案是在当前命名空间下创建端点和服务。然后我们可以在 pv yaml 中使用该服务，如下所示：

[root@localhost gluster]# cat glusterfs-endpoints.yaml 
---
kind: Endpoints
apiVersion: v1
metadata:
  name: glusterfs-cluster
  namespace: monitoring
subsets:
- addresses:
  - ip: 172.16.5.150
  - ip: 172.16.5.151
  - ip: 172.16.5.152
  ports:
  - port: 1
    protocol: TCP
[root@localhost gluster]# cat glusterfs-service.yaml 
apiVersion: v1
kind: Service
metadata:
  name: glusterfs-cluster
  namespace: monitoring
spec:
  ports:
    - port: 1
[root@localhost gluster]#

【讨论】：