RabbitMQ 安装 - pod 具有未绑定的即时 PersistentVolumeClaims

Question

我正在尝试在 Kubernetes 中安装 RabbitMQ，并按照 RabbitMQ 站点 https://www.rabbitmq.com/blog/2020/08/10/deploying-rabbitmq-to-kubernetes-whats-involved/ 上的条目进行操作。

请注意我的 CentOS 7 和 Kubernetes 1.18。此外，我什至不确定这是部署 RabbitMQ 的最佳方式，但它是我能找到的最好的文档。我确实发现了一些说 volumeClaimTemplates 不支持 NFS，所以我想知道这是否是问题所在。

我已经使用 NFS 添加了我的持久卷：

kind: PersistentVolume
apiVersion: v1
metadata:
  name: rabbitmq-nfs-pv
  namespace: ninegold-rabbitmq
spec:
  capacity:
    storage: 5Gi
  accessModes:
  - ReadWriteOnce
  nfs:
    path: /var/nfsshare
    server: 192.168.1.241
  persistentVolumeReclaimPolicy: Retain

它正确地创建了 PV。

[admin@centos-controller ~]$ kubectl get pv -n ninegold-rabbitmq
NAME                                      CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                                     STORAGECLASS   REASON   AGE
ninegold-platform-custom-config-br        1Gi        RWX            Retain           Bound       ninegold-platform/ninegold-db-pgbr-repo                           22d
ninegold-platform-custom-config-pgadmin   1Gi        RWX            Retain           Bound       ninegold-platform/ninegold-db-pgadmin                             21d
ninegold-platform-custom-config-pgdata    1Gi        RWX            Retain           Bound       ninegold-platform/ninegold-db                                     22d
rabbitmq-nfs-pv                           5Gi        RWO            Retain           Available                                                                     14h

然后我添加我的 StatefulSet。

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: rabbitmq
  namespace: ninegold-rabbitmq
spec:
  selector:
    matchLabels:
      app: "rabbitmq"
  # headless service that gives network identity to the RMQ nodes, and enables them to cluster
  serviceName: rabbitmq-headless # serviceName is the name of the service that governs this StatefulSet. This service must exist before the StatefulSet, and is responsible for the network identity of the set. Pods get DNS/hostnames that follow the pattern: pod-specific-string.serviceName.default.svc.cluster.local where "pod-specific-string" is managed by the StatefulSet controller.
  volumeClaimTemplates:
  - metadata:
      name: rabbitmq-data
      namespace: ninegold-rabbitmq
    spec:
      storageClassName: local-storage
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: "5Gi"
  template:
    metadata:
      name: rabbitmq
      namespace: ninegold-rabbitmq
      labels:
        app: rabbitmq
    spec:
      initContainers:
      # Since k8s 1.9.4, config maps mount read-only volumes. Since the Docker image also writes to the config file,
      # the file must be mounted as read-write. We use init containers to copy from the config map read-only
      # path, to a read-write path
      - name: "rabbitmq-config"
        image: busybox:1.32.0
        volumeMounts:
        - name: rabbitmq-config
          mountPath: /tmp/rabbitmq
        - name: rabbitmq-config-rw
          mountPath: /etc/rabbitmq
        command:
        - sh
        - -c
        # the newline is needed since the Docker image entrypoint scripts appends to the config file
        - cp /tmp/rabbitmq/rabbitmq.conf /etc/rabbitmq/rabbitmq.conf && echo '' >> /etc/rabbitmq/rabbitmq.conf;
          cp /tmp/rabbitmq/enabled_plugins /etc/rabbitmq/enabled_plugins
      volumes:
      - name: rabbitmq-config
        configMap:
          name: rabbitmq-config
          optional: false
          items:
          - key: enabled_plugins
            path: "enabled_plugins"
          - key: rabbitmq.conf
            path: "rabbitmq.conf"
      # read-write volume into which to copy the rabbitmq.conf and enabled_plugins files
      # this is needed since the docker image writes to the rabbitmq.conf file
      # and Kubernetes Config Maps are mounted as read-only since Kubernetes 1.9.4
      - name: rabbitmq-config-rw
        emptyDir: {}
      - name: rabbitmq-data
        persistentVolumeClaim:
          claimName: rabbitmq-data
      serviceAccount: rabbitmq
      # The Docker image runs as the `rabbitmq` user with uid 999 
      # and writes to the `rabbitmq.conf` file
      # The security context is needed since the image needs
      # permission to write to this file. Without the security 
      # context, `rabbitmq.conf` is owned by root and inaccessible
      # by the `rabbitmq` user
      securityContext:
        fsGroup: 999
        runAsUser: 999
        runAsGroup: 999
      containers:
      - name: rabbitmq
        # Community Docker Image
        image: rabbitmq:latest
        volumeMounts:
        # mounting rabbitmq.conf and enabled_plugins
        # this should have writeable access, this might be a problem
        - name: rabbitmq-config-rw
          mountPath: "/etc/rabbitmq"
          # mountPath: "/etc/rabbitmq/conf.d/"
          mountPath: "/var/lib/rabbitmq"
        # rabbitmq data directory
        - name: rabbitmq-data
          mountPath: "/var/lib/rabbitmq/mnesia"
        env:
        - name: RABBITMQ_DEFAULT_PASS
          valueFrom:
            secretKeyRef:
              name: rabbitmq-admin
              key: pass
        - name: RABBITMQ_DEFAULT_USER
          valueFrom:
            secretKeyRef:
              name: rabbitmq-admin
              key: user
        - name: RABBITMQ_ERLANG_COOKIE
          valueFrom:
            secretKeyRef:
              name: erlang-cookie
              key: cookie
        ports:
        - name: amqp
          containerPort: 5672
          protocol: TCP
        - name: management
          containerPort: 15672
          protocol: TCP
        - name: prometheus
          containerPort: 15692
          protocol: TCP
        - name: epmd
          containerPort: 4369
          protocol: TCP
        livenessProbe:
          exec:
            # This is just an example. There is no "one true health check" but rather
            # several rabbitmq-diagnostics commands that can be combined to form increasingly comprehensive
            # and intrusive health checks.
            # Learn more at https://www.rabbitmq.com/monitoring.html#health-checks.
            #
            # Stage 2 check:
            command: ["rabbitmq-diagnostics", "status"]
          initialDelaySeconds: 60
          # See https://www.rabbitmq.com/monitoring.html for monitoring frequency recommendations.
          periodSeconds: 60
          timeoutSeconds: 15
        readinessProbe: # probe to know when RMQ is ready to accept traffic
          exec:
            # This is just an example. There is no "one true health check" but rather
            # several rabbitmq-diagnostics commands that can be combined to form increasingly comprehensive
            # and intrusive health checks.
            # Learn more at https://www.rabbitmq.com/monitoring.html#health-checks.
            #
            # Stage 1 check:
            command: ["rabbitmq-diagnostics", "ping"]
          initialDelaySeconds: 20
          periodSeconds: 60
          timeoutSeconds: 10

但是我的状态集没有绑定，我收到以下错误：

running "VolumeBinding" filter plugin for pod "rabbitmq-0": pod has unbound immediate PersistentVolumeClaims

PVC 未正确绑定到 PV，但仍处于挂起状态。

[admin@centos-controller ~]$ kubectl get pvc -n ninegold-rabbitmq
NAME                       STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS    AGE
rabbitmq-data-rabbitmq-0   Pending                                      local-storage   14h

我已经仔细检查了容量，accessModes，我不确定为什么它没有绑定。 我的示例来自这里https://github.com/rabbitmq/diy-kubernetes-examples/tree/master/gke，我所做的唯一更改是绑定我的 NFS 卷。

任何帮助将不胜感激。

Answer 1

在您的 YAML 中，我发现了一些错误配置。

1. local-storage 班级。

我假设，您使用 Documentation example 创建了 local-storage。有资料表明：

本地卷目前不支持动态配置，但仍应创建 StorageClass 以延迟卷绑定直到 Pod 调度。

当您想使用volumeClaimTemplates 时，您将使用Dynamic Provisioning。 Medium article 中解释得很好。

StatefulSet 中的 PV

具体到卷部分，StatefulSet 提供了一个名为volumeClaimTemplates 的密钥。这样，您可以动态地从存储类中请求PVC。作为新的statefulset 应用定义的一部分，替换卷...PVC 被命名为volumeClaimTemplate name + pod-name + ordinal number。

由于local-storage不支持dynamic provisioning，所以它不起作用。您需要将NFS storageclass 与适当的配置器一起使用，或者手动创建PV。

此外，当您为每个 pod 使用 volumeClaimTemplates 时，它将创建 Pv 和 PVC。此外，PVC 和 PV 以 1:1 的关系绑定。更多详情可以查看this SO thread。

1. 错误unbound immediate PersistentVolumeClaims

这意味着dynamic provisioning 没有按预期工作。如果您检查kubectl get pv,pvc，您将看不到任何新的PV,PVC，名称为：volumeClaimTemplate name + pod-name + ordinal number。

1. claimName: rabbitmq-data

我假设，在此声明中，您想将它挂载到由 volumeClaimTemplates 创建的 PV，但它没有被创建。 PV 的第一个 pod 名称为 rabbitmq-data-rabbitmq-0，第二个 pod 名称为 rabbitmq-data-rabbitmq-1。

作为最后一部分，这篇文章 - Kubernetes : NFS and Dynamic NFS provisioning 可能会有所帮助。