如何为 Azure Keyvault + SecretProviderClass + imagePullSecrets + 私有 docker 存储库配置部署文件

Question

Azure Keyvault + SecretProviderClass + imagePullSecrets+ Private docker repository 组合如何配置部署文件。

我们有私有 Docker 存储库来维护映像，现在我们需要在 Azure 密钥库中维护该 Docker 存储库的凭据，使用 SecretProviderClass 将其导入 AKS，在“imagePullSecrets”下使用该密钥

# This is a SecretProviderClass example using system-assigned identity to access your key vault
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: azure-kvname-system-harbor
spec:
  provider: azure
  secretObjects:
  - secretName: harborcredentialvault
    data:
    - key: harborcredentialvaultkey
      objectName: harborcredentialvault
    type: kubernetes.io/dockerconfigjson
  parameters:
    usePodIdentity: "false"
    useVMManagedIdentity: "true"    # Set to true for using managed identity
    userAssignedIdentityID: ""      # If empty, then defaults to use the system assigned identity on the VM
    keyvaultName: "<Keyvault name>"
    cloudName: ""                   # [OPTIONAL for Azure] if not provided, the Azure environment defaults to AzurePublicCloud
    objects:  |
      array:
        - |
          objectName: harborcredentialvault
          objectType: secret        # object types: secret, key, or cert
          objectVersion: ""         # [OPTIONAL] object versions, default to latest if empty
    tenantId: "<tenant ID>"           # The tenant ID of the key vault

        - name: harborcredentialvault
          valueFrom: 
            secretKeyRef:
              name: keyvault-secret
              key: harborcredentialvaultkey
      imagePullSecrets:
       - name: ${harborcredentialvault}
        volumeMounts:
         - mountPath: "/mnt/secrets-store"
           name: secrets-store01-inline
           readOnly: true
       - name: secrets-store01-inline
         csi:
           driver: secrets-store.csi.k8s.io
           readOnly: true
           volumeAttributes:
             secretProviderClass: "azure-kvname-system-harbor"

Answer 1

由于您没有提供真正的问题或错误，我会有点笼统：

对于 AKS/KeyVault 集成，请务必了解您使用节点池的 Kubelet 身份访问 Key Vault，而不是使用 AKS 的托管身份（如 here 所述）。因此，如果您使用托管标识，userAssignedIdentityID 不应为空。

所以我们需要授予 Kubelet Identity 访问 Key Vault 的权限，例如：

export KUBE_ID=$(az aks show -g <resource group> -n <aks cluster name> --query identityProfile.kubeletidentity.objectId -o tsv)
export AKV_ID=$(az keyvault show -g <resource group> -n <akv name> --query id -o tsv)
az role assignment create --assignee $KUBE_ID --role "Key Vault Secrets Officer" --scope $AKV_ID

$KUBE_ID 的结果还需要加上SecretProviderClass：

userAssignedIdentityID: "RESULT"

来自官方的example，您的SecretProviderClass 看起来很适合这个用例。

这将是 pod 配置：

spec:
  containers:
  - name: demo
    image: demo
    volumeMounts:
    - name: secrets-store-inline
      mountPath: "/mnt/secrets-store"
      readOnly: true
  imagePullSecrets:
    - name: harborcredentialvault
  volumes:
  - name: secrets-store-inline
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: "azure-kvname-system-harbor"

这应该将 Key Vault 机密同步到 Kubernetes 机密。这里也是documentation。

您应该考虑的一件事是 = The secrets will only sync once you start a pod mounting the secrets. Solely relying on the syncing with Kubernetes secrets feature thus does not work.

话虽如此，您可能需要另一个具有公共映像的 pod 来同步您的集群的私有拉取机密，因为您的 pod 无法启动，因为它无法从您的私有注册表中提取映像。

Answer 2

@Philip Welz 的回答帮助我找到了以下解决方案

SecretProviderClass 示例 yaml

# This is a SecretProviderClass example using system-assigned identity to access your key vault
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: azure-kvname-system-harbor
spec:
  provider: azure
  secretObjects:
    - secretName: dockerconfig
      type: kubernetes.io/dockerconfigjson
      data:
        - objectName: harborcredentialvault
          key: .dockerconfigjson
  parameters:
    usePodIdentity: "false"
    useVMManagedIdentity: "true"    # Set to true for using managed identity
    userAssignedIdentityID: ""      # If empty, then defaults to use the system assigned identity on the VM
    keyvaultName: "<Keyvault name>"
    cloudName: ""                   # [OPTIONAL for Azure] if not provided, the Azure environment defaults to AzurePublicCloud
    objects:  |
      array:
        - |
          objectName: harborcredentialvault
          objectType: secret        # object types: secret, key, or cert
          objectVersion: ""         # [OPTIONAL] object versions, default to latest if empty
    tenantId: "<tenant ID>"           # The tenant ID of the key vault

部署示例 yaml 文件

spec:
  containers:
  - name: demo
    image: demo
    volumeMounts:
    - name: secrets-store-inline
      mountPath: "/mnt/secrets-store"
      readOnly: true
  imagePullSecrets:
    - name: dockerconfig
  volumes:
  - name: secrets-store-inline
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: "azure-kvname-system-harbor"

在 Keyvault 中创建 Secret，确保值应低于 JSON 格式

Key: harborcredentialvault
Value: {
"auths": {
"dockerwebsite.com": {
"username": "username",
"password": "password"
}
}

}