AppGateway / AKS / AGIC / Let's Encrypt 不工作 - 未找到秘密

Question

描述错误 在此处关注 doco，但它已经过时了，所以不得不猜测...https://docs.microsoft.com/en-us/azure/application-gateway/ingress-controller-letsencrypt-certificate-application-gateway。 应用清单时，它只创建一个 http 侦听器而不是 https。它没有创建证书，并因“未找到秘密”而出错。

agic = mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.4.0
cert-manager = quay.io/jetstack/cert-manager-controller:v1.4.3
aks kubernetes = 1.20.7

复制 请参阅下面的 yaml。如果我调整以使用手动创建的秘密/证书，这很好用。 当我尝试通过 letencrypt 创建时，我在 AGIC pod 上收到“SecretNotFound”错误。

入口控制器详细信息

• kubectl describe pod <ingress controller> 的输出。

Name:         ingress-appgw-deployment-9ffdc54cb-629hg
Namespace:    kube-system
Priority:     0
Node:         aks-default-32636497-vmss000000/10.94.112.4
Start Time:   Wed, 18 Aug 2021 09:59:16 +0100
Labels:       app=ingress-appgw
              kubernetes.azure.com/managedby=aks
              pod-template-hash=9ffdc54cb
Annotations:  checksum/config: 78a4d434072823accba40908961d40922d59acb0000a42182add8d60cde0c9a1
              cluster-autoscaler.kubernetes.io/safe-to-evict: true
              kubernetes.azure.com/metrics-scrape: true
              prometheus.io/path: /metrics
              prometheus.io/port: 8123
              prometheus.io/scrape: true
              resource-id:
                /subscriptions/2bc7b65e-18d6-42ae-afb2-e66d50be6b05/resourceGroups/rg-prd-agwaks-210818-0950/providers/Microsoft.ContainerService/managedC...
Status:       Running
IP:           10.94.112.10
IPs:
  IP:           10.94.112.10
Controlled By:  ReplicaSet/ingress-appgw-deployment-9ffdc54cb
Containers:
  ingress-appgw-container:
    Container ID:   containerd://93e66897c6646d7f6efbf9496646633f13424917a183e85790df0e6c17cc7a91
    Image:          mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.4.0
    Image ID:       sha256:533f2cbe57fa92d27be5939f8ef8dc50537d6e1240502c8c727ac4020545dd34
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Wed, 18 Aug 2021 09:59:18 +0100
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     700m
      memory:  100Mi
    Requests:
      cpu:      100m
      memory:   20Mi
    Liveness:   http-get http://:8123/health/alive delay=15s timeout=1s period=20s #success=1 #failure=3
    Readiness:  http-get http://:8123/health/ready delay=5s timeout=1s period=10s #success=1 #failure=3
    Environment Variables from:
      ingress-appgw-cm  ConfigMap  Optional: false
    Environment:
      AZURE_CLOUD_PROVIDER_LOCATION:  /etc/kubernetes/azure.json
      AGIC_POD_NAME:                  ingress-appgw-deployment-9ffdc54cb-629hg (v1:metadata.name)
      AGIC_POD_NAMESPACE:             kube-system (v1:metadata.namespace)
      KUBERNETES_PORT_443_TCP_ADDR:   aks-prd-agwaks-210818-0950-dns-37f5d052.hcp.northeurope.azmk8s.io
      KUBERNETES_PORT:                tcp://aks-prd-agwaks-210818-0950-dns-37f5d052.hcp.northeurope.azmk8s.io:443
      KUBERNETES_PORT_443_TCP:        tcp://aks-prd-agwaks-210818-0950-dns-37f5d052.hcp.northeurope.azmk8s.io:443
      KUBERNETES_SERVICE_HOST:        aks-prd-agwaks-210818-0950-dns-37f5d052.hcp.northeurope.azmk8s.io
    Mounts:
      /etc/kubernetes/azure.json from cloud-provider-config (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from ingress-appgw-sa-token-cdmtp (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  cloud-provider-config:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/kubernetes/azure.json
    HostPathType:  File
  ingress-appgw-sa-token-cdmtp:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ingress-appgw-sa-token-cdmtp
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:          <none>

• kubectl 日志的输出。

I0818 19:43:07.518122       1 configbuilder.go:221] Invalid custom port configuration (0). Setting listener port to default : 80
I0818 19:43:07.518180       1 requestroutingrules.go:111] Bound basic rule: rr-12754dc8633d87433e25740857ea6708 to listener: fl-12754dc8633d87433e25740857ea6708 ([dev.rhod3rz.com    ], 80) for backend pool pool-default-aspnetapp-dev-80-bp-80 and backend http settings bp-default-aspnetapp-dev-80-80-aspnetapp-dev
I0818 19:43:07.518319       1 event.go:278] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"aspnetapp-dev", UID:"8086e92d-f9a4-4806-afd1-42c24f4f0722", APIVersion:"extensions/v1beta1", ResourceVersion:"90240", FieldPath:""}): type: 'Warning' reason: 'SecretNotFound' Unable to find the secret associated to secretId: [default/dev]

• 清单文件。

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    email: rhod3rz@outlook.com
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: example-issuer-account-key
    solvers:
    - http01:
        ingress:
          class: azure/application-gateway
---
apiVersion: v1
kind: Pod
metadata:
  name: aspnetapp-dev
  labels:
    app: aspnetapp-dev
spec:
  containers:
  - image: "mcr.microsoft.com/dotnet/core/samples:aspnetapp"
    name: aspnetapp-image
    ports:
    - containerPort: 80
      protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  name: aspnetapp-dev
spec:
  selector:
    app: aspnetapp-dev
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: aspnetapp-dev
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway
    cert-manager.io/cluster-issuer: letsencrypt-staging
    cert-manager.io/acme-challenge-type: http01
spec:
  tls:
    - hosts:
      - "dev.rhod3rz.com"
    - secretName: dev
  rules:
  - host: "dev.rhod3rz.com"
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: aspnetapp-dev
            port:
              number: 80

• kubectl 描述入口。

Events:
  Type     Reason          Age                From                       Message
  ----     ------          ----               ----                       -------
  Warning  BadConfig       40m (x2 over 40m)  cert-manager               TLS entry 0 is invalid: TLS entry for hosts [dev.rhod3rz.com] must specify a secretName
  Warning  BadConfig       40m (x2 over 40m)  cert-manager               TLS entry 1 is invalid: secret "dev" for ingress TLS has no hosts specified
  Warning  SecretNotFound  40m (x2 over 40m)  azure/application-gateway  Unable to find the secret associated to secretId: [default/dev]

Answer 1

如果您将集群颁发者与入口一起使用，则必须传递值

privateKeySecretRef:
      name: example-issuer-account-key

入口内仅作为秘密。

如果你会使用命令检查

kubectl get secret

您将在名称空间内看到秘密：example-issuer-account-key

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    email: rhod3rz@outlook.com
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: example-issuer-account-key
    solvers:
    - http01:
        ingress:
          class: azure/application-gateway

入口

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: aspnetapp-dev
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway
    cert-manager.io/cluster-issuer: letsencrypt-staging
    cert-manager.io/acme-challenge-type: http01
spec:
  tls:
    - hosts:
      - "dev.rhod3rz.com"
    - secretName: example-issuer-account-key
  rules:
  - host: "dev.rhod3rz.com"
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: aspnetapp-dev
            port:
              number: 80

还请注意，您正在使用 let's encrypt 的暂存证书，因此您可能会在浏览器中看到 SSL 错误，因为它是暂存证书。

对于生产用例，您必须更改 clusterissuer 中的 server。

暂存：https://acme-staging-v02.api.letsencrypt.org/directory

制作：https://acme-v02.api.letsencrypt.org/directory

Answer 2

令人讨厌的是，它只是一个额外的连字符“-”破坏了一些东西:-(

• 片段

# - secretName: banana # arghh ... the '-' was what was causing it to fail :-(
    secretName: banana # < cert-manager will store the created certificate in this secret.

• 完整的工作清单。

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-issuer
spec:
  acme:
    email: rhod3rz@outlook.com
    server: https://acme-staging-v02.api.letsencrypt.org/directory  # < use this staging issuer when testing to avoid hitting rate limits on prod (50 per week).
    # server: https://acme-v02.api.letsencrypt.org/directory        # < use this prod issuer when ready to go live.
    privateKeySecretRef:
      name: apple
    solvers:
    - http01:
        ingress:
          class: azure/application-gateway
---
apiVersion: v1
kind: Pod
metadata:
  name: aspnetapp-dev
  labels:
    app: aspnetapp-dev
spec:
  containers:
  - image: "mcr.microsoft.com/dotnet/core/samples:aspnetapp"
    name: aspnetapp-image
    ports:
    - containerPort: 80
      protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  name: aspnetapp-dev
spec:
  selector:
    app: aspnetapp-dev
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80
---
# https://cert-manager.io/docs/usage/ingress/
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: aspnetapp-dev
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway # < add annotation indicating the ingress to use.
    cert-manager.io/cluster-issuer: letsencrypt-issuer     # < add annotation indicating the cert issuer to use.
    appgw.ingress.kubernetes.io/ssl-redirect: "true"       # < add annotation to redirect 80 requests to 443.
    # cert-manager.io/acme-challenge-type: http01          # < this is no longer required; works without it.
spec:
  rules:
  - host: dev.rhod3rz.com
    http:
      paths:
      - pathType: Prefix
        path: /
        backend:
          service:
            name: aspnetapp-dev
            port:
              number: 80
  tls: # < placing a host in the TLS config will determine what ends up in the cert's subjectAltNames.
  - hosts:
    - dev.rhod3rz.com
# - secretName: banana # arghh ... the '-' was what was causing it to fail :-(
    secretName: banana # < cert-manager will store the created certificate in this secret.