【问题标题】:Cert-Manager k8s, Renewing certificate processCert-Manager k8s,更新证书流程
【发布时间】:2021-09-20 10:17:54
【问题描述】:

我已经在 k8s 集群上安装了证书管理器:

helm install cert-manager jetstack/cert-manager --namespace cert-manager --create-namespace --version v1.5.3 --set installCRDs=true

我的目标是在同一名称空间中运行的微服务之间进行 mtls 通信。

为此,我创建了一个 ca issuer .i.e..

kubectl get issuer -n sandbox -o yaml
apiVersion: v1
items:
- apiVersion: cert-manager.io/v1
  kind: Issuer
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"cert-manager.io/v1","kind":"Issuer","metadata":{"annotations":{},"name":"ca-issuer","namespace":"sandbox"},"spec":{"ca":{"secretName":"tls-internal-ca"}}}
    creationTimestamp: "2021-09-16T17:24:58Z"
    generation: 1
    managedFields:
    - apiVersion: cert-manager.io/v1
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            .: {}
            f:kubectl.kubernetes.io/last-applied-configuration: {}
        f:spec:
          .: {}
          f:ca:
            .: {}
            f:secretName: {}
      manager: HashiCorp
      operation: Update
      time: "2021-09-16T17:24:58Z"
    - apiVersion: cert-manager.io/v1
      fieldsType: FieldsV1
      fieldsV1:
        f:status:
          .: {}
          f:conditions: {}
      manager: controller
      operation: Update
      time: "2021-09-16T17:24:58Z"
    name: ca-issuer
    namespace: sandbox
    resourceVersion: "3895820"
    selfLink: /apis/cert-manager.io/v1/namespaces/sandbox/issuers/ca-issuer
    uid: 90f0c811-b78d-4346-bb57-68bf607ee468
  spec:
    ca:
      secretName: tls-internal-ca
  status:
    conditions:
      message: Signing CA verified
      observedGeneration: 1
      reason: KeyPairVerified
      status: "True"
      type: Ready
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

使用这个 ca 颁发者,我为我的两个微服务创建了证书。即

kubectl get certificate -n sandbox      
NAME                       READY         SECRET             Age
service1-certificate        True    service1-certificate     3d     
service2-certificate        True    service2-certificate     2d23h

配置为

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  annotations:
    meta.helm.sh/release-name: service1
    meta.helm.sh/release-namespace: sandbox
  creationTimestamp: "2021-09-17T10:20:21Z"
  generation: 1
  labels:
    app.kubernetes.io/managed-by: Helm
  managedFields:
  - apiVersion: cert-manager.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:meta.helm.sh/release-name: {}
          f:meta.helm.sh/release-namespace: {}
        f:labels:
          .: {}
          f:app.kubernetes.io/managed-by: {}
      f:spec:
        .: {}
        f:commonName: {}
        f:dnsNames: {}
        f:duration: {}
        f:issuerRef:
          .: {}
          f:kind: {}
          f:name: {}
        f:renewBefore: {}
        f:secretName: {}
        f:subject:
          .: {}
          f:organizations: {}
        f:usages: {}
    manager: Go-http-client
    operation: Update
    time: "2021-09-17T10:20:21Z"
  - apiVersion: cert-manager.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:spec:
        f:privateKey: {}
      f:status:
        .: {}
        f:conditions: {}
        f:notAfter: {}
        f:notBefore: {}
        f:renewalTime: {}
        f:revision: {}
    manager: controller
    operation: Update
    time: "2021-09-20T05:14:12Z"
  name: service1-certificate
  namespace: sandbox
  resourceVersion: "5177051"
  selfLink: /apis/cert-manager.io/v1/namespaces/sandbox/certificates/service1-certificate
  uid: 0cf1ea65-92a1-4b03-944e-b847de2c80d9
spec:
  commonName: example.com
  dnsNames:
  - service1
  duration: 24h0m0s
  issuerRef:
    kind: Issuer
    name: ca-issuer
  renewBefore: 12h0m0s
  secretName: service1-certificate
  subject:
    organizations:
    - myorg
  usages:
  - client auth
  - server auth
status:
  conditions:
  - lastTransitionTime: "2021-09-20T05:14:13Z"
    message: Certificate is up to date and has not expired
    observedGeneration: 1
    reason: Ready
    status: "True"
    type: Ready
  notAfter: "2021-09-21T05:14:13Z"
  notBefore: "2021-09-20T05:14:13Z"
  renewalTime: "2021-09-20T17:14:13Z"
  revision: 5

现在您可以在配置中看到我已配置为在 12 小时内更新它们。但是,通过此自定义证书资源创建的机密仍然存在两天(它是创建的第一天)。我在想这个 tls 秘密每天都会通过证书管理器更新)。即

kubectl get secrets service1-certificate service2-certificate -n sandbox -o wide
NAME                        TYPE                DATA   AGE
service1-certificate        kubernetes.io/tls   3      2d23h
service2-certificate        kubernetes.io/tls   3      3d1h

我的理解有问题吗?在certmangager pod 日志中,我确实看到一些关于更新 .i.e. 的错误。

I0920 05:14:04.649158       1 trigger_controller.go:181] cert-manager/controller/certificates-trigger "msg"="Certificate must be re-issued" "key”=“sandbox/service1-certificate" "message"="Renewing certificate as renewal was scheduled at 2021-09-19 08:24:13 +0000 UTC" "reason"="Renewing"
    
I0920 05:14:04.649235       1 conditions.go:201] Setting lastTransitionTime for Certificate “service1-certificate" condition "Issuing" to 2021-09-20 05:14:04.649227766 +0000 UTC m=+87949.327215532
    
I0920 05:14:04.652174       1 trigger_controller.go:181] cert-manager/controller/certificates-trigger "msg"="Certificate must be re-issued" "key"="sandbox/service2 "message"="Renewing certificate as renewal was scheduled at 2021-09-19 10:20:22 +0000 UTC" "reason"="Renewing"
    
I0920 05:14:04.652231       1 conditions.go:201] Setting lastTransitionTime for Certificate “service2-certificate" condition "Issuing" to 2021-09-20 05:14:04.652224302 +0000 UTC m=+87949.330212052
    
I0920 05:14:04.671111       1 conditions.go:190] Found status change for Certificate “service2-certificate" condition "Ready": "True" -> "False"; setting lastTransitionTime to 2021-09-20 05:14:04.671094596 +0000 UTC m=+87949.349082328
I0920 05:14:04.671344       1 conditions.go:190] Found status change for Certificate “service1-certificate" condition "Ready": "True" -> "False"; setting lastTransitionTime to 2021-09-20 05:14:04.671332206 +0000 UTC m=+87949.349319948
    
    
I0920 05:14:12.703039       1 controller.go:161] cert-manager/controller/certificates-readiness "msg"="re-queuing item due to optimistic locking on resource" "key”=“sandbox/service2-certificate" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \”service2-certificate\": the object has been modified; please apply your changes to the latest version and try again"
    
    
I0920 05:14:12.703896       1 conditions.go:190] Found status change for Certificate “service2-certificate" condition "Ready": "True" -> "False"; setting lastTransitionTime to 2021-09-20 05:14:12.7038803 +0000 UTC m=+87957.381868045
    
    
I0920 05:14:12.749502       1 controller.go:161] cert-manager/controller/certificates-readiness "msg"="re-queuing item due to optimistic locking on resource" "key”=“sandbox/service1-certificate" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \”service1-certificate\": the object has been modified; please apply your changes to the latest version and try again"
    
    
I0920 05:14:12.750096       1 conditions.go:190] Found status change for Certificate “service1-certificate" condition "Ready": "True" -> "False"; setting lastTransitionTime to 2021-09-20 05:14:12.750082572 +0000 UTC m=+87957.428070303
    
I0920 05:14:13.009032       1 controller.go:161] cert-manager/controller/certificates-key-manager "msg"="re-queuing item due to optimistic locking on resource" "key"="sandbox/service1-certificate" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \”service1-certificate\": the object has been modified; please apply your changes to the latest version and try again"
    
I0920 05:14:13.117843       1 controller.go:161] cert-manager/controller/certificates-readiness "msg"="re-queuing item due to optimistic locking on resource" "key”=“sandbox/service2-certificate" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \”service2-certificate\": the object has been modified; please apply your changes to the latest version and try again"
    
I0920 05:14:13.119366       1 conditions.go:190] Found status change for Certificate “service2-certificate" condition "Ready": "True" -> "False"; setting lastTransitionTime to 2021-09-20 05:14:13.119351795 +0000 UTC m=+87957.797339520
    
I0920 05:14:13.122820       1 controller.go:161] cert-manager/controller/certificates-key-manager "msg"="re-queuing item due to optimistic locking on resource" "key”=“sandbox\service2-certificate" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \”service-certificate\": the object has been modified; please apply your changes to the latest version and try again"
    
I0920 05:14:13.123907       1 conditions.go:261] Setting lastTransitionTime for CertificateRequest “service2-certificate-t92qh" condition "Approved" to 2021-09-20 05:14:13.123896104 +0000 UTC m=+87957.801883833
I0920 05:14:13.248082       1 conditions.go:261] Setting lastTransitionTime for CertificateRequest “service1-certificate-p9stz" condition "Approved" to 2021-09-20 05:14:13.248071551 +0000 UTC m=+87957.926059296
I0920 05:14:13.253488       1 conditions.go:261] Setting lastTransitionTime for CertificateRequest “serivce2-certificate-t92qh" condition "Ready" to 2021-09-20 05:14:13.253474153 +0000 UTC m=+87957.931461871
I0920 05:14:13.388001       1 conditions.go:261] Setting lastTransitionTime for CertificateRequest “service1-certificate-p9stz" condition "Ready" to 2021-09-20 05:14:13.387983783 +0000 UTC m=+87958.065971525
    

【问题讨论】:

  • 您使用哪个发行商?我看到有一些与HashiCorp 有关的东西。还有关于日志...见github issue comment所以如果你需要更多细节,他们建议增加详细程度,这个错误本身没有帮助。

标签: kubernetes cert-manager


【解决方案1】:

简答

根据您提供的证书中的日志和详细信息,可以肯定地说它按预期工作。

注意证书中的revision: 5,表示证书已经更新了4次。如果您现在尝试查看,这将是 6 或 7,因为证书每 12 小时更新一次。

日志

真正令人困惑的第一件事是error messages 在cert-manager pod 中。这主要是嘈杂的消息,本身并没有真正的帮助。

在这里Github issue comment 和这里github issue 3667 了解它。

如果确实需要日志,则应通过在 cert-manager 部署中将 args 设置为 --v=5 来增加 verbosity level。要编辑部署运行以下命令:

kubectl edit deploy cert-manager -n cert-manager

如何检查证书/秘密

证书更新时,secret's和certificate的年龄不变,但内容被编辑,例如secret中的resourceVersion和证书中的revision。

以下是检查证书是否已更新的选项:

  1. 通过在更新前后在yaml 中获取秘密来检查这一点:

    kubectl get secret example-certificate -o yaml > secret-before
    

然后在它们之间运行diff。可以看到tls.crt和resourceVersion更新了。

  1. 查看证书revision 和dates 的状态 (我将持续时间设置为尽可能短的1h 和 renewBefore 55m,所以它每 5 分钟更新一次):

     $ kubectl get cert example-cert -o yaml
     notAfter: "2021-09-21T14:05:24Z"
     notBefore: "2021-09-21T13:05:24Z"
     renewalTime: "2021-09-21T13:10:24Z"
     revision: 7
    
  2. 检查部署证书/密钥的命名空间中的事件:

    $ kubectl get events
    117s        Normal   Issuing             certificate/example-cert                The certificate has been successfully issued
    117s        Normal   Reused              certificate/example-cert                Reusing private key stored in existing Secret resource "example-staging-certificate"
    6m57s       Normal   Issuing             certificate/example-cert                Renewing certificate as renewal was scheduled at 2021-09-21 13:00:24 +0000 UTC
    6m57s       Normal   Requested           certificate/example-cert                Created new CertificateRequest resource "example-cert-bs8g6"
    117s        Normal   Issuing             certificate/example-cert                Renewing certificate as renewal was scheduled at 2021-09-21 13:05:24 +0000 UTC
    117s        Normal   Requested           certificate/example-cert                Created new CertificateRequest resource "example-cert-7x8cf" UTC
    
  3. 看certificaterequests:

    $ kubectl get certificaterequests
    NAME                 APPROVED   DENIED   READY   ISSUER      REQUESTOR                                         AGE
    example-cert-2pxdd   True                True    ca-issuer   system:serviceaccount:cert-manager:cert-manager   14m
    example-cert-54zzc   True                True    ca-issuer      system:serviceaccount:cert-manager:cert-manager   4m29s
    example-cert-8vjcm   True                True    ca-issuer      system:serviceaccount:cert-manager:cert-manager   9m29s
    
  4. 查看cert-manager pod 中的日志以查看四个阶段:

    I0921 12:45:24.000726       1 trigger_controller.go:181] cert-manager/controller/certificates-trigger "msg"="Certificate must be re-issued" "key"="default/example-cert" "message"="Renewing certificate as renewal was scheduled at 2021-09-21 12:45:24 +0000 UTC" "reason"="Renewing"
    I0921 12:45:24.000761       1 conditions.go:201] Setting lastTransitionTime for Certificate "example-cert" condition "Issuing" to 2021-09-21 12:45:24.000756621 +0000 UTC m=+72341.194879378
    I0921 12:45:24.120503       1 conditions.go:261] Setting lastTransitionTime for CertificateRequest "example-cert-mxvbm" condition "Approved" to 2021-09-21 12:45:24.12049391 +0000 UTC m=+72341.314616684
    I0921 12:45:24.154092       1 conditions.go:261] Setting lastTransitionTime for CertificateRequest "example-cert-mxvbm" condition "Ready" to 2021-09-21 12:45:24.154081971 +0000 UTC m=+72341.348204734
    

注意

非常重要的是,并非所有issuers 都支持duration 和renewBefore 标志。例如。 letsencrypt 仍然无法使用它并且有 90 个默认天数。

Refence.

【讨论】:

    猜你喜欢
    • 2022-01-10
    • 1970-01-01
    • 2021-07-21
    • 2020-01-14
    • 1970-01-01
    • 2019-06-28
    • 2020-03-24
    • 2020-02-21
    • 2021-10-24
    相关资源
    最近更新 更多