EKS kube-system 部署 CrashLoopBackOff

Question

我正在尝试将 Kube 状态指标部署到运行 Kubernetes v1.14 的 EKS 集群 (eks.4) 中的 kube-system 命名空间中。

Kubernetes 连接

provider "kubernetes" {
  host                   = var.cluster.endpoint
  token                  = data.aws_eks_cluster_auth.cluster_auth.token
  cluster_ca_certificate = base64decode(var.cluster.certificate)
  load_config_file       = true
}

部署清单（作为 .tf）

resource "kubernetes_deployment" "kube_state_metrics" {
  metadata {
    name      = "kube-state-metrics"
    namespace = "kube-system"

    labels = {
      k8s-app = "kube-state-metrics"
    }
  }

  spec {
    replicas = 1

    selector {
      match_labels = {
        k8s-app = "kube-state-metrics"
      }
    }

    template {
      metadata {
        labels = {
          k8s-app = "kube-state-metrics"
        }
      }

      spec {
        container {
          name  = "kube-state-metrics"
          image = "quay.io/coreos/kube-state-metrics:v1.7.2"

          port {
            name           = "http-metrics"
            container_port = 8080
          }

          port {
            name           = "telemetry"
            container_port = 8081
          }

          liveness_probe {
            http_get {
              path = "/healthz"
              port = "8080"
            }

            initial_delay_seconds = 5
            timeout_seconds       = 5
          }

          readiness_probe {
            http_get {
              path = "/"
              port = "8080"
            }

            initial_delay_seconds = 5
            timeout_seconds       = 5
          }
        }

        service_account_name = "kube-state-metrics"
      }
    }
  }
}

我还从 https://github.com/kubernetes/kube-state-metrics/tree/master/kubernetes 部署了所有必需的 RBAC 清单 - 为简洁起见，此处已编辑。

当我在上面的部署中运行terraform apply 时，Terraform 输出如下： kubernetes_deployment.kube_state_metrics: Still creating... [6m50s elapsed]

最终在 10m 处超时。

这是kube-state-metrics pod 的日志输出

I0910 23:41:19.412496       1 main.go:140] metric white-blacklisting: blacklisting the following items:
W0910 23:41:19.412535       1 client_config.go:541] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
W0910 23:41:19.412565       1 client_config.go:546] error creating inClusterConfig, falling back to default config: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
F0910 23:41:19.412782       1 main.go:148] Failed to create client: invalid configuration: no configuration has been provided

Answer 1

我没有尝试使用 terraform。

我刚刚在本地运行了这个部署，我遇到了同样的错误。

请在本地运行您的部署以查看您的部署和 pod 的状态。

I0910 13:25:49.632847       1 main.go:140] metric white-blacklisting: blacklisting the following items:
W0910 13:25:49.632871       1 client_config.go:541] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.

 and finally:

I0910 13:25:49.634748       1 main.go:185] Testing communication with server
I0910 13:25:49.650994       1 main.go:190] Running with Kubernetes cluster version: v1.12+. git version: v1.12.8-gke.10. git tree state: clean. commit: f53039cc1e5295eed20969a4f10fb6ad99461e37. platform: linux/amd64
I0910 13:25:49.651028       1 main.go:192] Communication with server successful
I0910 13:25:49.651598       1 builder.go:126] Active collectors: certificatesigningrequests,configmaps,cronjobs,daemonsets,deployments,endpoints,horizontalpodautoscalers,ingresses,jobs,limitranges,namespaces,nodes,persistentvolumeclaims,persistentvolumes,poddisruptionbudgets,pods,replicasets,replicationcontrollers,resourcequotas,secrets,services,statefulsets,storageclasses
I0910 13:25:49.651607       1 main.go:226] Starting metrics server: 0.0.0.0:8080
I0910 13:25:49.652149       1 main.go:201] Starting kube-state-metrics self metrics server: 0.0.0.0:8081

验证：

Connected to kube-state-metrics (xx.xx.xx.xx) port 8080 (#0)
 GET /metrics HTTP/1.1
 Host: kube-state-metrics:8080
 User-Agent: curl/7.58.0
 Accept: */*

HTTP/1.1 200 OK
 Content-Type: text/plain; version=0.0.4
 Date: Tue, 10 Sep 2019 13:39:52 GMT
 Transfer-Encoding: chunked

 [49027 bytes data]
 HELP kube_certificatesigningrequest_labels Kubernetes labels converted to 
Prometheus labels.

如果您正在构建自己的图像，请关注gihtub 和docs 上的问题

更新：只是为了澄清。

正如我在回答中提到的那样。我没有尝试使用 terraform，但似乎第一个问题只描述了一个问题W0910 13:25:49.632871 1 client_config.go:541] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.

所以我建议在本地运行此部署并验证日志中的所有错误。后来发现automount_service_account_token有问题。这个重要的错误不适用于原始问题。 所以请关注terraform issues on github如何解决这个问题

根据description on github：

我花了几个小时试图弄清楚为什么服务帐户和部署在 Terraform 中不起作用，但在 kubectl 中没有问题 - 这是 AutomountServiceAccountToken 在部署资源中被硬编码为 False。

至少应在资源的 Terraform 文档中记录这一点，并指出该资源的行为不像 kubectl 那样。

我希望它能解释这个问题。

Answer 2

将以下内容添加到spec 已成功部署。

automount_service_account_token = true

为了后代：

resource "kubernetes_deployment" "kube_state_metrics" {
  metadata {
    name      = "kube-state-metrics"
    namespace = "kube-system"

    labels = {
      k8s-app = "kube-state-metrics"
    }
  }

  spec {
    replicas = 1

    selector {
      match_labels = {
        k8s-app = "kube-state-metrics"
      }
    }

    template {
      metadata {
        labels = {
          k8s-app = "kube-state-metrics"
        }
      }

      spec {
        automount_service_account_token = true
        container {
          name  = "kube-state-metrics"
          image = "quay.io/coreos/kube-state-metrics:v1.7.2"

          port {
            name           = "http-metrics"
            container_port = 8080
          }

          port {
            name           = "telemetry"
            container_port = 8081
          }

          liveness_probe {
            http_get {
              path = "/healthz"
              port = "8080"
            }

            initial_delay_seconds = 5
            timeout_seconds       = 5
          }

          readiness_probe {
            http_get {
              path = "/"
              port = "8080"
            }

            initial_delay_seconds = 5
            timeout_seconds       = 5
          }
        }

        service_account_name = "kube-state-metrics"
      }
    }
  }
}