启用 stackdriver 监控会导致 metadata-agent pod 崩溃

Question

启用监控时创建的 pod 列表：

➜ kubectl get pods --namespace=kube-system | grep metadata-agent
NAME                                                READY   STATUS    RESTARTS   AGE
metadata-agent-cluster-level-579ffb7c5f-vm8q8       1/1     Running   908        3d
metadata-agent-gdnb6                                1/1     Running   908        3d
metadata-agent-q7vct                                1/1     Running   885        3d
metadata-agent-rcfl8                                1/1     Running   907        3d
metadata-agent-vvtss                                1/1     Running   908        3d
metadata-agent-zvz6f                                1/1     Running   816        3d

来自元数据代理的日志：

➜ kubectl logs pods/metadata-agent-gdnb6  --namespace=kube-system
I0130 10:32:38 7eff97c7f740 updater.cc:40 Not starting DockerUpdater
I0130 10:32:38 7eff97c7f740 kubernetes.cc:1324 Watching for node-level metadata
I0130 10:32:38 7eff94e58700 kubernetes.cc:1163 Watch thread (pods) started for node gke-rain-rain-node-pool-16891a38-p99s
I0130 10:32:38 7eff8effd700 kubernetes.cc:1203 Watch thread (node) started for node gke-rain-rain-node-pool-16891a38-p99s
I0130 10:32:38 7eff7ffff700 reporter.cc:46 Metadata reporter started
I0130 10:32:41 7eff7ffff700 environment.cc:270 No credentials found at /etc/google/auth/application_default_credentials.json
I0130 10:32:41 7eff7ffff700 environment.cc:146 Got project id from metadata server: 11111111
I0130 10:32:41 7eff7ffff700 oauth2.cc:283 Getting auth token from metadata server
E0130 10:32:41 7eff7ffff700 reporter.cc:64 Metadata request unsuccessful: Server responded with 'Forbidden' (403): Transport endpoint is not connected
E0130 10:33:41 7eff7ffff700 reporter.cc:64 Metadata request unsuccessful: Server responded with 'Forbidden' (403): Transport endpoint is not connected
E0130 10:34:41 7eff7ffff700 reporter.cc:64 Metadata request unsuccessful: Server responded with 'Forbidden' (403): Transport endpoint is not connected
E0130 10:35:41 7eff7ffff700 reporter.cc:64 Metadata request unsuccessful: Server responded with 'Forbidden' (403): Transport endpoint is not connected
E0130 10:36:41 7eff7ffff700 reporter.cc:64 Metadata request unsuccessful: Server responded with 'Forbidden' (403): Transport endpoint is not connected
E0130 10:37:41 7eff7ffff700 reporter.cc:64 Metadata request unsuccessful: Server responded with 'Forbidden' (403): Transport endpoint is not connected

元数据：

• GKE 1.11.6-gke.3
• 通过云控制台启用 stackdriver 监控。

注意：

• 只有在创建集群后启用 stackdriver 监控时才会发生这种情况（不作为集群创建的一部分）。

Answer 1

Google Kubernetes Engine 默认使用 fluentd 作为日志代理，在进行研究时，我的想法是您进行了手动安装，根据 Kubernetes 监控documentation：

注意：不建议在 GKE 上手动安装。提供手动安装是为了避免在安装 Stackdriver Kubernetes Monitoring 的托管支持时出现临时问题。这个问题已经解决了。请参阅安装 Stackdriver Kubernetes Monitoring 以安装或升级到最新版本。

我的建议是使用默认代理来避免此类问题。