kube-scheduler Liveness probe failed: Get http://127.0.0.1:10251/healthz: dial tcp 127.0.0.1:10251: connect: connection denied答案

【问题标题】：kube-scheduler Liveness probe failed: Get http://127.0.0.1:10251/healthz: dial tcp 127.0.0.1:10251: connect: connection refusedkube-scheduler Liveness probe failed: Get http://127.0.0.1:10251/healthz: dial tcp 127.0.0.1:10251: connect: connection denied
【发布时间】：2019-05-29 03:54:51
【问题描述】：

所以我有这个不健康的集群在数据中心部分工作。这可能是我第 10 次按照以下说明进行重建：https://kubernetes.io/docs/setup/independent/high-availability/

我可以将一些 pod 应用到这个集群，它似乎可以工作，但最终它开始变慢并崩溃，如下所示。这是调度程序清单：

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    component: kube-scheduler
    tier: control-plane
  name: kube-scheduler
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-scheduler
    - --bind-address=127.0.0.1
    - --kubeconfig=/etc/kubernetes/scheduler.conf
    - --authentication-kubeconfig=/etc/kubernetes/scheduler.conf
    - --authorization-kubeconfig=/etc/kubernetes/scheduler.conf
    - --leader-elect=true
    image: k8s.gcr.io/kube-scheduler:v1.14.2
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 8
      httpGet:
        host: 127.0.0.1
        path: /healthz
        port: 10251
        scheme: HTTP
      initialDelaySeconds: 15
      timeoutSeconds: 15
    name: kube-scheduler
    resources:
      requests:
        cpu: 100m
    volumeMounts:
    - mountPath: /etc/kubernetes/scheduler.conf
      name: kubeconfig
      readOnly: true
  hostNetwork: true
  priorityClassName: system-cluster-critical
  volumes:
  - hostPath:
      path: /etc/kubernetes/scheduler.conf
      type: FileOrCreate
    name: kubeconfig
status: {}

$ kubectl -n kube-system 获取 pod

NAME                                       READY   STATUS             RESTARTS   AGE
coredns-fb8b8dccf-42psn                    1/1     Running            9          88m
coredns-fb8b8dccf-x9mlt                    1/1     Running            11         88m
docker-registry-dqvzb                      1/1     Running            1          2d6h
kube-apiserver-kube-apiserver-1            1/1     Running            44         2d8h
kube-apiserver-kube-apiserver-2            1/1     Running            34         2d7h
kube-controller-manager-kube-apiserver-1   1/1     Running            198        2d2h
kube-controller-manager-kube-apiserver-2   0/1     CrashLoopBackOff   170        2d7h
kube-flannel-ds-amd64-4mbfk                1/1     Running            1          2d7h
kube-flannel-ds-amd64-55hc7                1/1     Running            1          2d8h
kube-flannel-ds-amd64-fvwmf                1/1     Running            1          2d7h
kube-flannel-ds-amd64-ht5wm                1/1     Running            3          2d7h
kube-flannel-ds-amd64-rjt9l                1/1     Running            4          2d8h
kube-flannel-ds-amd64-wpmkj                1/1     Running            1          2d7h
kube-proxy-2n64d                           1/1     Running            3          2d7h
kube-proxy-2pq2g                           1/1     Running            1          2d7h
kube-proxy-5fbms                           1/1     Running            2          2d8h
kube-proxy-g8gmn                           1/1     Running            1          2d7h
kube-proxy-wrdrj                           1/1     Running            1          2d8h
kube-proxy-wz6gv                           1/1     Running            1          2d7h
kube-scheduler-kube-apiserver-1            0/1     CrashLoopBackOff   198        2d2h
kube-scheduler-kube-apiserver-2            1/1     Running            5          18m
nginx-ingress-controller-dz8fm             1/1     Running            3          2d4h
nginx-ingress-controller-sdsgg             1/1     Running            3          2d4h
nginx-ingress-controller-sfrgb             1/1     Running            1          2d4h

$ kubectl -n kube-system 描述 pod kube-scheduler-kube-apiserver-1

Containers:
  kube-scheduler:
    Container ID:  docker://c04f3c9061cafef8749b2018cd66e6865d102f67c4d13bdd250d0b4656d5f220
    Image:         k8s.gcr.io/kube-scheduler:v1.14.2
    Image ID:      docker-pullable://k8s.gcr.io/kube-scheduler@sha256:052e0322b8a2b22819ab0385089f202555c4099493d1bd33205a34753494d2c2
    Port:          <none>
    Host Port:     <none>
    Command:
      kube-scheduler
      --bind-address=127.0.0.1
      --kubeconfig=/etc/kubernetes/scheduler.conf
      --authentication-kubeconfig=/etc/kubernetes/scheduler.conf
      --authorization-kubeconfig=/etc/kubernetes/scheduler.conf
      --leader-elect=true
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Tue, 28 May 2019 23:16:50 -0400
      Finished:     Tue, 28 May 2019 23:19:56 -0400
    Ready:          False
    Restart Count:  195
    Requests:
      cpu:        100m
    Liveness:     http-get http://127.0.0.1:10251/healthz delay=15s timeout=15s period=10s #success=1 #failure=8
    Environment:  <none>
    Mounts:
      /etc/kubernetes/scheduler.conf from kubeconfig (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  kubeconfig:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/kubernetes/scheduler.conf
    HostPathType:  FileOrCreate
QoS Class:         Burstable
Node-Selectors:    <none>
Tolerations:       :NoExecute
Events:
  Type     Reason          Age                    From                       Message
  ----     ------          ----                   ----                       -------
  Normal   Created         4h56m (x104 over 37h)  kubelet, kube-apiserver-1  Created container kube-scheduler
  Normal   Started         4h56m (x104 over 37h)  kubelet, kube-apiserver-1  Started container kube-scheduler
  Warning  Unhealthy       137m (x71 over 34h)    kubelet, kube-apiserver-1  Liveness probe failed: Get http://127.0.0.1:10251/healthz: dial tcp 127.0.0.1:10251: connect: connection refused
  Normal   Pulled          132m (x129 over 37h)   kubelet, kube-apiserver-1  Container image "k8s.gcr.io/kube-scheduler:v1.14.2" already present on machine
  Warning  BackOff         128m (x1129 over 34h)  kubelet, kube-apiserver-1  Back-off restarting failed container
  Normal   SandboxChanged  80m                    kubelet, kube-apiserver-1  Pod sandbox changed, it will be killed and re-created.
  Warning  Failed          76m                    kubelet, kube-apiserver-1  Error: context deadline exceeded
  Normal   Pulled          36m (x7 over 78m)      kubelet, kube-apiserver-1  Container image "k8s.gcr.io/kube-scheduler:v1.14.2" already present on machine
  Normal   Started         36m (x6 over 74m)      kubelet, kube-apiserver-1  Started container kube-scheduler
  Normal   Created         32m (x7 over 74m)      kubelet, kube-apiserver-1  Created container kube-scheduler
  Warning  Unhealthy       20m (x9 over 40m)      kubelet, kube-apiserver-1  Liveness probe failed: Get http://127.0.0.1:10251/healthz: dial tcp 127.0.0.1:10251: connect: connection refused
  Warning  BackOff         2m56s (x85 over 69m)   kubelet, kube-apiserver-1  Back-off restarting failed container

我觉得我忽略了一个简单的选项或配置，但我找不到它，在处理这个问题和阅读文档数天后，我束手无策。

负载平衡器是一个 TCP 负载平衡器，似乎可以按预期工作，因为我可以从桌面查询集群。

目前绝对欢迎任何建议或故障排除提示。

谢谢。

【问题讨论】：

kubectl logs kube-scheduler-kube-apiserver-1 -n kube-system 的输出是什么？
由于我不能在这里发表长篇评论，我创建了这个要点：gist.github.com/dannymk/6000fe74e44105a770bb4087d9a5c082
为什么有多个控制器管理器和调度 pod？与 kube-scheduler-kube-apiserver-2 相同的问题还是仅与 apiserver-1 相关？
我有一个 api 集群，现在只有两个成员，但是一旦我解决了这个问题，我会添加第三个。我们想在 HA 模式下运行 Kubernetes。这并不重要，因为即使我只有一台主服务器也会发生此问题。见：kubernetes.io/docs/setup/independent/setup-ha-etcd-with-kubeadm

标签： kubernetes kubeadm

【解决方案1】：

我们的配置存在的问题是，一位好心的技术人员决定取消 kubernetes 主防火墙上的一条规则，该规则阻止主防火墙环回它需要探测的端口。这导致了各种奇怪的问题和误诊问题，这绝对是错误的方向。在我们允许服务器上的所有端口后，Kubernetes 恢复了正常行为。

【讨论】：