livenessprobe 因 EOF 失败（nginx 容器）

Question

我有一个运行 nginx 的容器，它在 pod id 的端口 443 上进行侦听。它自己运行良好；但是，如果我指定一个活性探测，那么探测将失败

5m54s       Warning   Unhealthy          Pod           Liveness probe failed: Get https://192.168.2.243:443/: EOF

谁能指出我做错了什么？谢谢。

当它在没有活性探针的情况下运行时：

root@ip-192-168-2-243:/etc/nginx# netstat -tupln | grep 443
tcp        0      0 192.168.2.243:1443      0.0.0.0:*               LISTEN      -
tcp        0      0 192.168.2.243:443       0.0.0.0:*               LISTEN      7/nginx: master pro

root@ip-192-168-2-243:/# telnet 192.168.2.243 443
Trying 192.168.2.243...
Connected to 192.168.2.243.
Escape character is '^]'.
^]
telnet> quit
Connection closed.

root@ip-192-168-2-243:/# curl https://192.168.2.243
curl: (77) error setting certificate verify locations:
  CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs

探测声明：

livenessProbe:
  initialDelaySeconds: 10
  timeoutSeconds: 4
  failureThreshold: 3
  httpGet:
    scheme: HTTPS
    port: 443

Nginx 拆分客户端声明：

split_clients "${remote_addr}AAA" $localips {
       *                 192.168.2.243;
}

dataplane/kubelet.service-ip：

事件：

skwok-mbp:kubernetes skwok$ kubectl get event -w
LAST SEEN   TYPE     REASON             OBJECT              MESSAGE
7s          Normal   SuccessfulDelete   statefulset/mnsvr   delete Pod mnsvr-0 in StatefulSet mnsvr successful
0s          Normal   Killing            pod/mnsvr-0         Killing container with id docker://mnsvr-proxy:Need to kill Pod
0s          Normal   Killing            pod/mnsvr-0         Killing container with id docker://mnsvr-node0:Need to kill Pod
0s          Normal   Killing            pod/mnsvr-0         Killing container with id docker://mnsvr-node1:Need to kill Pod
0s          Normal   SuccessfulCreate   statefulset/mnsvr   create Pod mnsvr-0 in StatefulSet mnsvr successful
0s          Normal   Scheduled          pod/mnsvr-0         Successfully assigned staging/mnsvr-0 to ip-192-168-2-243.us-west-2.compute.internal
0s          Normal   Pulled             pod/mnsvr-0         Container image "171421899218.dkr.ecr.us-west-2.amazonaws.com/mnsvr-proxy:0.96" already present on machine
0s          Normal   Created            pod/mnsvr-0         Created container
0s          Normal   Started            pod/mnsvr-0         Started container
0s          Normal   Pulled             pod/mnsvr-0         Container image "171421899218.dkr.ecr.us-west-2.amazonaws.com/mnsvr:1.1" already present on machine
0s          Normal   Created            pod/mnsvr-0         Created container
0s          Normal   Started            pod/mnsvr-0         Started container
0s          Normal   Pulled             pod/mnsvr-0         Container image "171421899218.dkr.ecr.us-west-2.amazonaws.com/mnsvr:1.1" already present on machine
0s          Normal   Created            pod/mnsvr-0         Created container
0s          Normal   Started            pod/mnsvr-0         Started container
0s          Warning   Unhealthy          pod/mnsvr-0         Liveness probe failed: Get https://192.168.2.243:443/: EOF
0s          Warning   Unhealthy          pod/mnsvr-0         Liveness probe failed: Get https://192.168.2.243:443/: EOF
0s          Warning   Unhealthy          pod/mnsvr-0         Liveness probe failed: Get https://192.168.2.243:443/: EOF
0s          Normal    Killing            pod/mnsvr-0         Killing container with id docker://mnsvr-proxy:Container failed liveness probe.. Container will be killed and recreated.
0s          Normal    Pulled             pod/mnsvr-0         Container image "171421899218.dkr.ecr.us-west-2.amazonaws.com/mnsvr-proxy:0.96" already present on machine
0s          Normal    Created            pod/mnsvr-0         Created container
0s          Normal    Started            pod/mnsvr-0         Started container
0s          Warning   Unhealthy          pod/mnsvr-0         Liveness probe failed: Get https://192.168.2.243:443/: EOF
0s          Warning   Unhealthy          pod/mnsvr-0         Liveness probe failed: Get https://192.168.2.243:443/: EOF
0s          Warning   Unhealthy          pod/mnsvr-0         Liveness probe failed: Get https://192.168.2.243:443/: EOF
0s          Normal    Killing            pod/mnsvr-0         Killing container with id docker://mnsvr-proxy:Container failed liveness probe.. Container will be killed and recreated.
0s          Normal    Pulled             pod/mnsvr-0         Container image "171421899218.dkr.ecr.us-west-2.amazonaws.com/mnsvr-proxy:0.96" already present on machine
0s          Normal    Created            pod/mnsvr-0         Created container
0s          Normal    Started            pod/mnsvr-0         Started container
0s          Warning   Unhealthy          pod/mnsvr-0         Liveness probe failed: Get https://192.168.2.243:443/: EOF
0s          Warning   Unhealthy          pod/mnsvr-0         Liveness probe failed: Get https://192.168.2.243:443/: EOF
0s          Warning   BackOff            pod/mnsvr-0         Back-off restarting failed container

主机/host.messages-ip-192-168-2-243：

应用程序/mnsvr 代理：

Answer 1

我认为 EOF 是 TLS 握手问题的症状。我目前看到的也是。

某些版本的 curl 可以产生类似的结果。 curl 的一种解决方法似乎是使用 --tls-max 1.2。

我目前的怀疑是客户端（探针）正在尝试与服务器协商 TLS 1.3，但失败了（可能是由于密码）。我正在尝试查看我们是否可以将 k8s 探针配置为使用 TLS 1.2。或者，我们可以在服务器端关闭 TLS 1.3。在你的情况下，那是在 nginx 上。就我而言，我有一个 JDK 11.0.6 的 jetty 9.4 服务器。

另一个选择可能是升级 k8s。我们似乎在 k8s v1.15 集群中看到了这一点，但在 k8s v1.16.2 集群中却没有。但我不确定这是因为 k8s 版本还是底层操作系统库（在我的例子中是 CentOS 7）。

Answer 2

Kubernetes 有两种单独的方法来跟踪 Pod 的运行状况，一种是在部署期间，另一种是在部署之后。 LivenessProbe 是导致 Kubernetes 用新的 Pod 替换失败的 Pod 的原因，但它在应用程序部署期间绝对没有影响。另一方面，就绪探测是 Kubernetes 用来确定 Pod 是否成功启动的。

因此，当您的容器成功运行时，您必须定义 readinessProbe。

有时，应用程序暂时无法提供流量。例如，应用程序可能需要在启动过程中加载大数据或配置文件，或者在启动后依赖外部服务。在这种情况下，您不想杀死应用程序，但也不想向它发送请求。 Kubernetes 提供就绪探针来检测和缓解这些情况。带有容器报告它们尚未准备好的 pod 不会通过 Kubernetes 服务接收流量。

描述探针的官方 Kubernetes 文档：kubernetes-probes。

这是一篇有用的文章：kubernetes-liveness-and-readiness-probes。