Kubernetes 中的 Nginx 控制器：与上游握手 - SSL 握手中的对等关闭连接答案

【问题标题】：Nginx Controller in Kubernetes: Handshaking to upstream - peer closed connection in SSL handshakeKubernetes 中的 Nginx 控制器：与上游握手 - SSL 握手中的对等关闭连接
【发布时间】：2021-02-05 22:02:00
【问题描述】：

在使用 kubeadm 构建的本地环境中。集群由 Master 和 2 个工作节点组成。

失败的原因：

通过服务类型 LoadBalancer 将 Nginx-ingress Controller 暴露给外部，尝试对 Kubernetes 集群进行 TLS 终止。

这里暴露的服务：

NAME            TYPE           CLUSTER-IP      EXTERNAL-IP    PORT(S)                      
nginx-ingress   LoadBalancer   10.101.5.75     192.168.1.82   80:30745/TCP,443:30092/TCP   
web-service     ClusterIP      10.101.26.176   <none>         8080/TCP

它的工作原理：

能够从外部通过 HTTP 端口 80 访问 Web 应用程序。

这里是集群中的 pod：

NAME                              READY   STATUS    RESTARTS   AGE   IP          NODE    
nginx-ingress-7c5588544d-4mw9d    1/1     Running   0          38m   10.44.0.2   node1   
web-deployment-7d84778bc6-52pq7   1/1     Running   1          19h   10.44.0.1   node1   
web-deployment-7d84778bc6-9wwmn   1/1     Running   1          19h   10.36.0.2   node2

TLS 终止测试结果

客户端：

curl -k https://example.com -v
* Rebuilt URL to https://example.com/
*   Trying 192.168.1.82...
* Connected to example.com (192.168.1.82) port 443 (#0)
* found 127 certificates in /etc/ssl/certs/ca-certificates.crt
* found 513 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
*    server certificate verification SKIPPED
*    server certificate status verification SKIPPED
*    common name: example.com (matched)
*    server certificate expiration date OK
*    server certificate activation date OK
*    certificate public key: RSA
*    certificate version: #1
*    subject: CN=example.com
*    start date: Thu, 22 Oct 2020 16:33:49 GMT
*    expire date: Fri, 22 Oct 2021 16:33:49 GMT
*    issuer: CN=My Cert Authority
*    compression: NULL
* ALPN, server accepted to use http/1.1
> GET / HTTP/1.1
> Host: example.com
> User-Agent: curl/7.47.0
> Accept: */*
> 
< HTTP/1.1 502 Bad Gateway
< Server: nginx/1.19.3

服务器端（来自 nginx-ingress pod 的日志）：

GET / HTTP/1.1
Connection: close
Host: example.com
X-Real-IP: 10.32.0.1
X-Forwarded-For: 10.32.0.1
X-Forwarded-Host: example.com
X-Forwarded-Port: 443
X-Forwarded-Proto: https
User-Agent: curl/7.47.0
Accept: */*

2020/10/22 18:38:59 [error] 23#23: *12 peer closed connection in SSL handshake while SSL handshaking  to upstream, client: 10.32.0.1, server: example.com, request: "GET / HTTP/1.1", upstream: "https://10.44.0.1:8081/", host: "example.com"

2020/10/22 18:38:59 [warn] 23#23: *12 upstream server temporarily disabled while SSL handshaking to upstream, client: 10.32.0.1, server: example.com, request: "GET / HTTP/1.1", upstream: "https://10.44.0.1:8081/", host: "example.com"

HTTP/1.1 502 Bad Gateway
Server: nginx/1.19.3

我检查了什么

通过以下链接生成 CA 和服务器证书： https://kubernetes.github.io/ingress-nginx/examples/PREREQUISITES/#tls-certificates

检查了 nginx-ingress Pod 的 /etc/nginx/secrets/default 下的服务器证书和服务器密钥，它们看起来是正确的。这里是资源 VirtualServer 的输出：

NAME         STATE   HOST          IP             PORTS      AGE
vs-example   Valid   example.com   192.168.1.82   [80,443]   121m

虚拟服务器：

apiVersion: k8s.nginx.org/v1
kind: VirtualServer
metadata:
  name: vs-example
  namespace: nginx-ingress
spec:
  host: example.com
  tls:
    secret: example-tls
  upstreams:
  - name: example
    service: web-service
    port: 8080
    tls:
      enable: true
  routes:
  - path: /v2
    action:
      pass: example
  routes:
  - path: /
    action:
      pass: example

秘密

apiVersion: v1
kind: Secret
metadata:
  name: example-tls
  namespace: nginx-ingress
data:
  tls.crt: (..omitted..)
  tls.key: (..omitted..)
type: kubernetes.io/tls

Nginx.conf

这里是从正在运行的 Pod 中提取的 nginx.conf 的摘录：

server {
    # required to support the Websocket protocol in VirtualServer/VirtualServerRoutes
    set $default_connection_header "";
    listen 80 default_server;
    listen 443 ssl default_server;
    ssl_certificate /etc/nginx/secrets/default;
    ssl_certificate_key /etc/nginx/secrets/default;
    server_name _;
    server_tokens "on";

在尝试使用 HTTPS 访问 Web 应用程序时找不到正在发生的事情。

【问题讨论】：

从日志看来，请求被传递到上游"https://10.44.0.1:8081/"，对吗？
10.44.0.1 是分配给 node1 上的 webapplication Pod 的 ip 地址。 8081是web应用容器上的监听端口
而 8081 端口用于 https 流量？我认为这可能是 http 与 https 的问题。
没有。我希望 10.44.0.2 上的 https 确实是入口控制器 Pod
在 VirtualServer 中添加 "ingressClassName: nginx-ingress" 解决了 SSL 问题。 nginx-ingress pod 日志中不再有错误。现在404返回给客户端

标签： kubernetes nginx-config nginx-ingress

【解决方案1】：

VirtualServer 中的几个错误，使用这些更改进行编辑解决了问题：

apiVersion: k8s.nginx.org/v1
kind: VirtualServer
metadata:
  name: vs-example
  namespace: nginx-ingress
spec:
  host: example.com
  tls:
    secret: example-tls

  upstreams:
  - name: example
    service: web-service
    port: 8080
#    tls:            --> this caused the SSL issue in the upstream
#      enable: true

  routes:
  - path: /
    action:
      pass: example
  - path: /v1
    action:
      pass: example

现在可以使用 HTTPS 访问 Web 应用程序

【讨论】：