Kubernetes NGINX 入口 TLS 问题答案

【问题标题】：Kubernetes NGINX Ingress TLS issueKubernetes NGINX 入口 TLS 问题
【发布时间】：2018-02-28 16:36:14
【问题描述】：

我在云中部署了 k8s 集群 (VMVare vSphere) - 3 个主节点和 1 个工作节点。然后用 helm 安装 nginx-ingress:

helm install stable/nginx-ingress

部署了几个简单的http-svcpods

将 nginx-controller 服务类型从 LoadBalancer 更改为 NodePort 并添加了 externalIPs（我的主节点的 IP 地址），所以它看起来像：

NAME                                TYPE        CLUSTER-IP      EXTERNAL-IP                              PORT(S)                       AGE
ing-nginx-ingress-controller        NodePort    10.233.15.202   172.16.40.21,172.16.40.22,172.16.40.23   80:31045/TCP,443:31427/TCP    1d
http-svc                            ClusterIP   10.233.13.55                                             80/TCP                        1d

创建的证书和密钥

openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /tmp/tls.key -out /tmp/tls.crt -subj "/CN=<FQDN_HERE>"
kubectl create secret tls secret --key /tmp/tls.key --cert /tmp/tls.crt

并创建入口：

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: some-ingress
  namespace: default
spec:
  tls:
  - hosts:
    - <FQDN_HERE>
    secretName: secret
  rules:
  - host: <FQDN_HERE>
    http:
      paths:
      - backend:
          serviceName: http-svc
          servicePort: 80
        path: /

如果我使用云 DNAT

external_ip:8443 -> master01_ip:443 (e.g. 172.16.40.21:443)

然后我有一个回应：

curl --resolve <FQDN>:8443:<external_ip> https://<FQDN>:8443 -v -k
* Added <FQDN>:8443:<external_ip> to DNS cache
* Rebuilt URL to: https://<FQDN>:8443/
* Hostname <FQDN> was found in DNS cache
*   Trying <external_ip>...
* TCP_NODELAY set
* Connected to <FQDN> (<external_ip>) port 8443 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=<FQDN>
*  start date: Feb 22 10:37:00 2018 GMT
*  expire date: Feb 22 10:37:00 2019 GMT
*  issuer: CN=<FQDN>
*  SSL certificate verify result: self signed certificate (18), continuing anyway.
> GET / HTTP/1.1
> Host: <FQDN>:8443
> User-Agent: curl/7.58.0

但如果我使用负载平衡功能（vEdge 网关）：

                -> 172.16.40.21:443
external_ip:443 -> 172.16.40.22:443
                -> 172.16.40.23:443

有问题：

curl --resolve <FQDN>:443:<external_ip> https://<FQDN> -vvvv -k
* Added <FQDN>:443:<external_ip> to DNS cache
* Rebuilt URL to: https://<FQDN>/
* Hostname <FQDN> was found in DNS cache
*   Trying <external_ip>...
* TCP_NODELAY set
* Connected to <FQDN> (<external_ip>) port 443 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to <FQDN>:443
* Closing connection 0
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to <FQDN>:443

尝试了两个带有 nginx 和自签名证书的独立虚拟机 - 按预期工作。云提供商表示 LB 在 k8s 入口中存在功能和问题。

谢谢！

【问题讨论】：

LB 应该配置为 TCP 模式，而不是 HTTP 模式。 vEdge 网关是否支持此功能？
@Nickolay 谢谢你的回答。是的，vEdge Gateway 支持 TCP 负载平衡，并且我对其进行了配置。它奏效了！但是有关于端口的警告。我不能将 443 用于外部 IP（vEdge 错误或配置错误）。我需要一些时间来询问有关此和测试的支持。

标签： ssl nginx kubernetes

【解决方案1】：

正如@Nickolay 所说，我应该在 TCP 模式下配置 HTTPS 模式 vEdge 网关负载均衡器。但 vEdge Gateway 不允许这样做，因为端口 443 严格绑定到 HTTPS。我通过将健康检查 (!) 配置为 TCP 而不是 SSL 解决了我的问题，现在一切正常。

【讨论】：