Kubernetes Haproxy Ingress + Nginx 作为后端 Https答案

【问题标题】：Kubernetes Haproxy Ingress + Nginx as backend HttpsKubernetes Haproxy Ingress + Nginx 作为后端 Https
【发布时间】：2019-05-14 17:20:38
【问题描述】：

我用这个软

Kubernetes 1.11.5
Haproxy：最后一个
Nginx：1.15.7

我用我购买的 comodo CA 制造的证书制作默认/tls-secret

并得到这个错误：

  Error code: SSL_ERROR_RX_RECORD_TOO_LONG

有我的配置 haproxy 入口

---
apiVersion: v1
kind: Namespace
metadata:
  name: ingress-controller
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ingress-controller
  namespace: ingress-controller
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: ingress-controller
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - "extensions"
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - events
    verbs:
      - create
      - patch
  - apiGroups:
      - "extensions"
    resources:
      - ingresses/status
    verbs:
      - update
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: ingress-controller
  namespace: ingress-controller
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - pods
      - secrets
      - namespaces
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - get
      - update
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - create
  - apiGroups:
      - ""
    resources:
      - endpoints
    verbs:
      - get
      - create
      - update
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: ingress-controller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: ingress-controller
subjects:
  - kind: ServiceAccount
    name: ingress-controller
    namespace: ingress-controller
  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: ingress-controller
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: ingress-controller
  namespace: ingress-controller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: ingress-controller
subjects:
  - kind: ServiceAccount
    name: ingress-controller
    namespace: ingress-controller
  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: ingress-controller
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    run: ingress-default-backend
  name: ingress-default-backend
  namespace: ingress-controller
spec:
  selector:
    matchLabels:
      run: ingress-default-backend
  template:
    metadata:
      labels:
        run: ingress-default-backend
    spec:
      containers:
      - name: ingress-default-backend
        image: gcr.io/google_containers/defaultbackend:1.0
        ports:
        - containerPort: 8080
        resources:
          limits:
            cpu: 10m
            memory: 20Mi
---
apiVersion: v1
kind: Service
metadata:
  name: ingress-default-backend
  namespace: ingress-controller
spec:
  ports:
  - port: 8080
  selector:
    run: ingress-default-backend
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: haproxy-ingress
  namespace: ingress-controller
data:
  ssl-options: force-tlsv12

---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  labels:
    run: haproxy-ingress
  name: haproxy-ingress
  namespace: ingress-controller
spec:
  updateStrategy:
    type: RollingUpdate
  selector:
    matchLabels:
      run: haproxy-ingress
  template:
    metadata:
      labels:
        run: haproxy-ingress
    spec:
      hostNetwork: true
      nodeSelector:
        role: edge-router
      serviceAccountName: ingress-controller
      containers:
      - name: haproxy-ingress
        image: quay.io/jcmoraisjr/haproxy-ingress
        args:
        - --default-backend-service=$(POD_NAMESPACE)/ingress-default-backend
        - --default-ssl-certificate=default/tls-secret
        - --configmap=$(POD_NAMESPACE)/haproxy-ingress
        - --sort-backends
        ports:
        - name: http
          containerPort: 80
        - name: https
          containerPort: 443
        - name: stat
          containerPort: 1936
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace

这是我的应用和入口配置

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: meteo
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: meteo
  template:
    metadata:
      labels:
        app: meteo
    spec:
      containers:
      - name: meteo
        image: devprofi/meteo:v39

        ports:
          - containerPort: 443
      imagePullSecrets:
        - name: meteo-secret

---
apiVersion: v1
kind: Service
metadata:
  name: meteo-svc
  namespace: default
spec:
  type: NodePort
  ports:
#    - port: 80
#      targetPort: 80
#      protocol: TCP
#      name: http
    - port: 443
      targetPort: 443
      protocol: TCP
      name: https
  selector:
    app: meteo

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    ingress.kubernetes.io/ssl-passthrough: "true"
    kubernetes.io/ingress.class: "haproxy"
    ingress.kubernetes.io/secure-backends: "true"   
    ingress.kubernetes.io/backend-protocol: "HTTPS"   
  name: meteo-ingress
  namespace: default
spec:
  tls:
  - hosts:
    - meteotravel.ru
    secretName: cafe-secret  # this is another copy of secret made from my buyed cert and key
  rules:
  - host: meteotravel.ru
    http:
      paths:
      - path: /
        backend:
          serviceName: meteo-svc
          servicePort: 443

我尝试这个命令并得到错误

openssl s_client -connect meteotravel.ru:443

-----END CERTIFICATE-----
subject=/OU=Domain Control Validated/OU=PositiveSSL/CN=meteotravel.ru
issuer=/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Domain Validation Secure Server CA
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 6055 bytes and written 312 bytes
Verification error: self signed certificate in certificate chain
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-GCM-SHA256
    Session-ID: BDD996AF8814404E3E385A6FBE49F56CA2668C54FD157FD3FB28F38DB64F771E
    Session-ID-ctx: 
    Master-Key: F8EB4A4DA674F286E44C71605DF1D7DE4A6FE58D249162B086CE17E899FAC88CFA213018F89B8A9939CB842639D2B68A
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket:
    0000 - 5e 9a 46 20 a8 60 30 88-fa 2e c5 37 b7 29 0b 4e   ^.F .`0....7.).N
    0010 - 41 67 2b b6 e7 8e 2e 12-8b 55 0c ad 59 80 f7 d5   Ag+......U..Y...
    0020 - d1 07 8e fc 92 a1 2e 01-59 cf 00 2d d5 39 11 10   ........Y..-.9..
    0030 - bf f3 89 af 2d 7a 02 59-49 54 3a bf e4 8b 97 f3   ....-z.YIT:.....
    0040 - 55 da 4b 6f 9b 86 c4 85-eb e4 f9 a1 e3 74 76 be   U.Ko.........tv.
    0050 - 65 57 76 ec e3 76 c9 c8-5a 47 c6 c2 ee eb bd ec   eWv..v..ZG......
    0060 - 61 88 7c 35 8c a6 c0 b3-25 b5 79 06 99 df 66 75   a.|5....%.y...fu
    0070 - 8e 9d 3a 17 61 40 7c 1c-09 e3 07 aa 49 b9 c3 cf   ..:.a@|.....I...
    0080 - d7 ff 7d 1b cc 3f b9 3f-c7 bd ad 4d f9 4f eb 6c   ..}..?.?...M.O.l
    0090 - 6f 42 2e c8 30 75 a9 07-d4 9e f0 12 6b 9c ca ac   oB..0u......k...

    Start Time: 1544706461
    Timeout   : 7200 (sec)
    Verify return code: 19 (self signed certificate in certificate chain)
    Extended master secret: no
---
HTTP/1.0 408 Request Time-out
Cache-Control: no-cache
Connection: close
Content-Type: text/html

<html><body><h1>408 Request Time-out</h1>
Your browser didn't send a complete request in time.
</body></html>
closed

我也试过这个命令

     curl -vL https://meteotravel.ru >/dev/null

* Rebuilt URL to: https://meteotravel.ru/
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 212.26.248.233...
* TCP_NODELAY set
* Connected to meteotravel.ru (212.26.248.233) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [222 bytes data]
* error:1408F10B:SSL routines:ssl3_get_record:wrong version number
* stopped the pause stream!
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Closing connection 0
curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number

我的 nginx 单独工作正常有配置

server {
access_log /var/log/nginx/default.access.log;
error_log /var/log/nginx/default.error.log warn;


    listen 443 ssl default;
    #listen 443 ssl http2 default reuseport;


   # Redirect HTTP to HTTPS
    if ($scheme = http) {
        return 301 https://$host$request_uri;
    }


    ssl_certificate /etc/nginx/ssl/meteotravel.ru/mt.crt;
    ssl_certificate_key /etc/nginx/ssl/meteotravel.ru/pk;
    server_name meteotravel.ru;



    root /var/www/html;

    # Add index.php to the list if you are using PHP
    index index.html index.htm index.nginx-debian.html;

#   server_name _;

    location /fop2{
        # First attempt to serve request as file, then
        # as directory, then fall back to displaying a 404.


    # Add index.php to the list if you are using PHP

        try_files $uri $uri/ =404;
    }
location /{
        try_files $uri $uri/ =404;
    }

}

哪里出错了？？

这是我在服务器上的 haproxy 配置

我还尝试为 nginx 设置 proxy_protocol：listen 443 ssl proxy_protocol; 和 haproxy-ingress 的选项：代理协议：[v1|v2|v2-ssl|v2-ssl-cn] 所有这一切分开并在 nginx 后端的日志中得到错误

��kjih9876��2�.�*�&��=5"，同时读取代理协议，客户端：10.244.5.0，服务器：0.0.0.0:443 2018/12/13 19:55:24 [错误] 7#7: 2271 标头损坏：“�ۆ�v+\��w�?��3�Alm9i��L$&��h ��0�,�(�$�� kjih9876��2�.��&��=5" 同时读取代理协议，客户端：10.244.6.0，服务器：0.0.0.0:443

【问题讨论】：

openssl 测试不是错误，您没有在预期时间内发送数据并且连接已关闭。 curl 测试听起来像是 TLS 握手失败，如果使用非常旧的客户端连接到非常新的服务器，可能会发生这种情况 - 如果您使用浏览器、wget 或其他 http 客户端怎么办？
由 linux-gnu 上的 Wget 1.19.4 创建的调试输出。从 /home/alex/.wget-hsts 读取 HSTS 条目 URI 编码 = 'UTF-8' 转换后的文件名 'index.html' (UTF-8) -> 'index.html' (UTF-8) --2018- 12-13 23:17:20-- meteotravel.ru 正在解析meteotravel.ru (meteotravel.ru)... 212.26.248.233 缓存meteotravel.ru => 212.26.248.233 连接到meteotravel.ru (meteotravel.ru)|212.26。 248.233|:443... 已连接。创建套接字 5。释放 0x000055a24edfc8f0（新引用计数 1）。启动 SSL 握手。 SSL 握手失败。关闭 fd 5 无法建立 SSL 连接
我也尝试ssllabs.com/ssltest/analyze.html?d=meteotravel.ru 并得到奇怪的结果，没有协议
@Алекс Денькин，您是否检查过容器日志中的 haproxy-ingress 和 meteo-ingress？
我修好了。谢谢

标签： ssl nginx kubernetes haproxy kubernetes-ingress

【解决方案1】：

这个问题是因为注释 ingress.kubernetes.io/secure-backends: "true" 。它不需要，因为我们从 haproxy it tcp 流中制作了已经安全的数据。有了这个注解，我们做了两次加密，nginx无法正确解密

【讨论】：