HTTPS 重定向不适用于 nginx-ingress-controller 的默认后端答案

【问题标题】：HTTPS redirect not working for default backend of nginx-ingress-controllerHTTPS 重定向不适用于 nginx-ingress-controller 的默认后端
【发布时间】：2019-07-16 02:06:58
【问题描述】：

对于 kubernetes 的 NGINX 入口控制器的默认后端，我无法从 HTTP -> HTTPS 发生自动重定向，其中控制器位于 AWS Classic ELB 后面；有可能吗？

According to the guide 好像默认启用了 HSTS

HTTP 严格传输安全
HTTP 严格传输安全 (HSTS) 是通过使用特殊响应标头指定的可选安全增强功能。一旦受支持的浏览器收到此标头，该浏览器将阻止通过 HTTP 向指定域发送任何通信，而是通过 HTTPS 发送所有通信。

HSTS 默认启用。

并且重定向 HTTP -> HTTPS 已启用

通过重定向执行服务器端 HTTPS
默认情况下，如果为该 Ingress 启用了 TLS，控制器会使用 308 永久重定向响应将 HTTP 客户端重定向到 HTTPS 端口 443。

但是，当我按照下面的配置部署控制器并导航到 http://<ELB>.elb.amazonaws.com 时，我无法得到任何响应（curl 报告 Empty reply from server）。我希望发生的事情是我应该看到 308 重定向到 https，然后是 404。

这个问题类似：Redirection from http to https not working for custom backend service in Kubernetes Nginx Ingress Controller，但他们通过部署自定义后端并在入口资源上指定使用 TLS 解决了这个问题。我试图避免部署自定义后端，只是想使用默认值，因此此解决方案不适用于我的情况。

I've shared my deployment files on gist 并将它们也复制到这里：

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: nginx-ingress-controller
  namespace: ingress-nginx-sit
  labels:
    app.kubernetes.io/name: ingress-nginx-sit
    app.kubernetes.io/part-of: ingress-nginx-sit
spec:
  minReadySeconds: 2
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: '50%'
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-nginx-sit
      app.kubernetes.io/part-of: ingress-nginx-sit
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ingress-nginx-sit
        app.kubernetes.io/part-of: ingress-nginx-sit
      annotations:
        prometheus.io/port: '10254'
        prometheus.io/scrape: 'true'
    spec:
      serviceAccountName: nginx-ingress-serviceaccount
      containers:
        - name: nginx-ingress-controller
          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.23.0
          args:
            - /nginx-ingress-controller
            - --configmap=$(POD_NAMESPACE)/nginx-configuration
            - --annotations-prefix=nginx.ingress.kubernetes.io
            - --publish-service=$(POD_NAMESPACE)/ingress-nginx
            - --ingress-class=$(POD_NAMESPACE)
            - --election-id=leader
            - --watch-namespace=$(POD_NAMESPACE)
          securityContext:
            allowPrivilegeEscalation: true
            capabilities:
              drop:
                - ALL
              add:
                - NET_BIND_SERVICE
            # www-data -> 33
            runAsUser: 33
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          ports:
          - name: http
            containerPort: 80
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 10
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 10

---

kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-configuration
  namespace: ingress-nginx-sit
  labels:
    app.kubernetes.io/name: ingress-nginx-sit
    app.kubernetes.io/part-of: ingress-nginx-sit
data:
  hsts: "true"
  ssl-redirect: "true"
  use-proxy-protocol: "false"
  use-forwarded-headers: "true"
  enable-access-log-for-default-backend: "true"
  enable-owasp-modsecurity-crs: "true"
  proxy-real-ip-cidr: "10.0.0.0/24,10.0.1.0/24" # restrict this to the IP addresses of ELB

kind: Service
apiVersion: v1
metadata:
  name: ingress-nginx
  namespace: ingress-nginx-sit
  labels:
    app.kubernetes.io/name: ingress-nginx-sit
    app.kubernetes.io/part-of: ingress-nginx-sit
  annotations:
    # replace with the correct value of the generated certificate in the AWS console
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:<region>:<account>:certificate/<id>"
    # Specify the ssl policy to apply to the ELB
    service.beta.kubernetes.io/aws-load-balancer-ssl-negotiation-policy: "ELBSecurityPolicy-TLS-1-2-2017-01"
    # the backend instances are HTTP
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "http"
    # Terminate ssl on https port
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "*"
    # Ensure the ELB idle timeout is less than nginx keep-alive timeout. By default,
    # NGINX keep-alive is set to 75s. If using WebSockets, the value will need to be
    # increased to '3600' to avoid any potential issues.
    service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "60"
    # Security group used for the load balancer.
    service.beta.kubernetes.io/aws-load-balancer-extra-security-groups: "sg-xxxxx"
spec:
  type: LoadBalancer
  selector:
    app.kubernetes.io/name: ingress-nginx-sit
    app.kubernetes.io/part-of: ingress-nginx-sit
  loadBalancerSourceRanges:
    # Restrict allowed source IP ranges
    - "192.168.1.1/16"
  ports:
    - name: http
      port: 80
      targetPort: http
      # The range of valid ports is 30000-32767
      nodePort: 30080
    - name: https
      port: 443
      targetPort: http
      # The range of valid ports is 30000-32767
      nodePort: 30443

【问题讨论】：

标签： ssl nginx kubernetes amazon-elb amazon-eks

【解决方案1】：

我想我找到了问题。

由于某种原因，默认服务器在确定是否应将传入请求重定向到 HTTPS 时将 force_ssl_redirect 设置为 false：

cat /etc/nginx/nginx.conf 注意到rewrite_by_lua_block 发送force_ssl_redirect = false

...
    ## start server _
    server {
        server_name _ ;

        listen 80 default_server reuseport backlog=511;

        set $proxy_upstream_name "-";
        set $pass_access_scheme $scheme;
        set $pass_server_port $server_port;
        set $best_http_host $http_host;
        set $pass_port $pass_server_port;

        listen 443  default_server reuseport backlog=511 ssl http2;

        # PEM sha: 601213c2dd57a30b689e1ccdfaa291bf9cc264c3
        ssl_certificate                         /etc/ingress-controller/ssl/default-fake-certificate.pem;
        ssl_certificate_key                     /etc/ingress-controller/ssl/default-fake-certificate.pem;

        ssl_certificate_by_lua_block {
            certificate.call()
        }

        location / {

            set $namespace      "";
            set $ingress_name   "";
            set $service_name   "";
            set $service_port   "0";
            set $location_path  "/";

            rewrite_by_lua_block {
                lua_ingress.rewrite({
                    force_ssl_redirect = false,
                    use_port_in_redirects = false,
                })
                balancer.rewrite()
                plugins.run()
            }
...

那么，LUA 代码需要force_ssl_redirect 和 redirect_to_https()

cat /etc/nginx/lua/lua_ingress.lua

...
  if location_config.force_ssl_redirect and redirect_to_https() then
    local uri = string_format("https://%s%s", redirect_host(), ngx.var.request_uri)

    if location_config.use_port_in_redirects then
      uri = string_format("https://%s:%s%s", redirect_host(), config.listen_ports.https, ngx.var.request_uri)
    end

    ngx_redirect(uri, config.http_redirect_code)
  end
...

据我所知，force_ssl_redirect 设置只有controlled at the Ingress resource 级别，通过注释nginx.ingress.kubernetes.io/force-ssl-redirect: "true"。因为我没有设置入口规则（这意味着它是不匹配任何入口的请求的默认服务器），所以我无法更改此设置。

所以我确定我必须做的是在另一个端口上定义我自己的自定义服务器 sn-p，并将 force_ssl_redirect 设置为 true，然后将服务负载均衡器指向该自定义服务器而不是默认值。具体来说：

添加到ConfigMap：

...
  http-snippet: |
    server {
      server_name _ ;
      listen 8080 default_server reuseport backlog=511;

      set $proxy_upstream_name "-";
      set $pass_access_scheme $scheme;
      set $pass_server_port $server_port;
      set $best_http_host $http_host;
      set $pass_port $pass_server_port;

      server_tokens off;
      location / {
        rewrite_by_lua_block {
            lua_ingress.rewrite({
                force_ssl_redirect = true,
                use_port_in_redirects = false,
            })
            balancer.rewrite()
            plugins.run()
        }
      }
      location /healthz {
        access_log off;
        return 200;
      }
    }
  server-snippet: |
    more_set_headers "Strict-Transport-Security: max-age=31536000; includeSubDomains; preload";

注意我还添加了server-snippet 以正确启用HSTS。我认为因为从 ELB 到 NGINX 的流量是 HTTP 而不是 HTTPS，所以默认情况下没有正确添加 HSTS 标头。

添加到DaemonSet：

...
        ports:
          - name: http
            containerPort: 80
          - name: http-redirect
            containerPort: 8080
...

修改了Service：

...
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "https"
...
  ports:
    - name: http
      port: 80
      targetPort: http-redirect
      # The range of valid ports is 30000-32767
      nodePort: 30080
    - name: https
      port: 443
      targetPort: http
      # The range of valid ports is 30000-32767
      nodePort: 30443
...

现在看来一切正常。我已经更新了the Gist，所以它包含了我正在使用的完整配置。

【讨论】：