Ingress 默认后端是基本身份验证返回 401 而不是 404答案

【问题标题】：Ingress default backend is basic auth returns 401 instead of 404Ingress 默认后端是基本身份验证返回 401 而不是 404
【发布时间】：2021-04-19 07:27:32
【问题描述】：

我已经将 ingress-nginx helm chart ( 3.20.1 , https://github.com/kubernetes/ingress-nginx/tree/master/charts/ingress-nginx ) 部署到 k8s 集群中。

为应用程序配置的一些 Ingress 使用基本身份验证 - 可以按预期工作。 IE。我可以在没有基本身份验证的情况下访问集群中的一些应用程序（按照配置）并且某些应用程序需要专用的基本身份验证凭据（按照配置）。

但是尝试未知/不受支持的 URL/路径在默认后端处理时会很尴尬。不知何故，默认后端现在也需要基本身份验证（作为具有基本身份验证的入口之一）？

我怎样才能找出这个基本身份验证在不应该启动的原因/位置？我确实希望默认后端在没有基本身份验证的情况下为 404 提供错误页面！

$ kubectl -n ingress get svc
NAME                                 TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                      AGE
ingress-nginx-controller             LoadBalancer   10.0.235.5     xx.yy.zzz.zzz   80:31294/TCP,443:31732/TCP   39d
ingress-nginx-controller-admission   ClusterIP      10.0.190.204   <none>          443/TCP                      39d
ingress-nginx-controller-metrics     ClusterIP      10.0.21.184    <none>          9913/TCP                     12d
ingress-nginx-defaultbackend         ClusterIP      10.0.109.195   <none>          80/TCP                       39d

当前部署的 Ingress：

$ kubectl get ingress -A
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
NAMESPACE       NAME                                      CLASS    HOSTS                ADDRESS         PORTS     AGE
dev-app-a       app-a                                     <none>   my-host.com          xx.yy.zzz.zzz   80, 443   46d
dev-jaeger      jaeger                                    <none>   my-host.com          xx.yy.zzz.zzz   80, 443   47d
dev-app-b       app-b                                     <none>   my-host.com          xx.yy.zzz.zzz   80, 443   17d
dev-app-c       app-c                                     <none>   my-host.com          xx.yy.zzz.zzz   80, 443   12d
dev-app-d       app-d                                     <none>   my-host.com          xx.yy.zzz.zzz   80, 443   4d22h
prom            prometheus-grafana                        <none>   my-other-host.com    xx.yy.zzz.zzz   80, 443   20d
prom            prometheus-kube-prometheus-alertmanager   <none>   my-other-host.com    xx.yy.zzz.zzz   80, 443   20d
prom            prometheus-kube-prometheus-prometheus     <none>   my-other-host.com    xx.yy.zzz.zzz   80, 443   20d
stage-app-a     app-a                                     <none>   my-third-host.com    xx.yy.zzz.zzz   80, 443   39d
stage-jaeger    jaeger                                    <none>   my-third-host.com    xx.yy.zzz.zzz   80, 443   39d

入口详情：

apiVersion: v1
items:
- apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    annotations:
      cert-manager.io/cluster-issuer: letsencrypt-prod
      kubernetes.io/ingress.class: nginx
      nginx.ingress.kubernetes.io/enable-cors: "false"
      nginx.ingress.kubernetes.io/proxy-buffering: "off"
    labels:
      app.kubernetes.io/name: app-a
    name: app-a
    namespace: foo-app-a
  spec:
    rules:
    - host: my-host.com
      http:
        paths:
        - backend:
            serviceName: app-a-http
            servicePort: http
          path: /abc/
          pathType: ImplementationSpecific
    tls:
    - hosts:
      - my-host.com
      secretName: app-a-tls
  status:
    loadBalancer:
      ingress:
      - ip: xx.xx.xx.xx
- apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    annotations:
      cert-manager.io/cluster-issuer: letsencrypt-prod
      kubernetes.io/ingress.class: nginx
      nginx.ingress.kubernetes.io/add-base-url: "true"
      nginx.ingress.kubernetes.io/auth-realm: Authentication Required - Jaeger UI
      nginx.ingress.kubernetes.io/auth-secret: jaeger-basic-auth
      nginx.ingress.kubernetes.io/auth-type: basic
      nginx.ingress.kubernetes.io/enable-cors: "false"
      nginx.ingress.kubernetes.io/from-to-www-redirect: "true"
      nginx.ingress.kubernetes.io/proxy-buffering: "off"
    labels:
      app.kubernetes.io/name: jaeger
    name: jaeger
    namespace: dev-jaeger
  spec:
    rules:
    - host: my-host.com
      http:
        paths:
        - backend:
            serviceName: jaeger
            servicePort: http-ui
          path: /jaeger/
          pathType: ImplementationSpecific
    tls:
    - hosts:
      - my-host.com
      secretName: jaeger-tls
  status:
    loadBalancer:
      ingress:
      - ip: xx.xx.xx.xx
- apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    annotations:
      cert-manager.io/cluster-issuer: letsencrypt-prod
      kubernetes.io/ingress.class: nginx
      nginx.ingress.kubernetes.io/enable-cors: "false"
      nginx.ingress.kubernetes.io/proxy-buffering: "off"
      nginx.ingress.kubernetes.io/rewrite-target: /$1
    labels:
      app.kubernetes.io/name: app-b
    name: app-b
    namespace: dev-app-b
  spec:
    rules:
    - host: my-host.com
      http:
        paths:
        - backend:
            serviceName: app-b
            servicePort: http
          path: /app-b/(.*)
          pathType: ImplementationSpecific
    tls:
    - hosts:
      - my-host.com
      secretName: app-b-tls
  status:
    loadBalancer:
      ingress:
      - ip: xx.xx.xx.xx
- apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    annotations:
      cert-manager.io/cluster-issuer: letsencrypt-prod
      kubernetes.io/ingress.class: nginx
      nginx.ingress.kubernetes.io/enable-cors: "false"
      nginx.ingress.kubernetes.io/proxy-buffering: "off"
      nginx.ingress.kubernetes.io/rewrite-target: /$1
    labels:
      app.kubernetes.io/name: app-c
    name: app-c
    namespace: dev-app-c
  spec:
    rules:
    - host: my-host.com
      http:
        paths:
        - backend:
            serviceName: app-c
            servicePort: http
          path: /app-c/(.*)
          pathType: ImplementationSpecific
    tls:
    - hosts:
      - my-host.com
      secretName: app-c-tls
  status:
    loadBalancer:
      ingress:
      - ip: xx.xx.xx.xx
- apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    annotations:
      kubernetes.io/ingress.class: nginx
      meta.helm.sh/release-name: prometheus
      meta.helm.sh/release-namespace: prom
      nginx.ingress.kubernetes.io/add-base-url: "true"
      nginx.ingress.kubernetes.io/auth-realm: Authentication Required - Grafana UI
      nginx.ingress.kubernetes.io/auth-secret: monitoring-basic-auth
      nginx.ingress.kubernetes.io/auth-type: basic
      nginx.ingress.kubernetes.io/enable-cors: "false"
      nginx.ingress.kubernetes.io/from-to-www-redirect: "true"
      nginx.ingress.kubernetes.io/proxy-buffering: "off"
    labels:
      app.kubernetes.io/instance: prometheus
      app.kubernetes.io/managed-by: Helm
      app.kubernetes.io/name: grafana
      app.kubernetes.io/version: 7.4.2
      helm.sh/chart: grafana-6.4.8
    name: prometheus-grafana
    namespace: prom
  spec:
    rules:
    - host: my-other-host.com
      http:
        paths:
        - backend:
            serviceName: prometheus-grafana
            servicePort: 80
          path: /monitoring/grafana/
          pathType: ImplementationSpecific
    tls:
    - hosts:
      - my-other-host.com
      secretName: prometheus-general-tls
  status:
    loadBalancer:
      ingress:
      - ip: xx.xx.xx.xx
- apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    annotations:
      kubernetes.io/ingress.class: nginx
      meta.helm.sh/release-name: prometheus
      meta.helm.sh/release-namespace: prom
      nginx.ingress.kubernetes.io/auth-realm: Authentication Required - AlertManager UI
      nginx.ingress.kubernetes.io/auth-secret: monitoring-basic-auth
      nginx.ingress.kubernetes.io/auth-type: basic
    labels:
      app: kube-prometheus-stack-alertmanager
      app.kubernetes.io/managed-by: Helm
      chart: kube-prometheus-stack-14.0.1
      heritage: Helm
      release: prometheus
    name: prometheus-kube-prometheus-alertmanager
    namespace: prom
  spec:
    rules:
    - host: my-other-host.com
      http:
        paths:
        - backend:
            serviceName: prometheus-kube-prometheus-alertmanager
            servicePort: 9093
          path: /monitoring/alertmanager/
          pathType: ImplementationSpecific
    tls:
    - hosts:
      - my-other-host.com
      secretName: prometheus-general-tls
  status:
    loadBalancer:
      ingress:
      - ip: xx.xx.xx.xx
- apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    annotations:
      cert-manager.io/cluster-issuer: letsencrypt-prod
      kubernetes.io/ingress.class: nginx
      meta.helm.sh/release-name: prometheus
      meta.helm.sh/release-namespace: prom
      nginx.ingress.kubernetes.io/auth-realm: Authentication Required - Prometheus UI
      nginx.ingress.kubernetes.io/auth-secret: monitoring-basic-auth
      nginx.ingress.kubernetes.io/auth-type: basic
    labels:
      app: kube-prometheus-stack-prometheus
      app.kubernetes.io/managed-by: Helm
      chart: kube-prometheus-stack-14.0.1
      heritage: Helm
      release: prometheus
    name: prometheus-kube-prometheus-prometheus
    namespace: prom
  spec:
    rules:
    - host: my-other-host.com
      http:
        paths:
        - backend:
            serviceName: prometheus-kube-prometheus-prometheus
            servicePort: 9090
          path: /monitoring/prometheus/
          pathType: ImplementationSpecific
    tls:
    - hosts:
      - my-other-host.com
      secretName: prometheus-general-tls
  status:
    loadBalancer:
      ingress:
      - ip: xx.xx.xx.xx
- apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    annotations:
      cert-manager.io/cluster-issuer: letsencrypt-prod
      kubernetes.io/ingress.class: nginx
      nginx.ingress.kubernetes.io/enable-cors: "false"
      nginx.ingress.kubernetes.io/proxy-buffering: "off"
    labels:
      app.kubernetes.io/managed-by: terraform
      app.kubernetes.io/name: app-a
    name: app-a
    namespace: stage-app-a
  spec:
    rules:
    - host: my-third-host.com
      http:
        paths:
        - backend:
            serviceName: app-a-http
            servicePort: http
          path: /controller/
          pathType: ImplementationSpecific
    tls:
    - hosts:
      - my-third-host.com
      secretName: app-a-tls
  status:
    loadBalancer:
      ingress:
      - ip: xx.xx.xx.xx
- apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    annotations:
      cert-manager.io/cluster-issuer: letsencrypt-prod
      kubernetes.io/ingress.class: nginx
      nginx.ingress.kubernetes.io/add-base-url: "true"
      nginx.ingress.kubernetes.io/auth-realm: Authentication Required - Jaeger UI
      nginx.ingress.kubernetes.io/auth-secret: jaeger-basic-auth
      nginx.ingress.kubernetes.io/auth-type: basic
      nginx.ingress.kubernetes.io/enable-cors: "false"
      nginx.ingress.kubernetes.io/from-to-www-redirect: "true"
      nginx.ingress.kubernetes.io/proxy-buffering: "off"
    labels:
      app.kubernetes.io/name: jaeger
    name: jaeger
    namespace: stage-jaeger
  spec:
    rules:
    - host: my-third-host.com
      http:
        paths:
        - backend:
            serviceName: jaeger
            servicePort: http-ui
          path: /jaeger/
          pathType: ImplementationSpecific
    tls:
    - hosts:
      - my-third-host.com
      secretName: jaeger-tls
  status:
    loadBalancer:
      ingress:
      - ip: xx.xx.xx.xx

例如在未知路径上尝试简单的 curl （我希望默认后端给定 404 响应）返回来自我的 Prometheus 端点的数据，这些端点受到基本身份验证保护：

（真实的主机名/ip已经被我混淆了！）

$ curl -v "https://my-other-host.com/bar"
*   Trying xx.xx.xx.xx...
* TCP_NODELAY set
* Connected to my-other-host.com (xx.xx.xx.xx) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=my-other-host.com
*  start date: Apr  6 14:28:20 2021 GMT
*  expire date: Jul  5 14:28:20 2021 GMT
*  subjectAltName: host "my-other-host.com" matched cert's "my-other-host.com"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fec1300d600)
> GET /bar HTTP/2
> Host: my-other-host.com
> User-Agent: curl/7.64.1
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 401
< date: Wed, 21 Apr 2021 15:37:24 GMT
< content-type: text/html
< content-length: 172
< www-authenticate: Basic realm="Authentication Required - Prometheus UI"
< strict-transport-security: max-age=15724800; includeSubDomains
<
<html>
<head><title>401 Authorization Required</title></head>
<body>
<center><h1>401 Authorization Required</h1></center>
<hr><center>nginx</center>
</body>
</html>
* Connection #0 to host my-other-host.com left intact
* Closing connection 0

或类似尝试在其他主机上时，我会从看起来像我的 Jaeger 端点而不是默认后端 404 的东西中获得 401：

$ curl -v "https://my-host.com/foo"
*   Trying xx.xx.xx.xx...
* TCP_NODELAY set
* Connected to my-host.com (xx.xx.xx.xx) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=my-host.com
*  start date: Mar 18 06:37:40 2021 GMT
*  expire date: Jun 16 06:37:40 2021 GMT
*  subjectAltName: host "my-host.com" matched cert's "my-host.com"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7f968480d600)
> GET /foo HTTP/2
> Host: my-host.com
> User-Agent: curl/7.64.1
> Accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 401 
< date: Wed, 21 Apr 2021 15:59:21 GMT
< content-type: text/html
< content-length: 172
< www-authenticate: Basic realm="Authentication Required - Jaeger UI"
< strict-transport-security: max-age=15724800; includeSubDomains
< 
<html>
<head><title>401 Authorization Required</title></head>
<body>
<center><h1>401 Authorization Required</h1></center>
<hr><center>nginx</center>
</body>
</html>
* Connection #0 to host my-host.com left intact
* Closing connection 0

【问题讨论】：

更多参考和复制目的，请添加Ingress清单。它应该可以解释为什么会发生这种情况。
我更新了问题并添加了入口清单。
我试图重现您的问题，但不幸的是我不能。 basic-auth 工作正常，默认后端不需要它。您能否说出当您到达这样的路径时显示了哪个：auth-realm（Jaeger、Prometheus、Grafana 等）？另外，您是如何配置默认后端的，因为我在 Ingress 清单中看不到它。
我已经用一些 curl 输出更新了这个问题。我没有配置任何默认后端，它只是入口舵图附带的默认后端运行。列出的入口是我集群中的完整列表。
@DawidKruk 你知道什么会导致这种奇怪的请求路由吗？非常感谢您在发现此问题时提供的帮助，感谢您查看它。

标签： kubernetes kubernetes-ingress

【解决方案1】：

通过为集群中服务的每个主机显式添加 Ingress 路由来修复（或解决该问题）：

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
  name: default-route-per-host
  namespace: ingress
spec:
  rules:
  - host: my-host.com
    http:
      paths:
      - backend:
          service:
            name: ingress-nginx-defaultbackend
            port:
              name: http
        path: /
        pathType: Prefix
  - host: my-other-host.com
    http:
      paths:
      - backend:
          service:
            name: ingress-nginx-defaultbackend
            port:
              name: http
        path: /
        pathType: Prefix
  - host: my-third-host.com
    http:
      paths:
      - backend:
          service:
            name: ingress-nginx-defaultbackend
            port:
              name: http
        path: /
        pathType: Prefix

【讨论】：