GKE - 无法部署入口：获取“默认后端 - 404”或“502”

Question

我目前有一个 Deployment 和一个 Service 在 GKE 上运行良好。我的问题是我想将我的外部 IP：端口“绑定”到域名（在 OVH 上），例如：

http://www.example.com/api/grpc -> 12.345.67.89:8080

http://www.example.com/api/rest -> 12.345.67.89:8081

经过大量搜索，我终于发现 Ingress 可能是我的解决方案。然后我更新了我的 yaml 以结合 Deployment、Service、Ingress 中的三个。

这是我的 yaml：

# Copyright 2016 Google Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License

# Use this file to deploy the container for the grpc-bookstore sample
# and the container for the Extensible Service Proxy (ESP) to
# Google Kubernetes Engine (GKE).

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: myservice
  labels:
    app: myservice
spec:
  replicas: 1
  selector:
    matchLabels:
      app: myservice
  template:
    metadata:
      labels:
        app: myservice
    spec:
      containers:
        - name: myservice
          image: gcr.io/<project_id>/myservice:latest
          imagePullPolicy: Always
          ports:
            - containerPort: 8080
            - containerPort: 8081
---
apiVersion: v1
kind: Service
metadata:
  name: myservice
spec:
  type: NodePort
  selector:
    app: myservice
  ports:
    # Port that accepts gRPC and JSON/HTTP2 requests over HTTP.
    - port: 8080
      targetPort: 8080
      protocol: TCP
      name: grpc
    # Port that accepts gRPC and JSON/HTTP2 requests over HTTP.
    - port: 8081
      targetPort: 8081
      protocol: TCP
      name: rest
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: myservice-ingress
spec:
  rules:
  - http:
      paths:
      - path: /grpc
        backend:
          serviceName: myservice
          servicePort: 8080
      - path: /rest
        backend:
          serviceName: myservice
          servicePort: 8081

然后我尝试使用：http://www.example.com/api/rest/test 对我的 REST API 运行一个简单的请求，其中包含我的姓名的 POST json 正文。 API 应该返回 Hello %s 但不，我得到了：

• 默认后端 - 404
• 502 服务器错误（服务器遇到临时错误，无法完成您的请求。请在 30 秒后重试。）

按照 Google 文档操作，我完全不知道可能是什么问题

编辑

我在示例中输入了http://www.example.com/api/rest，但以下内容均不起作用：

http://www.example.com/rest

http://12.345.67.89/rest

更新（2020 年 3 月 19 日）

Soo，我可以继续前进了，现在我的服务（不健康）是健康的，我可以连接到它，在我的 readinessProbe/livenessProbe 端点上运行 CURL 并获得 200 OK。

我的 yaml 的更新版本如下所示：

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: myservice
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      run: myservice
  template:
    metadata:
      labels:
        run: myservice
    spec:
      containers:
        - name: myservice
          image: gcr.io/<project_id>/myservice:latest
          imagePullPolicy: Always
          ports:
            - containerPort: 8080
            - containerPort: 8081
          readinessProbe:
            httpGet:
              path: /health_check
              port: 8081
          livenessProbe:
            httpGet:
              path: /health_check
              port: 8081
---
apiVersion: v1
kind: Service
metadata:
  name: myservice
  namespace: default
spec:
  type: NodePort
  selector:
    run: myservice
  ports:
    # Port that accepts gRPC and JSON/HTTP2 requests over HTTP.
    - port: 8080
      targetPort: 8080
      protocol: TCP
      name: grpc
    # Port that accepts gRPC and JSON/HTTP2 requests over HTTP.
    - port: 8081
      targetPort: 8081
      protocol: TCP
      name: rest
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: myservice-ingress
spec:
  backend:
    serviceName: myservice
    servicePort: 8081

kubectl 描述 pod

MacBook-Pro-de-Emixam23:~ emixam23$ kubectl describe pods
Name:               myservice-c57d64669-phrzr
Namespace:          default
Priority:           0
PriorityClassName:  <none>
Node:               gke-cluster-kuberne-default-pool-8b65afeb-qgcm/10.166.0.31
Start Time:         Thu, 19 Mar 2020 11:36:35 -0400
Labels:             pod-template-hash=c57d64669
                    run=myservice
Annotations:        kubernetes.io/limit-ranger: LimitRanger plugin set: cpu request for container myservice
Status:             Running
IP:                 10.4.2.28
Controlled By:      ReplicaSet/myservice-c57d64669
Containers:
  myservice:
    Container ID:   docker://3f9df91ec4e2631d85e0becdb8d1be64bf97fadb5a5b7049c7391eb8cfdf3eee
    Image:          gcr.io/<project_id>/myservice:latest
    Image ID:       docker-pullable://gcr.io/<project_id>/myservice@sha256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    Ports:          8080/TCP, 8081/TCP
    Host Ports:     0/TCP, 0/TCP
    State:          Running
      Started:      Thu, 19 Mar 2020 11:36:40 -0400
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:        100m
    Liveness:     http-get http://:8081/health_check delay=0s timeout=1s period=10s #success=1 #failure=3
    Readiness:    http-get http://:8081/health_check delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-6cppb (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  default-token-6cppb:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-6cppb
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason     Age    From                                                          Message
  ----    ------     ----   ----                                                          -------
  Normal  Scheduled  8m36s  default-scheduler                                             Successfully assigned default/myservice-c57d64669-phrzr to gke-cluster-kuberne-default-pool-8b65afeb-qgcm
  Normal  Pulling    8m35s  kubelet, gke-cluster-kuberne-default-pool-8b65afeb-qgcm  Pulling image "gcr.io/<project_id>/myservice:latest"
  Normal  Pulled     8m32s  kubelet, gke-cluster-kuberne-default-pool-8b65afeb-qgcm  Successfully pulled image "gcr.io/<project_id>/myservice:latest"
  Normal  Created    8m31s  kubelet, gke-cluster-kuberne-default-pool-8b65afeb-qgcm  Created container myservice
  Normal  Started    8m31s  kubelet, gke-cluster-kuberne-default-pool-8b65afeb-qgcm  Started container myservice

kubectl describe ingress myservice-ingress

MacBook-Pro-de-Emixam23:~ emixam23$ kubectl describe ingress myservice-ingress
Name:             myservice-ingress
Namespace:        default
Address:          XX.XXX.XXX.XXX
Default backend:  myservice:8081 (10.4.2.28:8081)
Rules:
  Host  Path  Backends
  ----  ----  --------
  *     *     myservice:8081 (10.4.2.28:8081)
Annotations:
  ingress.kubernetes.io/backends:                    {"k8s-be-31336--d1838223483f8e56":"HEALTHY"}
  ingress.kubernetes.io/forwarding-rule:             k8s-fw-default-myservice-ingress--d1838223483f8e0
  ingress.kubernetes.io/target-proxy:                k8s-tp-default-myservice-ingress--d1838223483f8e0
  ingress.kubernetes.io/url-map:                     k8s-um-default-myservice-ingress--d1838223483f8e0
  kubectl.kubernetes.io/last-applied-configuration:  {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{},"name":"myservice-ingress","namespace":"default"},"spec":{"backend":{"serviceName":"myservice","servicePort":8081}}}

Events:
  Type    Reason  Age   From                     Message
  ----    ------  ----  ----                     -------
  Normal  ADD     11m   loadbalancer-controller  default/myservice-ingress
  Normal  CREATE  11m   loadbalancer-controller  ip: XX.XXX.XXX.XXX

我没有看到任何错误，但是当我尝试点击 XX.XXX.XXX.XXX/health_check

时，我一直收到 404

更新（2020 年 3 月 19 日）- 2

我的入口现在看起来像这样：

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: myservice-ingress
spec:
  rules:
    - http:
        paths:
          - path: /grpc/*
            backend:
              serviceName: myservice
              servicePort: 8080
          - path: /rest/*
            backend:
              serviceName: myservice
              servicePort: 8081

/rest/* 端点返回 404，gRPC 尚未测试。关于健康，现在我有 3 项服务，其中一项不健康，我不知道为什么：

MacBook-Pro-de-Emixam23:~ emixam23$ kubectl describe ingress myservice-ingress
Name:             myservice-ingress
Namespace:        default
Address:          XX.XXX.XXX.XXX
Default backend:  default-http-backend:80 (10.4.2.7:8080)
Rules:
  Host  Path  Backends
  ----  ----  --------
  *     
        /grpc/*   myservice:8080 (10.4.1.23:8080)
        /rest/*   myservice:8081 (10.4.1.23:8081)
Annotations:
  kubectl.kubernetes.io/last-applied-configuration:  {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{},"name":"myservice-ingress","namespace":"default"},"spec":{"rules":[{"http":{"paths":[{"backend":{"serviceName":"myservice","servicePort":8080},"path":"/grpc/*"},{"backend":{"serviceName":"myservice","servicePort":8081},"path":"/rest/*"}]}}]}}

  ingress.kubernetes.io/backends:         {"k8s-be-30181--d1838223483f8e56":"UNHEALTHY","k8s-be-30368--d1838223483f8e56":"HEALTHY","k8s-be-31613--d1838223483f8e56":"HEALTHY"}
  ingress.kubernetes.io/forwarding-rule:  k8s-fw-default-myservice-ingress--d1838223483f8e0
  ingress.kubernetes.io/target-proxy:     k8s-tp-default-myservice-ingress--d1838223483f8e0
  ingress.kubernetes.io/url-map:          k8s-um-default-myservice-ingress--d1838223483f8e0
Events:
  Type    Reason  Age   From                     Message
  ----    ------  ----  ----                     -------
  Normal  ADD     14m   loadbalancer-controller  default/myservice-ingress
  Normal  CREATE  13m   loadbalancer-controller  ip: XX.XXX.XXX.XXX

另外：https://cloud.google.com/kubernetes-engine/docs/tutorials/http-balancer#step_6_optional_serve_multiple_applications_on_a_load_balancer

Answer 1

您提供的错误：

• 404（未找到）
• 502（错误网关）

与错误的入口配置有关。当请求与路径不匹配时，转发到默认后端。

例子：

      - path: /rest
        backend:
          serviceName: myservice
          servicePort: 8081

只有在请求中指定了/rest 时，此定义才会将流量路由到myservice。它不适用于/api/rest 和/rest/something。

您的后端可能无法正常工作，因为Ingress 资源没有足够的时间来完成配置。

要检查Ingress 资源是否配置正确，您可以这样描述：

$ kubectl describe ingress NAME_OF_INGRESS_RESOURCE

请看下面检查后端是否处于Healthy 或Unhealthy 状态的上述命令部分：

ingress.kubernetes.io/backends: {"k8s-be-31720--0838d11870ae50b1":"HEALTHY","k8s-be-32475--0838d11870ae50b1":"HEALTHY"}

有关更多信息，您可以访问： Google Cloud Platform -> Network services -> Load Balancing。找到对应的转发规则，看看后端服务。

在继续之前，请确保所有后端都处于 Healthy 状态。请参考Cloud.google.com: Kubernetes Engine: Ingress healthchecks上的官方文档，根据自己的需要进行配置。

---

GKE Ingress 资源默认在端口上运行：

• 80 (http)
• 443 (https)

请按照官方文档查看支持的协议：Cloud.google.com: Load Balance Ingress

如果您有兴趣在不同端口上公开您的应用程序，请考虑使用自定义部署的入口控制器之一，例如：

• Nginx 入口
• Traefik

请看官方文档Kubernetes.io: Ingress controllers

---

此外，请验证您的 pod 是否正常运行，并运行一些测试，如果它们以适当的响应进行响应。你可以这样做：

• kubectl exec -it NAME_OF_THE_POD -- /path/to/shell 并从该应用程序 pod 内部检查
• 运行一个 ubuntu pod，在其中执行并向您的应用程序 pod 发送请求。

如果有帮助，请告诉我。

编辑

试试这个Ingress 定义：

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: domain-ingress
spec:
  rules:
  - host: DOMAIN.NAME
    http:
      paths:
      - path: /api/*
        backend:
          serviceName: myservice
          servicePort: 8081
      - path: /admin/* 
        backend:
          serviceName: myservice
          servicePort: 8081  

请更改DOMAIN.NAME 以适合您的情况。