如何通过 Ingress on BareMetal 部署访问 Kubernetes 集群？

Question

我已经用了两天了，我仍然无法弄清楚。

整个部署都在裸机上。

为简单起见，我将集群从 HA 最小化为 1 个主节点和 2 个工作节点。

$ kubectl get nodes
NAME                          STATUS   ROLES    AGE    VERSION
worker1   Ready    <none>   99m    v1.19.2
worker2   Ready    <none>   99m    v1.19.2
master    Ready    master   127m   v1.19.2

我正在运行 Nginx-ingress，但我认为这无关紧要，因为同样的配置也应该应用在 HaProxy 上。

$ kubectl -n ingress-nginx get pod
NAME                                        READY   STATUS      RESTARTS   AGE
ingress-nginx-admission-create-g645g        0/1     Completed   0          129m
ingress-nginx-admission-patch-ftg7p         0/1     Completed   2          129m
ingress-nginx-controller-587cd59444-cxm7z   1/1     Running     0          129m

我可以看到集群上没有外部IP：

$ kubectl get service -A
NAMESPACE                NAME                                 TYPE        CLUSTER-IP        EXTERNAL-IP   PORT(S)                      AGE
cri-o-metrics-exporter   cri-o-metrics-exporter               ClusterIP   192.168.11.163    <none>        80/TCP                       129m
default                  kubernetes                           ClusterIP   192.168.0.1       <none>        443/TCP                      130m
ingress-nginx            ingress-nginx-controller             NodePort    192.168.30.224    <none>        80:32647/TCP,443:31706/TCP   130m
ingress-nginx            ingress-nginx-controller-admission   ClusterIP   192.168.212.9     <none>        443/TCP                      130m
kube-system              kube-dns                             ClusterIP   192.168.0.10      <none>        53/UDP,53/TCP,9153/TCP       130m
kube-system              metrics-server                       ClusterIP   192.168.178.171   <none>        443/TCP                      129m
kubernetes-dashboard     dashboard-metrics-scraper            ClusterIP   192.168.140.142   <none>        8000/TCP                     129m
kubernetes-dashboard     kubernetes-dashboard                 ClusterIP   192.168.100.126   <none>        443/TCP                      129m

ConfigMap 示例：

apiVersion: v1
kind: ConfigMap
metadata:
  name: dashboard-ingress-nginx
  namespace: kubernetes-dashboard
data:
  ssl-certificate: my-cert

Ingress conf 示例：

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: dashboard-ingress-ssl
  namespace: kubernetes-dashboard
  annotations:
    nginx.ingress.kubernetes.io/secure-backends: "true"
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
    nginx.ingress.kubernetes.io/whitelist-source-range: 10.96.0.0/16  #the IP to be allowed
  spec:
    tls:
      - hosts:
        - kube.my.domain.internal
        secretName: my-cert
    rules:
      - host: kube.my.domain.internal
        http:
          paths:
          - path: /
            backend:
              serviceName: kubernetes-dashboard
              servicePort: 443

如果将我的浏览器重定向到域，例如https://kube.my.domain.internal 我看到 403 被禁止。是否有可能是由于 RBAC 规则导致我无法查看仪表板？

我找到了相关问题，但尽管这似乎对其他用户有效，因为他们不ingress configuration for dashboard。我还尝试将大量 IP 列入白名单，如此处所述Restricting Access By IP (Allow/Block Listing) Using NGINX-Ingress Controller in Kubernetes，但结果仍然相同。

但我也无法理解为什么 Nginx-ingress 只在一个节点上启动，而我希望在两个节点（工作程序）上启动。我在任何节点上都没有标签。

我还阅读了有关 MetalLB Bare-metal considerations 的信息，但就我而言，我并没有尝试访问专用网络之外的网络，我只是尝试从集群外部访问节点进入集群。我可能是错的，但我认为此时不需要这样做。

更新：我已经设法使用 kubectl 代理启动仪表板，如官方页面 Web UI (Dashboard) 中所述，但由于我想将集群升级到 HA，这不是最佳解决方案。如果运行代理的节点出现故障，则仪表板将无法访问。

Update2：在遵循metallb/Layer 2 Configuration 的文档之后，我得到了以下几点：

$ kubectl get pods -A -o wide
NAMESPACE                NAME                                           READY   STATUS      RESTARTS   AGE     IP              NODE                          NOMINATED NODE   READINESS GATES
cri-o-metrics-exporter   cri-o-metrics-exporter-77c9cf9746-5xw4d        1/1     Running     0          30m     172.16.9.131    workerNode   <none>           <none>
ingress-nginx            ingress-nginx-admission-create-cz9h9           0/1     Completed   0          31m     172.16.9.132    workerNode   <none>           <none>
ingress-nginx            ingress-nginx-admission-patch-8fkhk            0/1     Completed   2          31m     172.16.9.129    workerNode   <none>           <none>
ingress-nginx            ingress-nginx-controller-8679c5678d-fmc2q      1/1     Running     0          31m     172.16.9.134    workerNode   <none>           <none>
kube-system              calico-kube-controllers-574d679d8c-7jt87       1/1     Running     0          32m     172.16.25.193   masterNode          <none>           <none>
kube-system              calico-node-sf2cn                              1/1     Running     0          9m11s   10.96.95.52     workerNode   <none>           <none>
kube-system              calico-node-zq9vf                              1/1     Running     0          32m     10.96.96.98     masterNode          <none>           <none>
kube-system              coredns-7588b55795-5pg6m                       1/1     Running     0          32m     172.16.25.195   masterNode          <none>           <none>
kube-system              coredns-7588b55795-n8z2p                       1/1     Running     0          32m     172.16.25.194   masterNode          <none>           <none>
kube-system              etcd-masterNode                      1/1     Running     0          32m     10.96.96.98     masterNode          <none>           <none>
kube-system              kube-apiserver-masterNode            1/1     Running     0          32m     10.96.96.98     masterNode          <none>           <none>
kube-system              kube-controller-manager-masterNode   1/1     Running     0          32m     10.96.96.98     masterNode          <none>           <none>
kube-system              kube-proxy-6d5sj                               1/1     Running     0          9m11s   10.96.95.52     workerNode   <none>           <none>
kube-system              kube-proxy-9dfbk                               1/1     Running     0          32m     10.96.96.98     masterNode          <none>           <none>
kube-system              kube-scheduler-masterNode            1/1     Running     0          32m     10.96.96.98     masterNode          <none>           <none>
kube-system              metrics-server-76bb4cfc9f-5tzfh                1/1     Running     0          31m     172.16.9.130    workerNode   <none>           <none>
kubernetes-dashboard     dashboard-metrics-scraper-5f644f6df-8sjsx      1/1     Running     0          31m     172.16.25.197   masterNode          <none>           <none>
kubernetes-dashboard     kubernetes-dashboard-85b6486959-thhnl          1/1     Running     0          31m     172.16.25.196   masterNode          <none>           <none>
metallb-system           controller-56f5f66c6f-5vvhf                    1/1     Running     0          31m     172.16.9.133    workerNode   <none>           <none>
metallb-system           speaker-n5gxx                                  1/1     Running     0          31m     10.96.96.98     masterNode          <none>           <none>
metallb-system           speaker-n9x9v                                  1/1     Running     0          8m51s   10.96.95.52     workerNode   <none>           <none>
$ kubectl get service -A
NAMESPACE                NAME                                 TYPE        CLUSTER-IP        EXTERNAL-IP   PORT(S)                      AGE
cri-o-metrics-exporter   cri-o-metrics-exporter               ClusterIP   192.168.74.27     <none>        80/TCP                       31m
default                  kubernetes                           ClusterIP   192.168.0.1       <none>        443/TCP                      33m
ingress-nginx            ingress-nginx-controller             NodePort    192.168.201.230   <none>        80:30509/TCP,443:31554/TCP   32m
ingress-nginx            ingress-nginx-controller-admission   ClusterIP   192.168.166.218   <none>        443/TCP                      32m
kube-system              kube-dns                             ClusterIP   192.168.0.10      <none>        53/UDP,53/TCP,9153/TCP       32m
kube-system              metrics-server                       ClusterIP   192.168.7.75      <none>        443/TCP                      31m
kubernetes-dashboard     dashboard-metrics-scraper            ClusterIP   192.168.51.178    <none>        8000/TCP                     31m
kubernetes-dashboard     kubernetes-dashboard                 ClusterIP   192.168.50.70     <none>        443/TCP                      31m

但我无法看到公共 IP，因此我可以通过 NAT 访问集群。

Answer 1

我进行 BareMetal 设置的方式是安装 METALLB 和 ingress-nxginx 并使用 NAT 将我的主机（端口 80 和 443）上收到的流量转发到 ingress-nginx。

# MetalLB installation

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/namespace.yaml
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/metallb.yaml

# On first install only
kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"
kubectl apply -f $YAML_FILES/0_cluster_setup_metallb_conf.yaml

0_cluster_setup_metallb_conf.yaml：

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 10.204.61.240-10.204.61.250

Answer 2

经过这么长时间，我终于弄明白了。我是 k8s 的初学者，所以该解决方案可能对其他初学者有所帮助。

我决定在仪表板运行的同一个命名空间上启动入口。用户可以选择不同的名称空间，只需确保使用将您的名称空间与 kubernetes-dashboard 服务连接即可。文档可以在这里找到Understanding namespaces and DNS。

适用于 NGINX 入口的完整示例代码：

apiVersion: networking.k8s.io/v1
      kind: Ingress
      metadata:
        name: kubernetes-dashboard-ingress
        namespace: kubernetes-dashboard
        annotations:
          kubernetes.io/ingress.class: "nginx"
          nginx.ingress.kubernetes.io/ssl-passthrough: "true"
          nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
      spec:
        tls:
        - hosts:
            - "dashboard.example.com"
          secretName: kubernetes-dashboard-secret
        rules:
        - host: "dashboard.example.com"
          http:
            paths:
            - path: /
              pathType: Prefix
              backend:
                service:
                  name: kubernetes-dashboard
                  port:
                    number: 443

请记住遵循入口控制器的注释（在示例 NGINX 上），因为它们可能会在以后的版本中更新。例如，由于版本 image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.26.1，注释 secure-backends 已替换为 backend-protocol。

同样在示例中，外部负载平衡器配置必须为 SSL Pass-Through。

更新：如果其他人是 k8s 的初学者，我不太清楚的次要输入是，如果用户决定使用 MetalLB，您需要指定 @入口文件上的 987654328@。