Kubernetes HA：Flannel 抛出 SubnetManager 错误

Question

我一直按照https://kubernetes.io/docs/setup/independent/high-availability/ 提供的步骤来启动 HA 集群。我正在使用 CoreOS 节点 (VERSION=1688.5.3) 和 Kubernetes 版本 v1.10。

我选择在主节点上运行所有三个 etcd。对于负载均衡器，我使用了在 https://github.com/alterway/docker-keepalived 找到的容器化 keepalived。已上传到容器化 keepalived 的 keepalived.conf 文件在 k8s HA 指南中给出。

当我到达配置 CNI 网络 (https://kubernetes.io/docs/setup/independent/high-availability/#install-cni-network) 的步骤时，flannel-ds pod 进入 CrashLoopBackoff 并出现错误：“Failed to create SubnetManager: error retrieving pod spec for 'kube-system/kube-flannel -ds-fjn6w': Get https://10.96.0.1:443/api/v1/namespaces/kube-system/pods/kube-flannel-ds-fjn6w: dial tcp 10.96.0.1:443: i/o timeout"

这可能是什么问题？以下是运行 flannel-ds pod 的主节点的 iptables：

The flannel pod is trying to retrieve its configuration from the API server using the service-IP 10.96.0.1, which is supposed to get DNAT to node IPs
    -A KUBE-SERVICES -d 10.96.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y
    -A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m statistic --mode random --probability 0.33332999982 -j KUBE-SEP-SIIK55AX7MK5ONR7
    -A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-GBLS75FLCCJBNQB6
    -A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -j KUBE-SEP-2CDZMOLH2PKAG52U

But I don’t see these rules being triggered at all.
    0     0 KUBE-SEP-SIIK55AX7MK5ONR7  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/kubernetes:https */ statistic mode random probability 0.33332999982
    0     0 KUBE-SEP-GBLS75FLCCJBNQB6  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/kubernetes:https */ statistic mode random probability 0.50000000000
    0     0 KUBE-SEP-2CDZMOLH2PKAG52U  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/kubernetes:https */


Curl to the service IP does not work, however a curl request to the kubernetes cluster IP gets a response:

    master # curl -k https://10.96.0.1:443/api/v1/namespaces/kube-system/pods/
    curl: (7) Failed to connect to 10.96.0.1 port 443: Connection timed out

    master # curl -k https://10.106.73.226:6443/api/v1/namespaces/kube-system/pods/
    {
      "kind": "Status",
      "apiVersion": "v1",
      "metadata": {

      },
      "status": "Failure",
      "message": "pods is forbidden: User \"system:anonymous\" cannot list pods in the namespace \"kube-system\"",
      "reason": "Forbidden",
      "details": {
        "kind": "pods"
      },
      "code": 403

Also note, the service endpoints have been set correctly to the cluster IP:

    master # kubectl describe svc kubernetes
    Name:              kubernetes
    Namespace:         default
    Labels:            component=apiserver
                       provider=kubernetes
    Annotations:       <none>
    Selector:          <none>
    Type:              ClusterIP
    IP:                10.96.0.1
    Port:              https  443/TCP
    TargetPort:        6443/TCP
    Endpoints:         10.106.73.226:6443
    Session Affinity:  ClientIP
    Events:            <none>

    master # kubectl cluster-info
    Kubernetes master is running at https://10.106.73.226:6443
    KubeDNS is running at https://10.106.73.226:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

我尝试添加 DNAT iptables 以手动将集群 IP 映射到服务 IP，但它似乎没有帮助...尽管我不确定我是否将规则添加到正确的 iptable 链中。

编辑 1 -- 完整的 iptables

master ~ # iptables -S -t nat 
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N DOCKER
-N KUBE-MARK-DROP
-N KUBE-MARK-MASQ
-N KUBE-NODEPORTS
-N KUBE-POSTROUTING
-N KUBE-SEP-PE4UL45OLJLNLYYS
-N KUBE-SERVICES
-N KUBE-SVC-NPX46M4PTMTKRN6Y
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A PREROUTING -d 10.96.0.1/32 -p tcp -m tcp --dport 443 -j DNAT --to-destination 10.106.73.226:6443
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4000/0x4000 -j MASQUERADE
-A KUBE-SEP-PE4UL45OLJLNLYYS -s 10.106.73.226/32 -m comment --comment "default/kubernetes:https" -j KUBE-MARK-MASQ
-A KUBE-SEP-PE4UL45OLJLNLYYS -p tcp -m comment --comment "default/kubernetes:https" -m recent --set --name KUBE-SEP-PE4UL45OLJLNLYYS --mask 255.255.255.255 --rsource -m tcp -j DNAT --to-destination 10.106.73.226:6443
-A KUBE-SERVICES ! -s 10.244.0.0/16 -d 10.96.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.96.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y
-A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m recent --rcheck --seconds 10800 --reap --name KUBE-SEP-PE4UL45OLJLNLYYS --mask 255.255.255.255 --rsource -j KUBE-SEP-PE4UL45OLJLNLYYS
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -j KUBE-SEP-PE4UL45OLJLNLYYS

注意：我手动添加了规则 -A PREROUTING -d 10.96.0.1/32 -p tcp -m tcp --dport 443 -j DNAT --to-destination 10.106.73.226:6443 希望将 10.96.0.1 映射到 apiserver IP，但这并没有改变 curl 请求或 flannel pod 的行为

master 上 pods 的当前状态：

master ~ # kubectl get pods -o wide --all-namespaces
NAME                                  READY     STATUS              RESTARTS   AGE       IP              NODE
etcd-master                           1/1       Running             0          13d       10.106.73.226   master
kube-apiserver-master                 1/1       Running             0          13d       10.106.73.226   master
kube-controller-manager-master        1/1       Running             1          13d       10.106.73.226   master
kube-dns-86f4d74b45-dkzlk             0/3       ContainerCreating   0          13d       <none>          master
kube-flannel-ds-j5fxd                 0/1       CrashLoopBackOff    3550       13d       10.106.73.226   master
kube-proxy-pml47                      1/1       Running             0          13d       10.106.73.226   master
kube-scheduler-master                 1/1       Running             0          13d       10.106.73.226   master

Answer 1

您的所有设置看起来都不错，包括路线和systctl 值。

我唯一能猜到的是防火墙规则中的某个问题。请确保您接受Forward链中的流量转发。

你可以这样检查：

1. 停止 K8s。
2. 停止防火墙。
3. 停止泊坞窗。

4. 写入文件/var/lib/iptables/rules-save 内容（如果文件存在则覆盖）：

   *filter
   :INPUT ACCEPT [0:0]
   :FORWARD ACCEPT [0:0]
   :OUTPUT ACCEPT [0:0]
   

5. 启动防火墙。

6. 启动泊坞窗。
7. 启动 K8s。
8. 检查服务。

这是我能想象到的服务问题的唯一原因。