k8s svc 在其他工作节点中找不到 pod答案

【问题标题】：k8s svc can not find pod in other worker nodek8s svc 在其他工作节点中找不到 pod
【发布时间】：2019-07-03 02:35:50
【问题描述】：

我有两个相同的pod 在两个工作节点上运行，它们通过svc 在外部提供服务。如下：

root@master1:~# kubectl get pod -o wide
NAME          READY   STATUS    RESTARTS   AGE   IP            NODE
kubia-nwjcc   1/1     Running   0          33m   10.244.1.27   worker1
kubia-zcpbb   1/1     Running   0          33m   10.244.2.11   worker2

root@master1:~# kubectl get svc
NAME         TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1     <none>        443/TCP   26h
kubia        ClusterIP   10.98.41.49   <none>        80/TCP    34m

但是当我尝试访问pod 之一中的svc 时，我只能在同一节点上获得pod 的响应。当 svc 访问另一个节点上的pod 时。当svc 尝试访问其他节点上的pod 时，它会返回command terminated with exit code 7。正确输出和错误输出似乎是随机生成的，如下

正确输出

root@master1:~# k exec kubia-nwjcc -- curl http://10.98.41.49
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    23    0    23    0     0   8543      0 --:--:-- --:--:-- --:--:-- 11500
You've hit kubia-nwjcc

输出错误

root@master1:~# kubectl exec kubia-nwjcc -- curl http://10.98.41.49   
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (7) Failed to connect to 10.98.41.49 port 80: No route to host
command terminated with exit code 7

以下是我正在使用的软件版本：

ubuntu:v18.04
kubelet/kubeadm/kubectl:v1.15.0
码头工人：v18.09.5

以下是svc描述：

root@master1:~# kubectl describe svc kubia
Name:              kubia
Namespace:         default
Labels:            <none>
Annotations:       <none>
Selector:          app=kubia
Type:              ClusterIP
IP:                10.98.41.49
Port:              <unset>  80/TCP
TargetPort:        8080/TCP
Endpoints:         10.244.1.27:8080,10.244.2.11:8080
Session Affinity:  None
Events:            <none>

以下是使用-v=9的返回结果：

root@master1:~# kubectl exec kubia-nwjcc -v=9 -- curl -s http://10.98.41.49
I0702 11:45:52.481239   23171 loader.go:359] Config loaded from file:  /root/.kube/config
I0702 11:45:52.501154   23171 round_trippers.go:419] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.15.0 (linux/amd64) kubernetes/e8462b5" 'https://192.168.56.11:6443/api/v1/namespaces/default/pods/kubia-nwjcc'
I0702 11:45:52.525926   23171 round_trippers.go:438] GET https://192.168.56.11:6443/api/v1/namespaces/default/pods/kubia-nwjcc 200 OK in 24 milliseconds
I0702 11:45:52.525980   23171 round_trippers.go:444] Response Headers:
I0702 11:45:52.525992   23171 round_trippers.go:447]     Content-Type: application/json
I0702 11:45:52.526003   23171 round_trippers.go:447]     Content-Length: 2374
I0702 11:45:52.526012   23171 round_trippers.go:447]     Date: Tue, 02 Jul 2019 11:45:52 GMT
I0702 11:45:52.526063   23171 request.go:947] Response Body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"kubia-nwjcc","generateName":"kubia-","namespace":"default","selfLink":"/api/v1/namespaces/default/pods/kubia-nwjcc","uid":"2fd67789-c48d-4459-8b03-ac562b4a3f5c","resourceVersion":"188689","creationTimestamp":"2019-07-02T10:51:34Z","labels":{"app":"kubia"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"kubia","uid":"f3a4c457-dee4-4aec-ad73-1f0ca41628aa","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"default-token-6pgh8","secret":{"secretName":"default-token-6pgh8","defaultMode":420}}],"containers":[{"name":"kubia","image":"luksa/kubia","ports":[{"containerPort":8080,"protocol":"TCP"}],"resources":{},"volumeMounts":[{"name":"default-token-6pgh8","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"Always"}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"worker1","securityContext":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2019-07-03T01:35:15Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2019-07-03T01:35:20Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2019-07-03T01:35:20Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2019-07-02T10:51:34Z"}],"hostIP":"192.168.56.21","podIP":"10.244.1.27","startTime":"2019-07-03T01:35:15Z","containerStatuses":[{"name":"kubia","state":{"running":{"startedAt":"2019-07-03T01:35:19Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"luksa/kubia:latest","imageID":"docker-pullable://luksa/kubia@sha256:3f28e304dc0f63dc30f273a4202096f0fa0d08510bd2ee7e1032ce600616de24","containerID":"docker://27da556930baf857e5af92b13934dcb1b2b2f001ecab5e7b952b2bda5aa27f0b"}],"qosClass":"BestEffort"}}
I0702 11:45:52.543108   23171 round_trippers.go:419] curl -k -v -XPOST  -H "X-Stream-Protocol-Version: v4.channel.k8s.io" -H "X-Stream-Protocol-Version: v3.channel.k8s.io" -H "X-Stream-Protocol-Version: v2.channel.k8s.io" -H "X-Stream-Protocol-Version: channel.k8s.io" -H "User-Agent: kubectl/v1.15.0 (linux/amd64) kubernetes/e8462b5" 'https://192.168.56.11:6443/api/v1/namespaces/default/pods/kubia-nwjcc/exec?command=curl&command=-s&command=http%3A%2F%2F10.98.41.49&container=kubia&stderr=true&stdout=true'
I0702 11:45:52.591166   23171 round_trippers.go:438] POST https://192.168.56.11:6443/api/v1/namespaces/default/pods/kubia-nwjcc/exec?command=curl&command=-s&command=http%3A%2F%2F10.98.41.49&container=kubia&stderr=true&stdout=true 101 Switching Protocols in 47 milliseconds
I0702 11:45:52.591208   23171 round_trippers.go:444] Response Headers:
I0702 11:45:52.591217   23171 round_trippers.go:447]     Connection: Upgrade
I0702 11:45:52.591221   23171 round_trippers.go:447]     Upgrade: SPDY/3.1
I0702 11:45:52.591225   23171 round_trippers.go:447]     X-Stream-Protocol-Version: v4.channel.k8s.io
I0702 11:45:52.591229   23171 round_trippers.go:447]     Date: Wed, 03 Jul 2019 02:29:33 GMT
F0702 11:45:53.783725   23171 helpers.go:114] command terminated with exit code 7

kube-system pod 和提供服务的两个 pod 的状态都是Running，如下：

root@master1:~/k8s-yaml# kubectl get --all-namespaces pod -o wide
NAMESPACE     NAME                              READY   STATUS    RESTARTS   AGE     IP              NODE      NOMINATED NODE   READINESS GATES
default       kubia-6pjz9                       1/1     Running   0          5m35s   10.244.2.12     worker2   <none>           <none>
default       kubia-nwjcc                       1/1     Running   0          16h     10.244.1.27     worker1   <none>           <none>
kube-system   coredns-bccdc95cf-792px           1/1     Running   4          5d19h   10.244.0.11     master1   <none>           <none>
kube-system   coredns-bccdc95cf-bc76j           1/1     Running   4          5d19h   10.244.0.10     master1   <none>           <none>
kube-system   etcd-master1                      1/1     Running   8          5d19h   192.168.56.11   master1   <none>           <none>
kube-system   kube-apiserver-master1            1/1     Running   7          5d19h   192.168.56.11   master1   <none>           <none>
kube-system   kube-controller-manager-master1   1/1     Running   7          5d18h   192.168.56.11   master1   <none>           <none>
kube-system   kube-flannel-ds-amd64-9trbq       1/1     Running   3          5d18h   192.168.56.21   worker1   <none>           <none>
kube-system   kube-flannel-ds-amd64-btt74       1/1     Running   5          5d18h   192.168.56.11   master1   <none>           <none>
kube-system   kube-flannel-ds-amd64-swzzd       1/1     Running   0          3d20h   192.168.56.22   worker2   <none>           <none>
kube-system   kube-proxy-27zfk                  1/1     Running   7          5d19h   192.168.56.11   master1   <none>           <none>
kube-system   kube-proxy-79kt9                  1/1     Running   0          3d20h   192.168.56.22   worker2   <none>           <none>
kube-system   kube-proxy-lx4gk                  1/1     Running   3          5d18h   192.168.56.21   worker1   <none>           <none>
kube-system   kube-scheduler-master1            1/1     Running   7          5d19h   192.168.56.11   master1   <none>           <none>

我的预期回报是：当使用curl访问svc时，svc可以在不同节点上随机访问pod如下：

root@master1:~# kubectl get po
NAME          READY   STATUS    RESTARTS   AGE
kubia-nwjcc   1/1     Running   0          50m
kubia-zcpbb   1/1     Running   0          50m
root@master1:~# k exec kubia-nwjcc -- curl -s http://10.98.41.49
You've hit kubia-nwjcc
root@master1:~# k exec kubia-nwjcc -- curl -s http://10.98.41.49
You've hit kubia-zcpbb
root@master1:~# k exec kubia-nwjcc -- curl -s http://10.98.41.49
You've hit kubia-zcpbb
root@master1:~# k exec kubia-nwjcc -- curl -s http://10.98.41.49
You've hit kubia-nwjcc

现在五次测试的返回结果如下：

root@master1:~# k exec kubia-nwjcc -- curl -s http://10.98.41.49
You've hit kubia-nwjcc
root@master1:~# k exec kubia-nwjcc -- curl -s http://10.98.41.49
You've hit kubia-nwjcc
root@master1:~# k exec kubia-nwjcc -- curl -s http://10.98.41.49
command terminated with exit code 7
root@master1:~# k exec kubia-nwjcc -- curl -s http://10.98.41.49
You've hit kubia-nwjcc
root@master1:~# k exec kubia-nwjcc -- curl -s http://10.98.41.49
command terminated with exit code 7

【问题讨论】：

嗨 hopgoldy，欢迎来到 SO。请包含来自kubectl get --all-namespaces pod -o wide 的输出，因为我非常怀疑您的 CNI 丢失、配置错误或位于CrashLoopBackOff/Unready
hi thx，我执行了这个命令，但是结果全部返回到Running。写得太小了，我把问题返回的具体结果加进去了，再次感谢您的回复。
我试图绕过svc并直接使用k exec kubia-nwjcc -- curl -s http://10.244.1.27:8080访问另一个节点上的pod（10.244.1.27是另一个pod ip），但它仍然无法使用return command terminated with exit code 7，是不是两个节点之间的网络有问题？
您可以使用kubectl -it exec kubia-6pjz9 -- curl http://localhost:8080 尝试从同一个 pod 执行 exec 吗？ kubia-6pjz9 - 是无法访问的 pod。然后尝试使用 IP 地址 - kubectl -it exec kubia-6pjz9 -- curl http://10.244.1.27:8080。同时发布 pod kubia-6pjz9 的日志。
经过测试，我认为我的CNI网络有问题。节点通过网卡2访问，我认为flanneld使用默认的网卡1。那么如何安全健康地修改flanneld的设置呢？

标签： curl kubernetes

【解决方案1】：

这个问题解决了。 flannel的官方文档中提到在vagrant类型的虚拟机中运行时需要使用--iface指定要使用的网卡。您可以使用命令kubectl edit daemonset kube-flannel-ds-amd64 -n kube-system 编辑fllannel 配置文件。然后对所有法兰绒吊舱使用kubectl delete pod -n kube-system <pod-name>。 K8s 将重建它们。

您可以在Kubernetes with Flannel和flannel - troubleshooting找到详细答案。

【讨论】：