【发布时间】：2020-03-16 21:21:03
【问题描述】：
我已将基于 CentOS 7 的 Kubernetes 集群的主节点从 1.15 版本更新到 1.16.8。我关注了官方manual。
更新成功完成后，我重启了master。从那时起，我在每个 kubectl 命令中都会收到一条消息，表明 API 不可用。 kubelet 的系统信息给了我同样的信息。
$ kubectl get pods
The connection to the server 10.6.231.50:6443 was refused - did you specify the right host or port?
$
$
$ journalctl -u kubelet.service -f
-- Logs begin at Fri 2020-01-31 13:50:09 CET. --
Jan 31 16:52:40 orbisos010 kubelet[14818]: E0131 16:52:40.137787 14818 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/kubelet.go:459: Failed to list *v1.Node: Get https://10.6.231.50:6443/api/v1/nodes?fieldSelector=metadata.name%3Dorbisos010&limit=500&resourceVersion=0: dial tcp 10.6.231.50:6443: connect: connection refused
Jan 31 16:52:41 orbisos010 kubelet[14818]: E0131 16:52:41.130807 14818 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/kubelet.go:450: Failed to list *v1.Service: Get https://10.6.231.50:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.6.231.50:6443: connect: connection refused
Jan 31 16:52:41 orbisos010 kubelet[14818]: E0131 16:52:41.137195 14818 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://10.6.231.50:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dorbisos010&limit=500&resourceVersion=0: dial tcp 10.6.231.50:6443: connect: connection refused
Jan 31 16:52:41 orbisos010 kubelet[14818]: E0131 16:52:41.138917 14818 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/kubelet.go:459: Failed to list *v1.Node: Get https://10.6.231.50:6443/api/v1/nodes?fieldSelector=metadata.name%3Dorbisos010&limit=500&resourceVersion=0: dial tcp 10.6.231.50:6443: connect: connection refused
Jan 31 16:52:42 orbisos010 kubelet[14818]: E0131 16:52:42.132181 14818 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/kubelet.go:450: Failed to list *v1.Service: Get https://10.6.231.50:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.6.231.50:6443: connect: connection refused
Jan 31 16:52:42 orbisos010 kubelet[14818]: E0131 16:52:42.138261 14818 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://10.6.231.50:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dorbisos010&limit=500&resourceVersion=0: dial tcp 10.6.231.50:6443: connect: connection refused
我已经用以下命令配置了master：
$ kubeadm init \
--apiserver-advertise-address=10.6.231.50 \
--pod-network-cidr=10.244.0.0/16 \
--service-cidr=10.96.0.0/12 \
--service-dns-domain=k8s.example.local
另外，我在/etc/sysconfig/kubeletEXTRA_ARGS 中设置了将crio 定义为容器运行时，将systemd 定义为cgroups 驱动程序。
KUBELET_EXTRA_ARGS=--container-runtime-endpoint=unix:///var/run/crio/crio.sock --cgroup-driver=systemd --runtime-cgroups=/systemd/system.slice --kubelet-cgroups=/systemd/system.slice
已安装 crio 1.13.11 版。
$ crio --version
crio version 1.13.11
编辑：交换分区被禁用/删除
$ cat /proc/swaps
Filename Type Size Used Priority
$ cat /etc/fstab
#
# /etc/fstab
# Created by anaconda on Fri Apr 5 09:20:29 2019
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=1725c823-3764-4e25-ba35-c638d6d34335 / ext4 defaults 1 1
编辑：升级后容器当前未运行 - k8s-api-server 日志
$ crictl logs 7c8d8481003b9
I0313 07:35:07.782793 1 main.go:514] Determining IP address of default interface
I0313 07:35:07.785403 1 main.go:527] Using interface with name eth0 and address 10.6.231.50
I0313 07:35:07.785455 1 main.go:544] Defaulting external address to interface address (10.6.231.50)
I0313 07:35:07.900430 1 kube.go:126] Waiting 10m0s for node controller to sync
I0313 07:35:07.900538 1 kube.go:309] Starting kube subnet manager
I0313 07:35:08.901017 1 kube.go:133] Node controller sync successful
I0313 07:35:08.901094 1 main.go:244] Created subnet manager: Kubernetes Subnet Manager - orbisos010
I0313 07:35:08.901112 1 main.go:247] Installing signal handlers
I0313 07:35:08.903298 1 main.go:386] Found network config - Backend type: vxlan
I0313 07:35:08.903501 1 vxlan.go:120] VXLAN config: VNI=1 Port=0 GBP=false DirectRouting=false
I0313 07:35:08.996070 1 main.go:351] Current network or subnet (10.244.0.0/16, 10.244.0.0/24) is not equal to previous one (0.0.0.0/0, 0.0.0.0/0), trying to recycle old iptables rules
I0313 07:35:09.005711 1 iptables.go:167] Deleting iptables rule: -s 0.0.0.0/0 -d 0.0.0.0/0 -j RETURN
I0313 07:35:09.007865 1 iptables.go:167] Deleting iptables rule: -s 0.0.0.0/0 ! -d 224.0.0.0/4 -j MASQUERADE --random-fully
I0313 07:35:09.082996 1 iptables.go:167] Deleting iptables rule: ! -s 0.0.0.0/0 -d 0.0.0.0/0 -j RETURN
I0313 07:35:09.084506 1 iptables.go:167] Deleting iptables rule: ! -s 0.0.0.0/0 -d 0.0.0.0/0 -j MASQUERADE --random-fully
I0313 07:35:09.087178 1 main.go:317] Wrote subnet file to /run/flannel/subnet.env
I0313 07:35:09.087192 1 main.go:321] Running backend.
I0313 07:35:09.087202 1 main.go:339] Waiting for all goroutines to exit
I0313 07:35:09.087227 1 vxlan_network.go:60] watching for new subnet leases
I0313 07:35:09.183701 1 iptables.go:145] Some iptables rules are missing; deleting and recreating rules
I0313 07:35:09.183790 1 iptables.go:167] Deleting iptables rule: -s 10.244.0.0/16 -d 10.244.0.0/16 -j RETURN
I0313 07:35:09.282337 1 iptables.go:167] Deleting iptables rule: -s 10.244.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE --random-fully
I0313 07:35:09.282804 1 iptables.go:145] Some iptables rules are missing; deleting and recreating rules
I0313 07:35:09.282825 1 iptables.go:167] Deleting iptables rule: -s 10.244.0.0/16 -j ACCEPT
I0313 07:35:09.285677 1 iptables.go:167] Deleting iptables rule: ! -s 10.244.0.0/16 -d 10.244.0.0/24 -j RETURN
I0313 07:35:09.286809 1 iptables.go:167] Deleting iptables rule: -d 10.244.0.0/16 -j ACCEPT
I0313 07:35:09.288137 1 iptables.go:167] Deleting iptables rule: ! -s 10.244.0.0/16 -d 10.244.0.0/16 -j MASQUERADE --random-fully
I0313 07:35:09.289182 1 iptables.go:155] Adding iptables rule: -s 10.244.0.0/16 -j ACCEPT
I0313 07:35:09.382765 1 iptables.go:155] Adding iptables rule: -s 10.244.0.0/16 -d 10.244.0.0/16 -j RETURN
I0313 07:35:09.488057 1 iptables.go:155] Adding iptables rule: -d 10.244.0.0/16 -j ACCEPT
I0313 07:35:09.491426 1 iptables.go:155] Adding iptables rule: -s 10.244.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE --random-fully
I0313 07:35:09.589312 1 iptables.go:155] Adding iptables rule: ! -s 10.244.0.0/16 -d 10.244.0.0/24 -j RETURN
I0313 07:35:09.687180 1 iptables.go:155] Adding iptables rule: ! -s 10.244.0.0/16 -d 10.244.0.0/16 -j MASQUERADE --random-fully
E0316 07:20:25.108176 1 reflector.go:304] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to watch *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=6128136&timeoutSeconds=450&watch=true: dial tcp 10.96.0.1:443: connect: connection refused
E0316 07:20:26.112371 1 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to list *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.96.0.1:443: connect: connection refused
E0316 07:50:17.738298 1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=11, ErrCode=NO_ERROR, debug=""
E0316 07:50:17.739007 1 reflector.go:304] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to watch *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=6131353&timeoutSeconds=510&watch=true: dial tcp 10.96.0.1:443: connect: connection refused
E0316 07:50:18.739790 1 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to list *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.96.0.1:443: connect: connection refused
E0316 07:50:19.741013 1 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to list *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.96.0.1:443: connect: network is unreachable
E0316 07:50:20.742425 1 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to list *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.96.0.1:443: connect: network is unreachable
E0316 07:50:21.748815 1 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to list *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.96.0.1:443: connect: network is unreachable
E0316 07:50:22.751387 1 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to list *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.96.0.1:443: connect: network is unreachable
E0316 07:50:23.752835 1 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to list *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.96.0.1:443: connect: network is unreachable
E0316 07:50:24.754022 1 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to list *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.96.0.1:443: connect: network is unreachable
E0316 07:50:25.755338 1 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to list *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.96.0.1:443: connect: network is unreachable
E0316 07:50:26.756589 1 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to list *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.96.0.1:443: connect: network is unreachable
E0316 07:50:27.757552 1 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to list *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.96.0.1:443: connect: network is unreachable
E0316 07:50:28.759233 1 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to list *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.96.0.1:443: connect: network is unreachable
E0316 07:50:29.760553 1 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to list *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.96.0.1:443: connect: network is unreachable
E0316 07:50:30.761582 1 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to list *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.96.0.1:443: connect: network is unreachable
E0316 07:50:31.762942 1 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to list *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.96.0.1:443: connect: network is unreachable
E0316 07:50:32.764278 1 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to list *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.96.0.1:443: connect: network is unreachable
E0316 07:50:33.765529 1 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to list *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.96.0.1:443: connect: network is unreachable
E0316 07:50:34.782413 1 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to list *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.96.0.1:443: connect: network is unreachable
E0316 07:50:35.783639 1 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to list *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.96.0.1:443: connect: network is unreachable
E0316 07:50:36.785241 1 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to list *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.96.0.1:443: connect: network is unreachable
E0316 07:50:37.787396 1 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to list *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.96.0.1:443: connect: network is unreachable
I0316 07:50:37.884172 1 main.go:370] shutdownHandler sent cancel signal...
E0316 07:50:38.788705 1 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to list *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.96.0.1:443: connect: network is unreachable
E0316 07:50:39.790020 1 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to list *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.96.0.1:443: connect: network is unreachable
E0316 07:50:40.791467 1 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to list *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.96.0.1:443: connect: network is unreachable
E0316 07:50:41.792689 1 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to list *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.96.0.1:443: connect: network is unreachable
E0316 07:50:42.793788 1 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to list *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.96.0.1:443: connect: network is unreachable
E0316 07:50:43.795046 1 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to list *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.96.0.1:443: connect: network is unreachable
E0316 07:50:44.797485 1 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to list *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.96.0.1:443: connect: network is unreachable
E0316 07:50:45.799345 1 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to list *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.96.0.1:443: connect: network is unreachable
E0316 07:50:46.800897 1 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to list *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.96.0.1:443: connect: network is unreachable
E0316 07:50:47.801961 1 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to list *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.96.0.1:443: connect: network is unreachable
任何想法为什么我无法连接到 10.6.231.50:6443？版本 1.16.8 中是否有任何重大更改？
沃尔克
【问题讨论】：
标签： kubernetes updates kubeadm