【发布时间】：2021-11-19 08:40:06
【问题描述】：
我要配置端口转发80->32181,443->30598。 32181 和 30598 是 k8s 入口控制器的 NodePort，我可以正确建立连接：
$ curl http://localhost:32181
<html>
<head><title>404 Not Found</title></head>
<body>
...
$ curl https://localhost:30598 -k
<html>
<head><title>404 Not Found</title></head>
<body>
...
我所做的是：
$ cat /proc/sys/net/ipv4/ip_forward
1
$ firewall-cmd --list-all
public (active)
target: default
icmp-block-inversion: no
interfaces: eth0
sources:
services: cockpit dhcpv6-client frp http https kube-apiserver kube-kubelet ssh
ports:
protocols:
forward: no
masquerade: yes
forward-ports:
port=80:proto=tcp:toport=32181:toaddr=
port=443:proto=tcp:toport=30598:toaddr=
source-ports:
icmp-blocks:
rich rules:
但我无法通过80 或443 访问我的nginx：
$ curl https://localhost:443 -k
curl: (7) Failed to connect to localhost port 443: Connection refused
以及更多信息：
centos：8.2 4.18.0-348.2.1.el8_5.x86_64
k8s: 1.22(带有calico(v3.21.0)网络插件)
防火墙：0.9.3
和 iptables 输出：
$ iptables -nvL -t nat --line-numbers
Chain PREROUTING (policy ACCEPT 51 packets, 2688 bytes)
num pkts bytes target prot opt in out source destination
1 51 2688 cali-PREROUTING all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:6gwbT8clXdHdC1b1 */
2 51 2688 KUBE-SERVICES all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service portals */
3 51 2688 DOCKER all -- * * 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL
Chain INPUT (policy ACCEPT 50 packets, 2648 bytes)
num pkts bytes target prot opt in out source destination
Chain POSTROUTING (policy ACCEPT 1872 packets, 112K bytes)
num pkts bytes target prot opt in out source destination
1 1894 114K cali-POSTROUTING all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:O3lYWMrLQYEMJtB5 */
2 1862 112K KUBE-POSTROUTING all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes postrouting rules */
3 0 0 MASQUERADE all -- * !docker0 172.17.0.0/16 0.0.0.0/0
Chain OUTPUT (policy ACCEPT 1922 packets, 116K bytes)
num pkts bytes target prot opt in out source destination
1 1894 114K cali-OUTPUT all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:tVnHkvAo15HuiPy0 */
2 1911 115K KUBE-SERVICES all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service portals */
3 758 45480 DOCKER all -- * * 0.0.0.0/0 !127.0.0.0/8 ADDRTYPE match dst-type LOCAL
Chain DOCKER (2 references)
num pkts bytes target prot opt in out source destination
1 0 0 RETURN all -- docker0 * 0.0.0.0/0 0.0.0.0/0
Chain KUBE-SERVICES (2 references)
num pkts bytes target prot opt in out source destination
1 0 0 KUBE-SVC-JD5MR3NA4I4DYORP tcp -- * * 0.0.0.0/0 10.96.0.10 /* kube-system/kube-dns:metrics cluster IP */ tcp dpt:9153
2 0 0 KUBE-SVC-Z6GDYMWE5TV2NNJN tcp -- * * 0.0.0.0/0 10.110.193.197 /* kubernetes-dashboard/dashboard-metrics-scraper cluster IP */ tcp dpt:8000
3 0 0 KUBE-SVC-NPX46M4PTMTKRN6Y tcp -- * * 0.0.0.0/0 10.96.0.1 /* default/kubernetes:https cluster IP */ tcp dpt:443
4 0 0 KUBE-SVC-EDNDUDH2C75GIR6O tcp -- * * 0.0.0.0/0 10.97.201.174 /* ingress-nginx/ingress-nginx-controller:https cluster IP */ tcp dpt:443
5 0 0 KUBE-SVC-EZYNCFY2F7N6OQA2 tcp -- * * 0.0.0.0/0 10.103.242.141 /* ingress-nginx/ingress-nginx-controller-admission:https-webhook cluster IP */ tcp dpt:443
6 0 0 KUBE-SVC-ERIFXISQEP7F7OF4 tcp -- * * 0.0.0.0/0 10.96.0.10 /* kube-system/kube-dns:dns-tcp cluster IP */ tcp dpt:53
7 0 0 KUBE-SVC-TCOU7JCQXEZGVUNU udp -- * * 0.0.0.0/0 10.96.0.10 /* kube-system/kube-dns:dns cluster IP */ udp dpt:53
8 0 0 KUBE-SVC-CEZPIJSAUFW5MYPQ tcp -- * * 0.0.0.0/0 10.97.166.112 /* kubernetes-dashboard/kubernetes-dashboard cluster IP */ tcp dpt:443
9 0 0 KUBE-SVC-H5K62VURUHBF7BRH tcp -- * * 0.0.0.0/0 10.104.154.95 /* lens-metrics/kube-state-metrics:metrics cluster IP */ tcp dpt:8080
10 0 0 KUBE-SVC-MOZMMOD3XZX35IET tcp -- * * 0.0.0.0/0 10.96.73.22 /* lens-metrics/prometheus:web cluster IP */ tcp dpt:80
11 0 0 KUBE-SVC-CG5I4G2RS3ZVWGLK tcp -- * * 0.0.0.0/0 10.97.201.174 /* ingress-nginx/ingress-nginx-controller:http cluster IP */ tcp dpt:80
12 1165 69528 KUBE-NODEPORTS all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service nodeports; NOTE: this must be the last rule in this chain */ ADDRTYPE match dst-type LOCAL
Chain KUBE-POSTROUTING (1 references)
num pkts bytes target prot opt in out source destination
1 1859 112K RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 mark match ! 0x4000/0x4000
2 3 180 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 MARK xor 0x4000
3 3 180 MASQUERADE all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service traffic requiring SNAT */ random-fully
Chain KUBE-MARK-DROP (0 references)
num pkts bytes target prot opt in out source destination
1 0 0 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 MARK or 0x8000
Chain KUBE-NODEPORTS (1 references)
num pkts bytes target prot opt in out source destination
1 2 120 KUBE-SVC-EDNDUDH2C75GIR6O tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* ingress-nginx/ingress-nginx-controller:https */ tcp dpt:30598
2 1 60 KUBE-SVC-CG5I4G2RS3ZVWGLK tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* ingress-nginx/ingress-nginx-controller:http */ tcp dpt:32181
Chain KUBE-MARK-MASQ (27 references)
num pkts bytes target prot opt in out source destination
1 3 180 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 MARK or 0x4000
Chain KUBE-SEP-IPE5TMLTCUYK646X (1 references)
num pkts bytes target prot opt in out source destination
1 0 0 KUBE-MARK-MASQ all -- * * 192.168.103.147 0.0.0.0/0 /* kube-system/kube-dns:metrics */
2 0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:metrics */ tcp to:192.168.103.147:9153
Chain KUBE-SEP-3LZLTHU4JT3FAVZK (1 references)
num pkts bytes target prot opt in out source destination
1 0 0 KUBE-MARK-MASQ all -- * * 192.168.103.149 0.0.0.0/0 /* kube-system/kube-dns:metrics */
2 0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:metrics */ tcp to:192.168.103.149:9153
Chain KUBE-SVC-JD5MR3NA4I4DYORP (1 references)
num pkts bytes target prot opt in out source destination
1 0 0 KUBE-MARK-MASQ tcp -- * * !192.168.0.0/16 10.96.0.10 /* kube-system/kube-dns:metrics cluster IP */ tcp dpt:9153
2 0 0 KUBE-SEP-IPE5TMLTCUYK646X all -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:metrics */ statistic mode random probability 0.50000000000
3 0 0 KUBE-SEP-3LZLTHU4JT3FAVZK all -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:metrics */
Chain KUBE-SEP-ZOAMCQDU54EOM4EJ (1 references)
num pkts bytes target prot opt in out source destination
1 0 0 KUBE-MARK-MASQ all -- * * 192.168.103.141 0.0.0.0/0 /* kubernetes-dashboard/dashboard-metrics-scraper */
2 0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes-dashboard/dashboard-metrics-scraper */ tcp to:192.168.103.141:8000
Chain KUBE-SVC-Z6GDYMWE5TV2NNJN (1 references)
num pkts bytes target prot opt in out source destination
1 0 0 KUBE-MARK-MASQ tcp -- * * !192.168.0.0/16 10.110.193.197 /* kubernetes-dashboard/dashboard-metrics-scraper cluster IP */ tcp dpt:8000
2 0 0 KUBE-SEP-ZOAMCQDU54EOM4EJ all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes-dashboard/dashboard-metrics-scraper */
Chain KUBE-SEP-HYE2IFAO6PORQFJR (1 references)
num pkts bytes target prot opt in out source destination
1 0 0 KUBE-MARK-MASQ all -- * * 192.168.0.176 0.0.0.0/0 /* default/kubernetes:https */
2 0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/kubernetes:https */ tcp to:192.168.0.176:6443
Chain KUBE-SVC-NPX46M4PTMTKRN6Y (1 references)
num pkts bytes target prot opt in out source destination
1 0 0 KUBE-MARK-MASQ tcp -- * * !192.168.0.0/16 10.96.0.1 /* default/kubernetes:https cluster IP */ tcp dpt:443
2 0 0 KUBE-SEP-HYE2IFAO6PORQFJR all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/kubernetes:https */
Chain KUBE-SEP-GJ4OJHBKIREWLMRS (1 references)
num pkts bytes target prot opt in out source destination
1 0 0 KUBE-MARK-MASQ all -- * * 192.168.103.146 0.0.0.0/0 /* ingress-nginx/ingress-nginx-controller:https */
2 2 120 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* ingress-nginx/ingress-nginx-controller:https */ tcp to:192.168.103.146:443
Chain KUBE-SVC-EDNDUDH2C75GIR6O (2 references)
num pkts bytes target prot opt in out source destination
1 0 0 KUBE-MARK-MASQ tcp -- * * !192.168.0.0/16 10.97.201.174 /* ingress-nginx/ingress-nginx-controller:https cluster IP */ tcp dpt:443
2 2 120 KUBE-MARK-MASQ tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* ingress-nginx/ingress-nginx-controller:https */ tcp dpt:30598
3 2 120 KUBE-SEP-GJ4OJHBKIREWLMRS all -- * * 0.0.0.0/0 0.0.0.0/0 /* ingress-nginx/ingress-nginx-controller:https */
Chain KUBE-SEP-K2CVHZPTBE2YAD6P (1 references)
num pkts bytes target prot opt in out source destination
1 0 0 KUBE-MARK-MASQ all -- * * 192.168.103.146 0.0.0.0/0 /* ingress-nginx/ingress-nginx-controller-admission:https-webhook */
2 0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* ingress-nginx/ingress-nginx-controller-admission:https-webhook */ tcp to:192.168.103.146:8443
Chain KUBE-SVC-EZYNCFY2F7N6OQA2 (1 references)
num pkts bytes target prot opt in out source destination
1 0 0 KUBE-MARK-MASQ tcp -- * * !192.168.0.0/16 10.103.242.141 /* ingress-nginx/ingress-nginx-controller-admission:https-webhook cluster IP */ tcp dpt:443
2 0 0 KUBE-SEP-K2CVHZPTBE2YAD6P all -- * * 0.0.0.0/0 0.0.0.0/0 /* ingress-nginx/ingress-nginx-controller-admission:https-webhook */
Chain KUBE-SEP-S6VTWHFP6KEYRW5L (1 references)
num pkts bytes target prot opt in out source destination
1 0 0 KUBE-MARK-MASQ all -- * * 192.168.103.147 0.0.0.0/0 /* kube-system/kube-dns:dns-tcp */
2 0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns-tcp */ tcp to:192.168.103.147:53
Chain KUBE-SEP-SFGZMYIS2CE4JD3K (1 references)
num pkts bytes target prot opt in out source destination
1 0 0 KUBE-MARK-MASQ all -- * * 192.168.103.149 0.0.0.0/0 /* kube-system/kube-dns:dns-tcp */
2 0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns-tcp */ tcp to:192.168.103.149:53
Chain KUBE-SVC-ERIFXISQEP7F7OF4 (1 references)
num pkts bytes target prot opt in out source destination
1 0 0 KUBE-MARK-MASQ tcp -- * * !192.168.0.0/16 10.96.0.10 /* kube-system/kube-dns:dns-tcp cluster IP */ tcp dpt:53
2 0 0 KUBE-SEP-S6VTWHFP6KEYRW5L all -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns-tcp */ statistic mode random probability 0.50000000000
3 0 0 KUBE-SEP-SFGZMYIS2CE4JD3K all -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns-tcp */
Chain KUBE-SEP-IJUMPPTQDLYXOX4B (1 references)
num pkts bytes target prot opt in out source destination
1 0 0 KUBE-MARK-MASQ all -- * * 192.168.103.147 0.0.0.0/0 /* kube-system/kube-dns:dns */
2 0 0 DNAT udp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns */ udp to:192.168.103.147:53
Chain KUBE-SEP-C4W6TKYY5HHEG4RV (1 references)
num pkts bytes target prot opt in out source destination
1 0 0 KUBE-MARK-MASQ all -- * * 192.168.103.149 0.0.0.0/0 /* kube-system/kube-dns:dns */
2 0 0 DNAT udp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns */ udp to:192.168.103.149:53
Chain KUBE-SVC-TCOU7JCQXEZGVUNU (1 references)
num pkts bytes target prot opt in out source destination
1 0 0 KUBE-MARK-MASQ udp -- * * !192.168.0.0/16 10.96.0.10 /* kube-system/kube-dns:dns cluster IP */ udp dpt:53
2 0 0 KUBE-SEP-IJUMPPTQDLYXOX4B all -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns */ statistic mode random probability 0.50000000000
3 0 0 KUBE-SEP-C4W6TKYY5HHEG4RV all -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns */
Chain KUBE-SEP-GX372II3CQAGUHFM (1 references)
num pkts bytes target prot opt in out source destination
1 0 0 KUBE-MARK-MASQ all -- * * 192.168.103.145 0.0.0.0/0 /* kubernetes-dashboard/kubernetes-dashboard */
2 0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes-dashboard/kubernetes-dashboard */ tcp to:192.168.103.145:8443
Chain KUBE-SVC-CEZPIJSAUFW5MYPQ (1 references)
num pkts bytes target prot opt in out source destination
1 0 0 KUBE-MARK-MASQ tcp -- * * !192.168.0.0/16 10.97.166.112 /* kubernetes-dashboard/kubernetes-dashboard cluster IP */ tcp dpt:443
2 0 0 KUBE-SEP-GX372II3CQAGUHFM all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes-dashboard/kubernetes-dashboard */
Chain KUBE-SEP-I3RZS3REJP7POFLG (1 references)
num pkts bytes target prot opt in out source destination
1 0 0 KUBE-MARK-MASQ all -- * * 192.168.103.143 0.0.0.0/0 /* lens-metrics/kube-state-metrics:metrics */
2 0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* lens-metrics/kube-state-metrics:metrics */ tcp to:192.168.103.143:8080
Chain KUBE-SVC-H5K62VURUHBF7BRH (1 references)
num pkts bytes target prot opt in out source destination
1 0 0 KUBE-MARK-MASQ tcp -- * * !192.168.0.0/16 10.104.154.95 /* lens-metrics/kube-state-metrics:metrics cluster IP */ tcp dpt:8080
2 0 0 KUBE-SEP-I3RZS3REJP7POFLG all -- * * 0.0.0.0/0 0.0.0.0/0 /* lens-metrics/kube-state-metrics:metrics */
Chain KUBE-SEP-ROTMHDCXAI3T7IOR (1 references)
num pkts bytes target prot opt in out source destination
1 0 0 KUBE-MARK-MASQ all -- * * 192.168.103.144 0.0.0.0/0 /* lens-metrics/prometheus:web */
2 0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* lens-metrics/prometheus:web */ tcp to:192.168.103.144:9090
Chain KUBE-SVC-MOZMMOD3XZX35IET (1 references)
num pkts bytes target prot opt in out source destination
1 0 0 KUBE-MARK-MASQ tcp -- * * !192.168.0.0/16 10.96.73.22 /* lens-metrics/prometheus:web cluster IP */ tcp dpt:80
2 0 0 KUBE-SEP-ROTMHDCXAI3T7IOR all -- * * 0.0.0.0/0 0.0.0.0/0 /* lens-metrics/prometheus:web */
Chain KUBE-SEP-OAYGOO6JHJEB65WC (1 references)
num pkts bytes target prot opt in out source destination
1 0 0 KUBE-MARK-MASQ all -- * * 192.168.103.146 0.0.0.0/0 /* ingress-nginx/ingress-nginx-controller:http */
2 1 60 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* ingress-nginx/ingress-nginx-controller:http */ tcp to:192.168.103.146:80
Chain KUBE-SVC-CG5I4G2RS3ZVWGLK (2 references)
num pkts bytes target prot opt in out source destination
1 0 0 KUBE-MARK-MASQ tcp -- * * !192.168.0.0/16 10.97.201.174 /* ingress-nginx/ingress-nginx-controller:http cluster IP */ tcp dpt:80
2 1 60 KUBE-MARK-MASQ tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* ingress-nginx/ingress-nginx-controller:http */ tcp dpt:32181
3 1 60 KUBE-SEP-OAYGOO6JHJEB65WC all -- * * 0.0.0.0/0 0.0.0.0/0 /* ingress-nginx/ingress-nginx-controller:http */
Chain KUBE-PROXY-CANARY (0 references)
num pkts bytes target prot opt in out source destination
Chain cali-nat-outgoing (1 references)
num pkts bytes target prot opt in out source destination
1 49 3274 MASQUERADE all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:flqWnvo8yq4ULQLa */ match-set cali40masq-ipam-pools src ! match-set cali40all-ipam-pools dst random-fully
Chain cali-POSTROUTING (1 references)
num pkts bytes target prot opt in out source destination
1 1894 114K cali-fip-snat all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:Z-c7XtVd2Bq7s_hA */
2 1894 114K cali-nat-outgoing all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:nYKhEzDlr11Jccal */
3 0 0 MASQUERADE all -- * tunl0 0.0.0.0/0 0.0.0.0/0 /* cali:SXWvdsbh4Mw7wOln */ ADDRTYPE match src-type !LOCAL limit-out ADDRTYPE match src-type LOCAL random-fully
Chain cali-PREROUTING (1 references)
num pkts bytes target prot opt in out source destination
1 51 2688 cali-fip-dnat all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:r6XmIziWUJsdOK6Z */
Chain cali-fip-snat (1 references)
num pkts bytes target prot opt in out source destination
Chain cali-OUTPUT (1 references)
num pkts bytes target prot opt in out source destination
1 1894 114K cali-fip-dnat all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:GBTAv2p5CwevEyJm */
Chain cali-fip-dnat (2 references)
num pkts bytes target prot opt in out source destination
Chain KUBE-KUBELET-CANARY (0 references)
num pkts bytes target prot opt in out source destination
【问题讨论】：
-
你是如何创建你的 nginx 服务的？你能提供你使用的命令吗？
-
Calico 也支持 NetworkPolicy，请检查您是否定义了一个拒绝这一点的定义。
kubectl get NetworkPolicy -
我使用 ingress-nginx 清单解决方案：
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.0.5/deploy/static/provider/baremetal/deploy.yaml，任何命名空间中都没有 NetworkPolicy。 -
你能禁用它并确保它有效吗？您是如何设置集群的？
-
嗯，我意识到的一件事是，当它转发到常规端口时，例如 NetCat 监听的随机端口，它工作得很好。只有当它转发到一个 k8s 服务 NodePort 时，才会有问题。最后，我放弃了端口转发的解决方案，并设置了一个外部 Nginx 作为 tcp 代理来避免这个问题:(。抱歉回复晚了。
标签： linux kubernetes networking firewalld