容器间通信（TCP 和 UDP）将网关 IP 显示为源 IP答案

【问题标题】：Inter-container communication (TCP and UDP) shows gateway IP as source IP容器间通信（TCP 和 UDP）将网关 IP 显示为源 IP
【发布时间】：2020-11-29 10:41:15
【问题描述】：

我有一个 docker 版本 18.09.2 的 docker 主机。当我尝试在两个容器之间使用 TCP 通信时，在目标主机上，源 IP 始终显示网关的 IP，而不是实际的源主机。例如：

Container1（IP为172.17.0.2）：

# docker run -itd --name h1 --hostname h1 centos /bin/bash
7031102d30f669388b4938fa953172762ec3bfb3ab1a6422b53aa3b0857c85e8

Container2（IP为172.17.0.3）：

# docker run -itd --name h2 --hostname h2 centos /bin/bash
3598edc19987680c208d8d540d4f774e2c68c7f4689611c845ccf94f99aeb156

在 container1 上，向 container2 发起 TCP 连接：

[h1 #] nc 172.17.0.3 9000

container1 上的 tcpdump 显示了预期的 src 和 dst IP：

[h1 #] tcpdump -nn -i any port 9000                                                                                                                                                 
18:03:26.138787 IP 172.17.0.2.43244 > 172.17.0.3.9000: Flags [S], seq 218108783, win 29200, options [mss 1460,sackOK,TS val 172963671 ecr 0,nop,wscale 7], length 0
18:03:26.138886 IP 172.17.0.3.9000 > 172.17.0.2.43244: Flags [R.], seq 0, ack 218108784, win 0, length 0

但是，在 container2 上，src IP 是网关：

[h2 #] tcpdump -nn -i any port 9000                                                                                                                                                  
18:03:26.138822 IP 172.17.0.1.43244 > 172.17.0.3.9000: Flags [S], seq 218108783, win 29200, options [mss 1460,sackOK,TS val 172963671 ecr 0,nop,wscale 7], length 0
18:03:26.138872 IP 172.17.0.3.9000 > 172.17.0.1.43244: Flags [R.], seq 0, ack 218108784, win 0, length 0

这是默认的桥接网络配置：

# docker network inspect bridge
[
    {
        "Name": "bridge",
        "Id": "d927b5b9a76fdc5ed562ae035730625cbc9edbb02f659a00ac0384214828d67b",
        "Created": "2020-08-10T01:55:54.480991622+08:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.17.0.0/16",
                    "Gateway": "172.17.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "3598edc19987680c208d8d540d4f774e2c68c7f4689611c845ccf94f99aeb156": {
                "Name": "h2",
                "EndpointID": "66e696e9fada081773ae577ae5367d7d5f4df3665a8a0e2be6318993a7b87807",
                "MacAddress": "02:42:ac:11:00:03",
                "IPv4Address": "172.17.0.3/16",
                "IPv6Address": ""
            },
            "7031102d30f669388b4938fa953172762ec3bfb3ab1a6422b53aa3b0857c85e8": {
                "Name": "h1",
                "EndpointID": "52b1ab4ce9a6bc5136db13c44bc9913fe31189863d44595d172373a222786c90",
                "MacAddress": "02:42:ac:11:00:02",
                "IPv4Address": "172.17.0.2/16",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.bridge.default_bridge": "true",
            "com.docker.network.bridge.enable_icc": "true",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
            "com.docker.network.bridge.name": "docker0",
            "com.docker.network.driver.mtu": "1500"
        },
        "Labels": {}
    }
]

我尝试使用不同的图像，但结果都相同。有什么想法吗？

谢谢啦

附加我的 iptables 规则：

[docker_host #] iptables -n -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0            udp dpt:53
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:53
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0            udp dpt:67
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:67
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0            udp dpt:53
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:53
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0            udp dpt:67
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:67
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0
INPUT_direct  all  --  0.0.0.0/0            0.0.0.0/0
INPUT_ZONES_SOURCE  all  --  0.0.0.0/0            0.0.0.0/0
INPUT_ZONES  all  --  0.0.0.0/0            0.0.0.0/0
DROP       all  --  0.0.0.0/0            0.0.0.0/0            ctstate INVALID
REJECT     all  --  0.0.0.0/0            0.0.0.0/0            reject-with icmp-host-prohibited

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
DOCKER-USER  all  --  0.0.0.0/0            0.0.0.0/0
DOCKER-ISOLATION-STAGE-1  all  --  0.0.0.0/0            0.0.0.0/0
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
DOCKER     all  --  0.0.0.0/0            0.0.0.0/0
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0
ACCEPT     all  --  0.0.0.0/0            192.168.122.0/24     ctstate RELATED,ESTABLISHED
ACCEPT     all  --  192.168.122.0/24     0.0.0.0/0
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0
REJECT     all  --  0.0.0.0/0            0.0.0.0/0            reject-with icmp-port-unreachable
REJECT     all  --  0.0.0.0/0            0.0.0.0/0            reject-with icmp-port-unreachable
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0
REJECT     all  --  0.0.0.0/0            0.0.0.0/0            reject-with icmp-port-unreachable
REJECT     all  --  0.0.0.0/0            0.0.0.0/0            reject-with icmp-port-unreachable
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0
FORWARD_direct  all  --  0.0.0.0/0            0.0.0.0/0
FORWARD_IN_ZONES_SOURCE  all  --  0.0.0.0/0            0.0.0.0/0
FORWARD_IN_ZONES  all  --  0.0.0.0/0            0.0.0.0/0
FORWARD_OUT_ZONES_SOURCE  all  --  0.0.0.0/0            0.0.0.0/0
FORWARD_OUT_ZONES  all  --  0.0.0.0/0            0.0.0.0/0
DROP       all  --  0.0.0.0/0            0.0.0.0/0            ctstate INVALID
REJECT     all  --  0.0.0.0/0            0.0.0.0/0            reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0            udp dpt:68
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0            udp dpt:68
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0
OUTPUT_direct  all  --  0.0.0.0/0            0.0.0.0/0

Chain DOCKER (1 references)
target     prot opt source               destination

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination
DOCKER-ISOLATION-STAGE-2  all  --  0.0.0.0/0            0.0.0.0/0
DROP       all  -- !10.70.0.0/16         0.0.0.0/0
DROP       all  --  0.0.0.0/0           !10.70.0.0/16
RETURN     all  --  0.0.0.0/0            0.0.0.0/0

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
target     prot opt source               destination
DROP       all  --  0.0.0.0/0            0.0.0.0/0
RETURN     all  --  0.0.0.0/0            0.0.0.0/0

Chain DOCKER-USER (1 references)
target     prot opt source               destination
RETURN     all  --  0.0.0.0/0            0.0.0.0/0

Chain FORWARD_IN_ZONES (1 references)
target     prot opt source               destination
FWDI_public  all  --  0.0.0.0/0            0.0.0.0/0           [goto]
FWDI_public  all  --  0.0.0.0/0            0.0.0.0/0           [goto]
FWDI_public  all  --  0.0.0.0/0            0.0.0.0/0           [goto]

Chain FORWARD_IN_ZONES_SOURCE (1 references)
target     prot opt source               destination

Chain FORWARD_OUT_ZONES (1 references)
target     prot opt source               destination
FWDO_public  all  --  0.0.0.0/0            0.0.0.0/0           [goto]
FWDO_public  all  --  0.0.0.0/0            0.0.0.0/0           [goto]
FWDO_public  all  --  0.0.0.0/0            0.0.0.0/0           [goto]

Chain FORWARD_OUT_ZONES_SOURCE (1 references)
target     prot opt source               destination
Chain FORWARD_direct (1 references)
target     prot opt source               destination

Chain FWDI_public (3 references)
target     prot opt source               destination
FWDI_public_log  all  --  0.0.0.0/0            0.0.0.0/0
FWDI_public_deny  all  --  0.0.0.0/0            0.0.0.0/0
FWDI_public_allow  all  --  0.0.0.0/0            0.0.0.0/0
ACCEPT     icmp --  0.0.0.0/0            0.0.0.0/0

Chain FWDI_public_allow (1 references)
target     prot opt source               destination

Chain FWDI_public_deny (1 references)
target     prot opt source               destination

Chain FWDI_public_log (1 references)
target     prot opt source               destination

Chain FWDO_public (3 references)
target     prot opt source               destination
FWDO_public_log  all  --  0.0.0.0/0            0.0.0.0/0
FWDO_public_deny  all  --  0.0.0.0/0            0.0.0.0/0
FWDO_public_allow  all  --  0.0.0.0/0            0.0.0.0/0

Chain FWDO_public_allow (1 references)
target     prot opt source               destination
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            ctstate NEW,UNTRACKED
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            ctstate NEW,UNTRACKED

Chain FWDO_public_deny (1 references)
target     prot opt source               destination

Chain FWDO_public_log (1 references)
target     prot opt source               destination

Chain INPUT_ZONES (1 references)
target     prot opt source               destination
IN_public  all  --  0.0.0.0/0            0.0.0.0/0           [goto]
IN_public  all  --  0.0.0.0/0            0.0.0.0/0           [goto]
IN_public  all  --  0.0.0.0/0            0.0.0.0/0           [goto]

Chain INPUT_ZONES_SOURCE (1 references)
target     prot opt source               destination

Chain INPUT_direct (1 references)
target     prot opt source               destination

Chain IN_public (3 references)
target     prot opt source               destination
IN_public_log  all  --  0.0.0.0/0            0.0.0.0/0
IN_public_deny  all  --  0.0.0.0/0            0.0.0.0/0
IN_public_allow  all  --  0.0.0.0/0            0.0.0.0/0
ACCEPT     icmp --  0.0.0.0/0            0.0.0.0/0

Chain IN_public_allow (1 references)
target     prot opt source               destination
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:22 ctstate NEW,UNTRACKED

Chain IN_public_deny (1 references)
target     prot opt source               destination

Chain IN_public_log (1 references)
target     prot opt source               destination

Chain OUTPUT_direct (1 references)
target     prot opt source               destination

[docker_host #] iptables -t nat -n -L
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
PREROUTING_direct  all  --  0.0.0.0/0            0.0.0.0/0
PREROUTING_ZONES_SOURCE  all  --  0.0.0.0/0            0.0.0.0/0
PREROUTING_ZONES  all  --  0.0.0.0/0            0.0.0.0/0
DOCKER     all  --  0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
OUTPUT_direct  all  --  0.0.0.0/0            0.0.0.0/0
DOCKER     all  --  0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
MASQUERADE  all  --  0.0.0.0/0            0.0.0.0/0            ADDRTYPE match src-type LOCAL
MASQUERADE  all  --  172.17.0.0/16        0.0.0.0/0
RETURN     all  --  192.168.122.0/24     224.0.0.0/24
RETURN     all  --  192.168.122.0/24     255.255.255.255
MASQUERADE  tcp  --  192.168.122.0/24    !192.168.122.0/24     masq ports: 1024-65535
MASQUERADE  udp  --  192.168.122.0/24    !192.168.122.0/24     masq ports: 1024-65535
MASQUERADE  all  --  192.168.122.0/24    !192.168.122.0/24
POSTROUTING_direct  all  --  0.0.0.0/0            0.0.0.0/0
POSTROUTING_ZONES_SOURCE  all  --  0.0.0.0/0            0.0.0.0/0
POSTROUTING_ZONES  all  --  0.0.0.0/0            0.0.0.0/0

Chain DOCKER (2 references)
target     prot opt source               destination

Chain OUTPUT_direct (1 references)
target     prot opt source               destination

Chain POSTROUTING_ZONES (1 references)
target     prot opt source               destination
POST_public  all  --  0.0.0.0/0            0.0.0.0/0           [goto]
POST_public  all  --  0.0.0.0/0            0.0.0.0/0           [goto]
POST_public  all  --  0.0.0.0/0            0.0.0.0/0           [goto]

Chain POSTROUTING_ZONES_SOURCE (1 references)
target     prot opt source               destination

Chain POSTROUTING_direct (1 references)
target     prot opt source               destination

Chain POST_public (3 references)
target     prot opt source               destination
POST_public_log  all  --  0.0.0.0/0            0.0.0.0/0
POST_public_deny  all  --  0.0.0.0/0            0.0.0.0/0
POST_public_allow  all  --  0.0.0.0/0            0.0.0.0/0

Chain POST_public_allow (1 references)
target     prot opt source               destination
MASQUERADE  all  --  0.0.0.0/0            0.0.0.0/0
MASQUERADE  all  --  0.0.0.0/0            0.0.0.0/0

Chain POST_public_deny (1 references)
target     prot opt source               destination

Chain POST_public_log (1 references)
target     prot opt source               destination

Chain PREROUTING_ZONES (1 references)
target     prot opt source               destination
PRE_public  all  --  0.0.0.0/0            0.0.0.0/0           [goto]
PRE_public  all  --  0.0.0.0/0            0.0.0.0/0           [goto]
PRE_public  all  --  0.0.0.0/0            0.0.0.0/0           [goto]

Chain PREROUTING_ZONES_SOURCE (1 references)
target     prot opt source               destination

Chain PREROUTING_direct (1 references)
target     prot opt source               destination

Chain PRE_public (3 references)
target     prot opt source               destination
PRE_public_log  all  --  0.0.0.0/0            0.0.0.0/0
PRE_public_deny  all  --  0.0.0.0/0            0.0.0.0/0
PRE_public_allow  all  --  0.0.0.0/0            0.0.0.0/0

Chain PRE_public_allow (1 references)
target     prot opt source               destination

Chain PRE_public_deny (1 references)
target     prot opt source               destination

Chain PRE_public_log (1 references)
target     prot opt source               destination

【问题讨论】：

1- 您使用的是哪个操作系统？ 2- 容器内的ip route 给了你什么？
@gmolaire docker 主机是 RHEL7.7 映像是 RHEL7.3 [h1 #] ip route default via 172.17.0.1 dev eth0 172.17.0.0/16 dev eth0 proto kernel scope link src 172.17.0.2 [h2 #] ip route default via 172.17.0.1 dev eth0 172.17.0.0/16 dev eth0 proto kernel scope link src 172.17.0.3

标签： docker networking communication

【解决方案1】：

我最终复制了，但我没有遇到您描述的行为。你身边一定发生了一些非常有趣和奇怪的事情。

# h1 上的tcpdump：

sh-4.4# tcpdump -nn -i any port 9000
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
21:07:08.516964 IP 172.17.0.2.60074 > 172.17.0.3.9000: Flags [S], seq 2385135450, win 64240, options [mss 1460,sackOK,TS val 3644697290 ecr 0,nop,wscale 7], length 0
21:07:08.517085 IP 172.17.0.3.9000 > 172.17.0.2.60074: Flags [R.], seq 0, ack 2385135451, win 0, length 0

# h2 上的tcpdump：

sh-4.4# tcpdump -nn -i any port 9000
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
21:07:08.517033 IP 172.17.0.2.60074 > 172.17.0.3.9000: Flags [S], seq 2385135450, win 64240, options [mss 1460,sackOK,TS val 3644697290 ecr 0,nop,wscale 7], length 0
21:07:08.517062 IP 172.17.0.3.9000 > 172.17.0.2.60074: Flags [R.], seq 0, ack 2385135451, win 0, length 0

我这边桥接网络的配置（我没发现有什么不同）：

docker network inspect bridge
[
    {
        "Name": "bridge",
        "Id": "71db10f2d5fe7b737b94313d6a2e49c3c6d90bbfab357f9c6f0a6ef33cd54cbe",
        "Created": "2020-08-08T04:46:05.307778298+03:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.17.0.0/16",
                    "Gateway": "172.17.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "5dc46e9d625d1a4c683fbc9f8c523eeafdd1cc7434a48cf0bebf7f4c3e091e6f": {
                "Name": "h2",
                "EndpointID": "fa4343e05938f6e9b23ed89a9c6bbdf3320665d2ec4b7a7efe5f5a3c83c9bc58",
                "MacAddress": "02:42:ac:11:00:03",
                "IPv4Address": "172.17.0.3/16",
                "IPv6Address": ""
            },
            "5f2e38bec9df6cb26018c60ea237278cdff100ecdf37ab04a1dc5a87f69b114b": {
                "Name": "h1",
                "EndpointID": "da9e723e4e1f4406828d66fde1d5958013064c93082ef9cdcb0d6374d0cadf0e",
                "MacAddress": "02:42:ac:11:00:02",
                "IPv4Address": "172.17.0.2/16",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.bridge.default_bridge": "true",
            "com.docker.network.bridge.enable_icc": "true",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
            "com.docker.network.bridge.name": "docker0",
            "com.docker.network.driver.mtu": "1500"
        },
        "Labels": {}
    }
]

您是否遇到从#h2 到#h1 的相同行为？
您能否在您的帖子中分享traceroute 172.17.0.3、ip route 的输出从# h1 执行。
此外，您的 docker 主机上的 /etc/docker/daemon.json 或您可能在该网络或 iptables 上执行的任何自定义更改...
你试过重启docker服务吗？

【讨论】：

daemon.json 为空。是的，从 h2 到 h1 的行为是相同的。我尝试重新启动 docker，重新启动主机，重新安装 docker，都导致相同的结果。我昨天发现了这个 [link] stackoverflow.com/questions/45293932/… 非常相似。而且我确实有很多 iptables 规则。但是，我不确定是哪一个原因。我会将所有规则粘贴到新帖子中。