控制台生产者错误连接到节点 kafka1:9092 (org.apache.kafka.clients.NetworkClient) java.net.UnknownHostException

Question

我正在通过 docker-compose 成功启动一个带有 kafka 的容器。成功我的意思是，我可以登录这样的 kafka 容器并直接从那里生成和使用消息。我还可以看到另一个容器（filebeat），这取决于成功连接的此类 kafka 容器。

docker-compose.yml

version: '3.2'
services:

  zoo1:
    image: elevy/zookeeper:latest
    environment:
      MYID: 1
      SERVERS: zoo1
    ports:
      - "2181:2181"

  kafka1:
    image: wurstmeister/kafka
    command: [start-kafka.sh]
    depends_on:
      - zoo1
    links:
      - zoo1
    ports:
      - "9092:9092"
    environment:
      KAFKA_LISTENERS: PLAINTEXT://:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka1:9092
      KAFKA_BROKER_ID: 1
      KAFKA_ADVERTISED_PORT: 9092
      KAFKA_LOG_RETENTION_HOURS: "168"
      KAFKA_LOG_RETENTION_BYTES: "100000000"
      KAFKA_ZOOKEEPER_CONNECT:  zoo1:2181
      KAFKA_CREATE_TOPICS: "log:1:1"
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'true'

  filebeat:
    image: docker.elastic.co/beats/filebeat:7.5.2
    command: filebeat -e -strict.perms=false
    volumes:
      - "//c/Users/Cast/megalog-try-1/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro"
      - "//c/Users/Cast/megalog-try-1/sample-logs:/sample-logs"
    links:
      - kafka1
    depends_on:
      - kafka1

尽管如此，当我尝试从本地 kafka 向此容器 kafka 生成一条非常简单的消息时，连接被拒绝。

我发现有人有同样的例外但情况有点相似。我试过his solution 没有成功。基本上它指向“将adverted.listeners 调整为外部IP”。我在我的 docker-compose 中做到了，但没有任何改变。即使这行得通，我想这也不是最终的解决方案，因为对我来说，因为这家伙在 KAFKA_ADVERTISED_LISTENERS 中硬编码他的计算机 IP 地址

所以我的问题是如何从在我的 Windows 10 中运行的本地 kafka 连接到 kafka 容器以发送消息？

这是我迄今为止尝试过的所有内容：

1 - 尝试使用本地 ip

ipconfig
...
Endereço IPv4. . . . . . . .  . . . . . . . : 192.168.129.97

kafka-console-producer.bat --broker-list 192.168.129.97:9092 --topic log

C:\tools\kafka\bin\windows>kafka-console-producer.bat --broker-list 192.168.129.97:9092 --topic log
>[2020-03-02 16:54:44,155] WARN [Producer clientId=console-producer] Connection to node -1 (/192.168.129.97:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2020-03-02 16:54:46,212] WARN [Producer clientId=console-producer] Connection to node -1 (/192.168.129.97:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

2 - 尝试使用 docker 机器 ip

C:\tools\kafka\bin\windows>docker-machine ip
192.168.99.100

kafka-console-producer.bat --broker-list 192.168.99.100:9092 --topic log


[2020-03-02 17:00:22,500] WARN [Producer clientId=console-producer] Error connecting to node kafka1:9092 (id: 1 rack: null) (org.apache.kafka.clients.NetworkClient)
java.net.UnknownHostException: kafka1
        at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:797)
        at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1505)
        at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1364)
        at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1298)
        at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:104)
        at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:403)
        at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:363)
        at org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:151)
        at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:955)
        at org.apache.kafka.clients.NetworkClient.access$600(NetworkClient.java:69)
        at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1126)
        at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1017)
        at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:538)
        at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:335)
        at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:244)
        at java.base/java.lang.Thread.run(Thread.java:834)

3 - 尝试使用容器 ip

docker ps
...
3eb114e24b53        wurstmeister/kafka                                    "start-kafka.sh"      0.0.0.0:9092->9092/tcp                                     megalog-try-1_kafka1_1

docker inspect 3eb114e24b53
...
"IPAddress": "172.18.0.4",

C:\tools\kafka\bin\windows>kafka-console-producer.bat --broker-list 172.18.0.4:9092 --topic log
>testing
[2020-03-02 16:57:54,471] WARN [Producer clientId=console-producer] Connection to node -1 (/172.18.0.4:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
org.apache.kafka.common.KafkaException: Producer closed while send in progress
        at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:888)

***根据 CRU 的建议添加

C:\tools\kafka\bin\windows>docker exec -it megalog-try-1_zoo1_1 bash
bash-4.3# ls
bin            entrypoint.sh  home           lib64          mnt            root           sbin           sys            usr            zookeeper
dev            etc            lib            media          proc           run            srv            tmp            var
bash-4.3# cd zookeeper/
bash-4.3# ls
bin                   conf                  data                  lib                   log                   wal                   zookeeper-3.4.10.jar
bash-4.3# cd conf
bash-4.3# ls
log4j.properties  zoo.cfg
bash-4.3# cat zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/zookeeper/data
dataLogDir=/zookeeper/wal
#snapCount=100000
autopurge.purgeInterval=1
clientPort=2181
quorumListenOnAllIPs=true
server.1=zoo1:2888:3888bash-4.3#

来自kafka容器

C:\tools\kafka\bin\windows>docker exec -it megalog-try-1_kafka1_1 bash
bash-4.4# pwd
/
bash-4.4# ls
bin    dev    etc    home   kafka  lib    lib64  media  mnt    opt    proc   root   run    sbin   srv    sys    tmp    usr    var
bash-4.4# cd kafka
bash-4.4# ls
kafka-logs-0f385d1d435e
bash-4.4# cd kafka-logs-0f385d1d435e/
bash-4.4# ls
app_logs-0                        log-0                             meta.properties                   replication-offset-checkpoint
cleaner-offset-checkpoint         log-start-offset-checkpoint       recovery-point-offset-checkpoint  request_logs-0
bash-4.4# cat meta.properties
#
#Mon Mar 02 21:09:33 GMT 2020
cluster.id=-qitBVmjSUGo7Zd7P5cetw
version=0
broker.id=1
bash-4.4#

***已添加

在偶然发现this article 之后，我可以编写一个 docker-compose 来运行我的本地 SpringBoot 可以发送消息的 kafka 容器。尽管如此，我必须在我的 docker-compose 中手动对 docker-machine ip 进行硬编码。好吧，我不认为这是唯一的方法，否则每次我启动 Docker ToolBox 时，我都必须运行 docker-machine ip 并替换 IP 地址。我想知道是否有更友好的解决方案。

192.168.99.100 是我的 docker-machine ip。

version: "3.2"
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
  kafka:
    image: wurstmeister/kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENERS:
        "INTERNAL://kafka:9090,\
         EXTERNAL://:9092"
      KAFKA_ADVERTISED_LISTENERS:
        "INTERNAL://kafka:9090,\
         EXTERNAL://192.168.99.100:9092"
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP:
        "INTERNAL:PLAINTEXT,\
         EXTERNAL:PLAINTEXT"
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL

根据 Cricket_007 的建议，我仔细阅读了another article，但它让我得出结论，我必须使用 DNS 解析，但据我所知，我无法在我的 Windows 机器中设置专有的 dns 解析。

Answer 1

1. KAFKA_ADVERTISED_PORT 已弃用。只需使用广告的听众。

2. 您公布的侦听器是 Docker 服务名称，而不是在同一 Docker 网络上的容器外运行的任何代码的可解析地址。

3. 您需要添加KAFKA_LISTENER_SECURITY_PROTOCOL_MAP

4. 只需使用现有的functional Docker Compose

---

顺便说一句，links 也是一个已弃用的 Compose 选项