为什么不能在 Kafka 中正确使用消费者？

Question

有3个kafka服务器

在所有 3 台服务器上设置 /etc/hosts。

192.168.0.1  kafka1
192.168.0.2  kafka2
192.168.0.3  kafka3

通过配置在所有这些设备上安装了 zookeeper 和 kafka

/usr/local/kafka_2.12-2.6.0/config/server.properties

kafka1

#
broker.id=1
listeners=PLAINTEXT://kafka1:9092
advertised.listeners=PLAINTEXT://kafka1:9092

kafka2

#
broker.id=2
listeners=PLAINTEXT://kafka2:9092
advertised.listeners=PLAINTEXT://kafka2:9092

kafka3

#
broker.id=3
listeners=PLAINTEXT://kafka3:9092
advertised.listeners=PLAINTEXT://kafka3:9092

启动zookeeper和kafka后，新建一个topic

[kafka@kafka1 ~]$ bin/kafka-topics.sh --create --zookeeper kafka1:2181,kafka2:2181,kafka3:2181 --replication-factor 1 --partitions 6 --topic topic1 --config cleanup.policy=delete --config delete.retention.ms=60000

检查所有三个节点上的集群状态

kafka1

[kafka@kafka1 ~]$ bin/kafka-topics.sh --describe --zookeeper kafka1:2181 --topic topic1
Topic: topic1   PartitionCount: 6   ReplicationFactor: 1    Configs: cleanup.policy=delete,delete.retention.ms=60000
    Topic: topic1   Partition: 0    Leader: 2   Replicas: 2 Isr: 2
    Topic: topic1   Partition: 1    Leader: 3   Replicas: 3 Isr: 3
    Topic: topic1   Partition: 2    Leader: 1   Replicas: 1 Isr: 1
    Topic: topic1   Partition: 3    Leader: 2   Replicas: 2 Isr: 2
    Topic: topic1   Partition: 4    Leader: 3   Replicas: 3 Isr: 3
    Topic: topic1   Partition: 5    Leader: 1   Replicas: 1 Isr: 1

kafka2

[kafka@kafka2 ~]$ bin/kafka-topics.sh --describe --zookeeper kafka1:2181 --topic topic1
Topic: topic1   PartitionCount: 6   ReplicationFactor: 1    Configs: cleanup.policy=delete,delete.retention.ms=60000
    Topic: topic1   Partition: 0    Leader: 2   Replicas: 2 Isr: 2
    Topic: topic1   Partition: 1    Leader: 3   Replicas: 3 Isr: 3
    Topic: topic1   Partition: 2    Leader: 1   Replicas: 1 Isr: 1
    Topic: topic1   Partition: 3    Leader: 2   Replicas: 2 Isr: 2
    Topic: topic1   Partition: 4    Leader: 3   Replicas: 3 Isr: 3
    Topic: topic1   Partition: 5    Leader: 1   Replicas: 1 Isr: 1

kafka3

[kafka@kafka3 ~]$ bin/kafka-topics.sh --describe --zookeeper kafka1:2181 --topic topic1
Topic: topic1   PartitionCount: 6   ReplicationFactor: 1    Configs: cleanup.policy=delete,delete.retention.ms=60000
    Topic: topic1   Partition: 0    Leader: 2   Replicas: 2 Isr: 2
    Topic: topic1   Partition: 1    Leader: 3   Replicas: 3 Isr: 3
    Topic: topic1   Partition: 2    Leader: 1   Replicas: 1 Isr: 1
    Topic: topic1   Partition: 3    Leader: 2   Replicas: 2 Isr: 2
    Topic: topic1   Partition: 4    Leader: 3   Replicas: 3 Isr: 3
    Topic: topic1   Partition: 5    Leader: 1   Replicas: 1 Isr: 1

但是在测试集群时，在 kafka1 上创建了一个生产者

[kafka@kafka1 ~]$ bin/kafka-console-producer.sh --broker-list kafka1:9092 --topic topic1
>

在任何其他节点（包括）当前 kafka1 节点上，运行消费者

bin/kafka-console-consumer.sh --bootstrap-server kafka1:9092 --topic topic1

遇到同样的错误：

# kafka1
[2020-08-21 02:09:57,299] WARN [Consumer clientId=consumer-console-consumer-39789-1, groupId=console-consumer-39789] Connection to node 2147483645 (kafka2/192.168.0.2:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
# kafka2
[2020-08-21 03:05:00,573] WARN [Consumer clientId=consumer-console-consumer-71891-1, groupId=console-consumer-71891] Connection to node -1 (kafka1/192.168.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
# kafka3
[2020-08-21 03:05:14,331] WARN [Consumer clientId=consumer-console-consumer-55574-1, groupId=console-consumer-55574] Connection to node -1 (kafka1/192.168.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

在/usr/local/kafka_2.12-2.6.0/config/server.properties，我也尝试设置为

listeners=PLAINTEXT://localhost:9092
advertised.listeners=PLAINTEXT://localhost:9092

在所有 3 个节点上。但同样的问题。

Answer 1

“侦听器”应代表您的生产者和消费者可以访问的节点 FQDN 和端口。 当生产者/消费者连接到 Kafka 时，就像在您的帖子中一样：

bin/kafka-console-consumer.sh --bootstrap-server kafka1:9092 --topic topic1

它打开到 kafka1:9092 的连接，然后从 Kafka 获取包含所有代理、它们的 FQDN、PORT 和每个代理作为领导者的主题分区的集群地图。

您设法创建了使用 zookeeper 完成的主题并且有效。

根据您的 WARN 日志，“broker1,2,3”解析为 192.168.0.1/2/3，所以 IP 解析没问题。

尝试检查端口 9092 上的网络连接： 从 broker1 运行：“telnet broker2 9092”

或运行生产者命令访问远程代理，即：

[kafka@**kafka1** ~]$ bin/kafka-console-producer.sh --broker-list **kafka2**:9092 --topic topic1

这将告诉您是否能够在端口 9092 上从 broker1 连接到 broker2。