Kafka Zookeeper 连接不断下降答案

【问题标题】：Kafka Zookeeper Connection drop continuouslyKafka Zookeeper 连接不断下降
【发布时间】：2018-07-30 19:51:40
【问题描述】：

我已经在不同的节点上设置了 Kafka 3 节点集群和 Zookeeper 3 节点集群。使用 Kafka，我可以成功地生成和使用消息，并运行 kafka-topic.sh 之类的命令从 Zookeeper 获取主题列表及其信息，但是 Kafka server.log 文件存在一些错误。持续出现以下警告：

[2018-02-18 21:50:01,241] WARN Client session timed out, have not heard from server in 320190154ms for sessionid 0x161a94b101f0001 (org.apache.zookeeper.ClientCnxn)
[2018-02-18 21:50:01,242] INFO Client session timed out, have not heard from server in 320190154ms for sessionid 0x161a94b101f0001, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
[2018-02-18 21:50:01,343] INFO zookeeper state changed (Disconnected) (org.I0Itec.zkclient.ZkClient)
[2018-02-18 21:50:01,989] INFO Opening socket connection to server zookeeper3/192.168.1.206:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2018-02-18 21:50:02,008] INFO Socket connection established to zookeeper3/192.168.1.206:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2018-02-18 21:50:02,042] INFO Session establishment complete on server zookeeper3/192.168.1.206:2181, sessionid = 0x161a94b101f0001, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
[2018-02-18 21:50:02,042] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient)
[2018-02-18 21:59:31,570] INFO [Group Metadata Manager on Broker 102]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)

zookeeper 中的 Kafka 会话似乎会定期过期！

Zookeeper 日志中也包含以下警告：

2018-02-18 18:20:06,149 [myid:1] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@368] - caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x161a94b101f0001, likely client has closed socket
    at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
    at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
    at java.lang.Thread.run(Thread.java:748)
2018-02-18 18:20:06,151 [myid:1] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1044] - Closed socket connection for client /192.168.1.203:43162 which had sessionid 0x161a94b101f0001
2018-02-18 18:20:06,781 [myid:1] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@368] - caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x161a94b101f0002, likely client has closed socket
    at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
    at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
    at java.lang.Thread.run(Thread.java:748)
2018-02-18 18:20:06,782 [myid:1] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1044] - Closed socket connection for client /192.168.1.201:45330 which had sessionid 0x161a94b101f0002
2018-02-18 18:37:29,127 [myid:1] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted socket connection from /192.168.1.202:52480
2018-02-18 18:37:29,139 [myid:1] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@942] - Client attempting to establish new session at /192.168.1.202:52480
2018-02-18 18:37:29,143 [myid:1] - INFO  [CommitProcessor:1:ZooKeeperServer@687] - Established session 0x161a94b101f0003 with negotiated timeout 30000 for client /192.168.1.202:52480
2018-02-18 18:37:29,432 [myid:1] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1044] - Closed socket connection for client /192.168.1.202:52480 which had sessionid 0x161a94b101f0003

我认为这是因为 zookeeper 无法从 Kafka 节点获取心跳。以下为Zookeeperzoo.cfg：

tickTime=2000
dataDir=/var/zookeeper/
clientPort=2181
initLimit=5
syncLimit=2
server.1=zookeeper1:2888:3888
server.2=zookeeper2:2888:3888
server.3=zookeeper3:2888:3888

和Kafkaserver.properties自定义设置：

broker.id=1
listeners = PLAINTEXT://kafka1:9092
num.partitions=24
delete.topic.enable=true
default.replication.factor=2
log.dirs=/data/kafka/data
zookeeper.connect=zookeeper1:2181,zookeeper2:2181,zookeeper3:2181
log.retention.hours=168

我为 Hadoop HA 使用相同的 zookeeper 集群没有任何问题。我认为 Kafka 属性 listeners 和 advertised.listeners 有问题。我阅读了 Kafka 文档，但无法理解它们的含义。

在所有操作系统的主机文件中，主机名如zookeeper1 至zookeeper3 和kafka1 至kafka3 均已定义并可通过ping 命令访问。我从主机中删除了以下行：

127.0.0.1       localhost
127.0.1.1       hostname

我认为这不会导致问题。

Kafka 版本：0.11
动物园管理员版本：3.4.10

谁能帮忙？

【问题讨论】：

您能找出问题所在吗？我遇到了类似的问题。重新启动 kafka 代理可以工作，但这只是一个临时解决方案。
我猜是因为 Java GC，在 GC 期间，broker 无法心跳 Zookeeper。
你找到解决方案了吗，我也面临同样的问题，有什么可以帮忙的吗@SoheilPourbafrani
你找到解决方案了吗，我也面临同样的问题，有什么可以帮忙的吗@BandiKishore
@andani 是的，正如 Soheil 指出的那样，当我验证 GC 日志时，主要 GC 持续运行。这是因为 kafka w.r.t 中的内存泄漏错误。指标。 issues.apache.org/jira/browse/KAFKA-6307

标签： apache-kafka apache-zookeeper

【解决方案1】：

我们在使用 Kafka 时遇到了类似的问题。正如@Soheil 指出的那样，这是由于运行了 Major GC。

当一个 Major GC 运行时，Kafka 有时会无法向 zookeeper 发送心跳。对我们来说，Major GC 几乎每 15 秒运行一次。在进行堆转储时，我们意识到这是由于Kafka 中的度量内存泄漏。

【讨论】：