卡夫卡一遍又一遍地重播消息 - 心跳会话过期 - 标记协调器死亡答案

【问题标题】：Kafka replays messages over and over - Heartbeat session expired - marking coordinator dead卡夫卡一遍又一遍地重播消息 - 心跳会话过期 - 标记协调器死亡
【发布时间】：2018-03-11 20:41:43
【问题描述】：

使用 python kafka api 从一个只有少量消息的主题中读取消息。 Kafka 不断地重复播放队列中的消息。

它从我的主题接收一条消息（返回每个消息内容），然后抛出 ERROR - Heartbeat session expired - marking coordinator dead 并继续循环其余消息并继续重播它们。更多日志：

kafka.coordinator - ERROR - Heartbeat session expired - marking coordinator dead
kafka.coordinator - WARNING - Marking the coordinator dead (node 1) for group GROUPID1: Heartbeat session expired.
kafka.coordinator.consumer - WARNING - Auto offset commit failed for group GROUPID1: CommitFailedError: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured session.timeout.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing the session timeout or by reducing the maximum size of batches returned in poll() with max.poll.records.
kafka.cluster - INFO - Group coordinator for GROUPID1 is BrokerMetadata(nodeId=1, host='HOST', port=PORT, rack=None)
kafka.coordinator - INFO - Discovered coordinator 1 for group GROUPID1
kafka.coordinator - INFO - Skipping heartbeat: no auto-assignment or waiting on rebalance
kafka.coordinator.consumer - ERROR - Offset commit failed: This is likely to cause duplicate message delivery
Traceback (most recent call last):
  File "/path/python3.5/site-packages/kafka/coordinator/consumer.py", line 407, in _maybe_auto_commit_offsets_sync
    self.commit_offsets_sync(self._subscription.all_consumed_offsets())
  File "/path/python3.5/site-packages/kafka/coordinator/consumer.py", line 398, in commit_offsets_sync
    raise future.exception # pylint: disable-msg=raising-bad-type
kafka.errors.CommitFailedError: CommitFailedError: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured session.timeout.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing the session timeout or by reducing the maximum size of batches returned in poll() with max.poll.records.
kafka.coordinator.consumer - INFO - Revoking previously assigned partitions {TopicPartition(topic='TOPIC1', partition=0)} for group GROUPID1

【问题讨论】：

标签： python apache-kafka kafka-consumer-api

【解决方案1】：

似乎您需要调整您的消费者配置，最有可能查看日志似乎消费者的心跳会话即将到期，并且由于会话过期而无法提交最后轮询的记录，并且它正在触发重新平衡，因此它将从最后一个未提交的记录再次轮询

要检查的配置 -

heartbeat.interval.ms
session.timeout.ms
max.poll.interval.ms

【讨论】：