【问题标题】:kafka connect - Restating the worker causing rebalance issuekafka connect - 重新启动导致重新平衡问题的工作人员
【发布时间】:2021-10-19 02:52:32
【问题描述】:

我在分布式模式下使用 2 节点 Kafka Connect。它们运行良好,但是当我重新启动 Worker 服务时,在该节点上运行的连接器转到 UNASSIGNED,然后恰好在 5 分钟后它更改为 ASSIGNED。我不知道为什么会发生这种情况,因为通常它必须将该连接器的任务移动到另一个正在运行的节点,对吗?

这里是日志:(工人重启 5 分钟后)

Rebalance started [org.apache.kafka.connect.runtime.distributed.WorkerCoordinator:221]
[2021-08-17 07:23:46,120] [INFO] [Worker clientId=connect-1, groupId=debezium-cluster1] (Re-)joining group [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:538]
[2021-08-17 07:23:46,124] [INFO] [Worker clientId=connect-1, groupId=debezium-cluster1] Successfully joined group with generation Generation{generationId=27, memberId='connect-1-56d39766-4974-4203-945b-6eee4fe811e7', protocol='sessioned'} [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:594]
[2021-08-17 07:23:46,128] [INFO] [Worker clientId=connect-1, groupId=debezium-cluster1] Successfully synced group in generation Generation{generationId=27, memberId='connect-1-56d39766-4974-4203-945b-6eee4fe811e7', protocol='sessioned'} [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:758]
[2021-08-17 07:23:46,129] [INFO] [Worker clientId=connect-1, groupId=debezium-cluster1] Joined group at generation 27 with protocol version 2 and got assignment: Assignment{error=0, leader='connect-1-ccdf6d6a-eeab-423c-9611-56795d0deca9', leaderUrl='http://172.30.32.13:8083/', offset=20, connectorIds=[mysql-connector-01], taskIds=[mysql-connector-01-0], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 [org.apache.kafka.connect.runtime.distributed.DistributedHerder:1694]
[2021-08-17 07:23:46,129] [INFO] [Worker clientId=connect-1, groupId=debezium-cluster1] Starting connectors and tasks using config offset 20 [org.apache.kafka.connect.runtime.distributed.DistributedHerder:1244]
[2021-08-17 07:23:46,130] [INFO] [Worker clientId=connect-1, groupId=debezium-cluster1] Starting task mysql-connector-01-0 [org.apache.kafka.connect.runtime.distributed.DistributedHerder:1286]
[2021-08-17 07:23:46,131] [INFO] [Worker clientId=connect-1, groupId=debezium-cluster1] Starting connector mysql-connector-01 [org.apache.kafka.connect.runtime.distributed.DistributedHerder:1321]

我尝试重新启动连接器,但它不起作用。

curl -X POST 172.30.34.99:8083/connectors/mysql-connector-01/restart

{"error_code":409,"message":"Cannot complete request momentarily due to no known leader URL, likely because a rebalance was underway."}

【问题讨论】:

    标签: apache-kafka apache-kafka-connect


    【解决方案1】:

    我找到了原因,这是由于 Kafka 计划的重新平衡延迟。一个很棒的博客来了解更多 - https://www.confluent.io/blog/incremental-cooperative-rebalancing-in-kafka/

    【讨论】:

      猜你喜欢
      • 1970-01-01
      • 2018-01-20
      • 2021-09-08
      • 2020-03-23
      • 2013-04-17
      • 1970-01-01
      • 2018-11-12
      • 1970-01-01
      • 1970-01-01
      相关资源
      最近更新 更多