kafka + 如何验证 kafka broker 领导者是否与 kafka cli 命令平衡

Question

我们有不同版本的 Kafka 集群

kafka 集群与 1.0 版融合
kafka 集群 Apache 与 2.7 版

我们想测试领导者是否在我们的主题上保持平衡 目标是在每个主题上验证领导者是否平衡到分区

这是 Kafka 代理领导者平衡时的示例

我们可以看到我们有 4 (1001 ) , 3 (1002 ) , 3 ( 1003 )

Topic:drive.car.cost.val           PartitionCount:10     ReplicationFactor:3     Configs:segment.bytes=53687091,retention.bytes=53687091
        Topic: drive.car.cost.val       Partition: 0    Leader: 1001    Replicas: 1001,1002,1003        Isr: 1001,1002,1003
        Topic: drive.car.cost.val       Partition: 1    Leader: 1002    Replicas: 1002,1003,1001        Isr: 1002,1001,1003
        Topic: drive.car.cost.val       Partition: 2    Leader: 1003    Replicas: 1003,1001,1002        Isr: 1001,1002,1003
        Topic: drive.car.cost.val       Partition: 3    Leader: 1001    Replicas: 1001,1003,1002        Isr: 1001,1002,1003
        Topic: drive.car.cost.val       Partition: 4    Leader: 1002    Replicas: 1002,1001,1003        Isr: 1002,1001,1003
        Topic: drive.car.cost.val       Partition: 5    Leader: 1003    Replicas: 1003,1002,1001        Isr: 1002,1001,1003
        Topic: drive.car.cost.val       Partition: 6    Leader: 1001    Replicas: 1001,1002,1003        Isr: 1001,1002,1003
        Topic: drive.car.cost.val       Partition: 7    Leader: 1002    Replicas: 1002,1003,1001        Isr: 1002,1001,1003
        Topic: drive.car.cost.val       Partition: 8    Leader: 1003    Replicas: 1003,1001,1002        Isr: 1001,1002,1003
        Topic: drive.car.cost.val       Partition: 9    Leader: 1001    Replicas: 1001,1003,1002        Isr: 1001,1002,1003
        
    

这里是领导者经纪人不平衡的例子

我们可以看到我们有 1 (1001 ) , 2 (1002 ) , 7 ( 1003 )

Topic:drive.car.cost.val           PartitionCount:10     ReplicationFactor:3     Configs:segment.bytes=53687091,retention.bytes=53687091
        Topic: drive.car.cost.val       Partition: 0    Leader: 1001    Replicas: 1001,1002,1003        Isr: 1001,1002,1003
        Topic: drive.car.cost.val       Partition: 1    Leader: 1002    Replicas: 1002,1003,1001        Isr: 1002,1001,1003
        Topic: drive.car.cost.val       Partition: 2    Leader: 1003    Replicas: 1003,1001,1002        Isr: 1001,1002,1003
        Topic: drive.car.cost.val       Partition: 3    Leader: 1003    Replicas: 1001,1003,1002        Isr: 1001,1002,1003
        Topic: drive.car.cost.val       Partition: 4    Leader: 1003    Replicas: 1002,1001,1003        Isr: 1002,1001,1003
        Topic: drive.car.cost.val       Partition: 5    Leader: 1003    Replicas: 1003,1002,1001        Isr: 1002,1001,1003
        Topic: drive.car.cost.val       Partition: 6    Leader: 1003    Replicas: 1001,1002,1003        Isr: 1001,1002,1003
        Topic: drive.car.cost.val       Partition: 7    Leader: 1002    Replicas: 1002,1003,1001        Isr: 1002,1001,1003
        Topic: drive.car.cost.val       Partition: 8    Leader: 1003    Replicas: 1003,1001,1002        Isr: 1001,1002,1003
        Topic: drive.car.cost.val       Partition: 9    Leader: 1003    Replicas: 1001,1003,1002        Isr: 1001,1002,1003
        
    
    

所以我们构建了计算 kafka 领导者的 python 脚本，并验证我们是否平衡

但我想知道 kafka 本身是否有一些 cli 可以告诉我们每个分区的领导者是否平衡

Answer 1

你在使用 Confluent 吗？如果是这样，他们提供了命令行工具，请注意您还需要收集指标才能使其正常工作

https://docs.confluent.io/platform/current/kafka/rebalancer/index.html

   ./bin/confluent-rebalancer execute --bootstrap-server localhost:9092 --metrics-bootstrap-server localhost:9092 --throttle 10000000 --verbose --dry-run 

输出看起来像

Computing the rebalance plan (this may take a while) ...
You are about to move 17 replica(s) for 14 partitions to 4 broker(s) with total size 827.2 MB.
The preferred leader for 14 partition(s) will be changed.
In total, the assignment for 15 partitions will be changed.
The minimum free volume space is set to 20.0%.

The following brokers will have less than 40% of free volume space during the rebalance:
      Broker     Current Size (MB)  Size During Rebalance (MB)   Free % During Rebalance      Size After Rebalance (MB)    Free % After Rebalance
      0          413.6              620.4                        30.1                         519.6                        30.5
      2          620.4              723.8                        30.1                         520.8                        30.5
      3          0                  517                          30.1                         520.8                        30.5
      1          1,034              1,034                        30.1                         519.6                        30.5

Min/max stats for brokers (before -> after):
      Type  Leader Count                 Replica Count                Size (MB)
      Min   12 (id: 3) -> 17 (id: 0)     37 (id: 3) -> 43 (id: 3)     0 (id: 3) -> 517 (id: 1)
      Max   21 (id: 0) -> 17 (id: 0)     51 (id: 1) -> 45 (id: 0)     1,034 (id: 1) -> 517 (id: 3)
No racks are defined.

Broker stats (before -> after):
  Broker     Leader Count    Replica Count   Size (MB)            Free Space (%)
      0          21 -> 17        48 -> 45        413.6 -> 517       30.5 -> 30.5
      1          20 -> 17        51 -> 44        1,034 -> 517       30.5 -> 30.5
      2          15 -> 17        40 -> 44        620.4 -> 517       30.5 -> 30.5
      3          12 -> 17        37 -> 43        0 -> 517           30.5 -> 30.5

Would you like to continue? (y/n):

Answer 2

Kafka 中没有内置这样的命令。

除非您担心特定代理的网络饱和，否则领导者信息不会跟踪“数据平衡”