如何增加主题的分区数量并更改每个分区的领导者并重新平衡它们

Question

如果我们有一个在 kafka 中的主题并且它有 5 个分区，我们可以将分区数量增加到 30 个。此外，在增加分区数量之后，我们按照代理 ID 的顺序更改每个分区的领导者并重新平衡该特定主题的集群。我们该怎么做呢？

Answer 1

我发现了它是如何工作的。

1) 先找到已有话题的信息

2) 找出集群中所有具有其 ID 的代理

3) 分区的规模化数量

4) 运行集群重新分配脚本进行重新平衡

给出实际的 zookeeper url 或我使用 localhost 的 ips。

1) 在 bin 目录下，我们有 kafka-topics.sh ./kafka-topics.sh --zookeeper localhost:2181 --topic dummytopic --describe

向您展示所有关于分区领导者和副本存在的主题

2) ./zookeeper-shell.sh localhost:2181 做 ls /brokers/ids

为您提供所有经纪人 ID 的列表

3)./kafka-topics.sh --alter --zookeeper localhost:2181 --topic dummytopic --partitions 30

增加分区数

4) 在运行此命令之前，您需要一个 json 文件，它可以让您更改特定主题的分区领导

为此，我开发了一个简单的工具，为非常大的分区数生成 json

https://github.com/chandradeepak/kafka-reassignment-gen

去构建 && topic=dummytopic num_partitions=30 brokerid_start=1022 replica_count=3 ./kafka-reassignment-gen

这将生成一个我们可以用于 expand-cluster-reassignment.json 的 json。它看起来像这样

{"version":1,"partitions":[{"topic":"dummytopic","partition":0,"replicas":[1001,1002,1003]},{"topic":"dummytopic ","partition":1,"replicas":[1002,1003,1004]},{"topic":"dummytopic","partition":2,"replicas":[1003,1004,1005]},{ "topic":"dummytopic","partition":3,"replicas":[1004,1005,1006]},{"topic":"dummytopic","partition":4,"replicas":[1005,1006 ,1007]},{"topic":"dummytopic","partition":5,"replicas":[1006,1007,1008]},{"topic":"dummytopic","partition":6,"replicas ":[1007,1008,1009]},{"topic":"dummytopic","partition":7,"replicas":[1008,1009,1010]},{"topic":"dummytopic","partition ":8,"replicas":[1009,1010,1011]},{"topic":"dummytopic","partition":9,"replicas":[1010,1011,1012]},{"topic": "dummytopic","partition":10,"replicas":[1011,1012,1013]},{"topic":"dummytopic","partition":11,"replicas":[1012,1013,1014]} ,{"topic":"dummytopic","partition":12,"replicas":[1013,1014,1015]},{"topic":"dummytopic","partition":13,"replicas":[1014 ,1015,1016]},{"topic":"dummytopic","partition":14,"replicas":[1015,10 16,1017]},{"topic":"dummytopic","partition":15,"replicas":[1016,1017,1018]},{"topic":"dummytopic","partition":16,"副本":[1017,1018,1019]},{"topic":"dummytopic","partition":17,"replicas":[1018,1019,1020]},{"topic":"dummytopic"," partition":18,"replicas":[1019,1020,1021]},{"topic":"dummytopic","partition":19,"replicas":[1020,1021,1022]},{"topic" :"dummytopic","partition":20,"replicas":[1021,1022,1023]},{"topic":"dummytopic","partition":21,"replicas":[1022,1023,1024] },{"topic":"dummytopic","partition":22,"replicas":[1023,1024,1025]},{"topic":"dummytopic","partition":23,"replicas":[ 1024,1025,1026]},{"topic":"dummytopic","partition":24,"replicas":[1025,1026,1027]},{"topic":"dummytopic","partition":25 ,"副本":[1026,1027,1028]},{"topic":"dummytopic","partition":26,"replicas":[1027,1028,1029]},{"topic":"dummytopic" ,"partition":27,"replicas":[1028,1029,1030]},{"topic":"dummytopic","partition":28,"replicas":[1029,1030,1001]},{" topic":"dummytopic","partition":29,"replicas":[1030,1001,1002]}] }

./kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file expand-cluster-reassignment.json --execute

这将执行集群重新分配并将分区领导者更改为您所期望的。