【问题标题】:What is Difference between broker-list and bootstrap servers?代理列表和引导服务器有什么区别?
【发布时间】:2018-10-23 01:36:01
【问题描述】:

卡夫卡有什么区别

broker-listbootstrap servers

【问题讨论】:

  • 它们是别名(至少从 librdkafka 的角度来看)。见:github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md
  • 使用 kafka 2.0.1。为什么控制台消费者使用--bootstrap-server 而控制台生产者使用--broker-list?如果是同样的东西,那就是一个可怕的迷失方向错误!

标签: apache-kafka


【解决方案1】:

我也讨厌阅读“像墙一样的文字”卡夫卡文档:P
据我了解:

  • 经纪人名单

    • 完整的服务器列表,如果缺少任何生产者可能无法工作
    • 与生产者命令相关
  • 引导服务器

    • 一个足以发现所有其他人
    • 与消费者命令相关
    • 涉及动物园管理员

很抱歉这么...简短。下次我会更专注于细节以更清楚。 为了解释我的观点,我将使用 Kafka 1.0.1 控制台脚本。

kafka-console-consumer.sh

The console consumer is a tool that reads data from Kafka and outputs it to standard output.
Option                                   Description
------                                   -----------
--blacklist <String: blacklist>          Blacklist of topics to exclude from
                                           consumption.
--bootstrap-server <String: server to    REQUIRED (unless old consumer is
  connect to>                              used): The server to connect to.
--consumer-property <String:             A mechanism to pass user-defined
  consumer_prop>                           properties in the form key=value to
                                           the consumer.
--consumer.config <String: config file>  Consumer config properties file. Note
                                           that [consumer-property] takes
                                           precedence over this config.
--csv-reporter-enabled                   If set, the CSV metrics reporter will
                                           be enabled
--delete-consumer-offsets                If specified, the consumer path in
                                           zookeeper is deleted when starting up
--enable-systest-events                  Log lifecycle events of the consumer
                                           in addition to logging consumed
                                           messages. (This is specific for
                                           system tests.)
--formatter <String: class>              The name of a class to use for
                                           formatting kafka messages for
                                           display. (default: kafka.tools.
                                           DefaultMessageFormatter)
--from-beginning                         If the consumer does not already have
                                           an established offset to consume
                                           from, start with the earliest
                                           message present in the log rather
                                           than the latest message.
--group <String: consumer group id>      The consumer group id of the consumer.
--isolation-level <String>               Set to read_committed in order to
                                           filter out transactional messages
                                           which are not committed. Set to
                                           read_uncommittedto read all
                                           messages. (default: read_uncommitted)
--key-deserializer <String:
  deserializer for key>
--max-messages <Integer: num_messages>   The maximum number of messages to
                                           consume before exiting. If not set,
                                           consumption is continual.
--metrics-dir <String: metrics           If csv-reporter-enable is set, and
  directory>                               this parameter isset, the csv
                                           metrics will be output here
--new-consumer                           Use the new consumer implementation.
                                           This is the default, so this option
                                           is deprecated and will be removed in
                                           a future release.
--offset <String: consume offset>        The offset id to consume from (a non-
                                           negative number), or 'earliest'
                                           which means from beginning, or
                                           'latest' which means from end
                                           (default: latest)
--partition <Integer: partition>         The partition to consume from.
                                           Consumption starts from the end of
                                           the partition unless '--offset' is
                                           specified.
--property <String: prop>                The properties to initialize the
                                           message formatter.
--skip-message-on-error                  If there is an error when processing a
                                           message, skip it instead of halt.
--timeout-ms <Integer: timeout_ms>       If specified, exit if no message is
                                           available for consumption for the
                                           specified interval.
--topic <String: topic>                  The topic id to consume on.
--value-deserializer <String:
  deserializer for values>
--whitelist <String: whitelist>          Whitelist of topics to include for
                                           consumption.
--zookeeper <String: urls>               REQUIRED (only when using old
                                           consumer): The connection string for
                                           the zookeeper connection in the form
                                           host:port. Multiple URLS can be
                                           given to allow fail-over.

kafka-console-producer.sh
Read data from standard input and publish it to Kafka.
Option                                   Description
------                                   -----------
--batch-size <Integer: size>             Number of messages to send in a single
                                           batch if they are not being sent
                                           synchronously. (default: 200)
--broker-list <String: broker-list>      REQUIRED: The broker list string in
                                           the form HOST1:PORT1,HOST2:PORT2.
--compression-codec [String:             The compression codec: either 'none',
  compression-codec]                       'gzip', 'snappy', or 'lz4'.If
                                           specified without value, then it
                                           defaults to 'gzip'
--key-serializer <String:                The class name of the message encoder
  encoder_class>                           implementation to use for
                                           serializing keys. (default: kafka.
                                           serializer.DefaultEncoder)
--line-reader <String: reader_class>     The class name of the class to use for
                                           reading lines from standard in. By
                                           default each line is read as a
                                           separate message. (default: kafka.
                                           tools.
                                           ConsoleProducer$LineMessageReader)
--max-block-ms <Long: max block on       The max time that the producer will
  send>                                    block for during a send request
                                           (default: 600

如您所见,bootstrap-server 参数仅适用于消费者。另一方面 - broker-list 仅在生产者的参数列表中。

此外:

kafka-console-consumer.sh --zookeeper localost:2181 --topic bets
Using the ConsoleConsumer with old consumer is deprecated and will be removed in a future major release. Consider using the new consumer by passing [bootstrap-server] instead of [zookeeper].

所以 cricket-007 注意到 bootstrap-server 和 zookeeper 看起来有相似的目的。不同的是 --zookeeper 应该指向另一边的 Zookeeper 节点 --bootstrap-server 指向 Kafka 节点和端口。

Reasuming,bootstrap-server 被用作消费者参数,broker-list 被用作生产者参数。

【讨论】:

  • 你能解释一下“涉及动物园管理员”吗?鉴于新的消费者 API 可以使用 --zookeeper--bootstrap-server
  • 我认为动物园管理员参与了两者。对我来说,真正的答案是简单地 --brokers-list 已被弃用并保留以保持兼容性。对于最新版本,它不存在。我刚刚检查过。
【解决方案2】:

这个答案仅供参考,因为我没有使用 --broker-list 所以我很困惑然后我意识到它已被弃用。

目前我使用的是 Kafka 版本 2.6.0。

现在对于生产者和消费者,我们必须使用 --bootstrap-server 而不是 --broker-list,因为它现在已被弃用。

您可以在 Kafka 控制台脚本中检查这一点。

bin/kafka-console-producer.sh

如您所见, -- broker-list 已被 Kafka-console-producer.sh 弃用

bin/kafka-console-consumer.sh

【讨论】:

    【解决方案3】:

    其他人已经很好回答了,我只是想在这里分享一些额外的信息。

    bin 目录下的那些命令行工具没有记录详细用法。

    当然,您可以调用--help 来打印给定命令支持的语法和选项的描述。

    例如:bin/kafka-console-producer.sh --help

    --bootstrap-server <String: server to    REQUIRED unless --broker-list
      connect to>                              (deprecated) is specified. The server
                                               (s) to connect to. The broker list
                                               string in the form HOST1:PORT1,HOST2:
                                               PORT2.
    --broker-list <String: broker-list>      DEPRECATED, use --bootstrap-server
                                               instead; ignored if --bootstrap-
                                               server is specified.  The broker
                                               list string in the form HOST1:PORT1,
                                               HOST2:PORT2.
    

    但您可以直接从源代码中找到最新信息,而不是运行命令,就在core/src/main/scala/kafka目录中,相应的scala类可以在tools下目录,或admin 目录。

    例如:kafka-console-producer.sh 脚本实际上调用了来自 ConsoleProducer.scala 类的函数。在那里你可以很容易地发现 broker-list 已弃用

    享受阅读源代码的乐趣:)

    【讨论】:

      猜你喜欢
      • 1970-01-01
      • 1970-01-01
      • 2019-05-16
      • 2018-11-14
      • 2017-01-09
      • 1970-01-01
      • 2019-02-26
      • 1970-01-01
      • 1970-01-01
      相关资源
      最近更新 更多