无法在 CentOS 7 上启动 kafka 代理答案

【问题标题】：unable to bring up kafka broker on CentOS 7无法在 CentOS 7 上启动 kafka 代理
【发布时间】：2017-04-28 10:06:41
【问题描述】：

Kafka 新手，我尝试在 Centos 服务器上安装 kafka，但出现以下错误，这是全新安装，我没有配置任何东西来将数据推送到 kafka 生产者

[2016-12-12 18:20:18,531] INFO zookeeper state changed (SyncConnected)     (org.I0Itec.zkclient.ZkClient)
[2016-12-12 18:20:18,601] INFO Loading logs. (kafka.log.LogManager)
[2016-12-12 18:20:18,609] INFO Logs loading complete. (kafka.log.LogManager)
[2016-12-12 18:20:18,676] INFO Starting log cleanup with a period of 300000 ms. (kafka.log.LogManager)
[2016-12-12 18:20:18,678] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager)
[2016-12-12 18:20:18,734] INFO Awaiting socket connections on 172.16.23.13:9092. (kafka.network.Acceptor)
[2016-12-12 18:20:18,770] INFO [Socket Server on Broker 0], Started 1 acceptor threads (kafka.network.SocketServer)
[2016-12-12 18:20:18,796] INFO [ExpirationReaper-0], Starting  (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2016-12-12 18:20:18,798] INFO [ExpirationReaper-0], Starting  (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2016-12-12 18:20:18,876] INFO Creating /controller (is it secure? false) (kafka.utils.ZKCheckedEphemeral)
[2016-12-12 18:20:18,887] INFO Result of znode creation is: OK (kafka.utils.ZKCheckedEphemeral)
[2016-12-12 18:20:18,888] INFO 0 successfully elected as leader (kafka.server.ZookeeperLeaderElector)
[2016-12-12 18:20:18,980] INFO [ExpirationReaper-0], Starting  (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2016-12-12 18:20:18,981] INFO [ExpirationReaper-0], Starting  (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2016-12-12 18:20:18,995] INFO [GroupCoordinator 0]: Starting up. (kafka.coordinator.GroupCoordinator)
[2016-12-12 18:20:18,996] INFO [GroupCoordinator 0]: Startup complete. (kafka.coordinator.GroupCoordinator)
[2016-12-12 18:20:19,001] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 10 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-12-12 18:20:19,016] INFO [ThrottledRequestReaper-Produce], Starting  (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2016-12-12 18:20:19,017] INFO [ThrottledRequestReaper-Fetch], Starting  (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2016-12-12 18:20:19,024] INFO Will not load MX4J, mx4j-tools.jar is not in the classpath (kafka.utils.Mx4jLoader$)
[2016-12-12 18:20:19,053] INFO Creating /brokers/ids/0 (is it secure? false) (kafka.utils.ZKCheckedEphemeral)
[2016-12-12 18:20:19,056] INFO Result of znode creation is: OK (kafka.utils.ZKCheckedEphemeral)
[2016-12-12 18:20:19,058] INFO Registered broker 0 at path /brokers/ids/0 with addresses: PLAINTEXT -> EndPoint(172.16.23.13,9092,PLAINTEXT) (kafka.utils.ZkUtils)
[2016-12-12 18:20:19,079] INFO Kafka version : 0.10.0.0 (org.apache.kafka.common.utils.AppInfoParser)
[2016-12-12 18:20:19,079] INFO Kafka commitId : b8642491e78c5a13 (org.apache.kafka.common.utils.AppInfoParser)
[2016-12-12 18:20:19,080] INFO [Kafka Server 0], started (kafka.server.KafkaServer)
[2016-12-12 18:20:19,082] INFO New leader is 0 (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
[2016-12-12 18:20:20,170] ERROR Processor got uncaught exception. (kafka.network.Processor)
java.lang.OutOfMemoryError: Java heap space
    at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
    at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
    at    org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:93)
    at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:71)
    at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:154)
    at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:135)
    at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:323)
    at org.apache.kafka.common.network.Selector.poll(Selector.java:283)
    at kafka.network.Processor.poll(SocketServer.scala:472)
    at kafka.network.Processor.run(SocketServer.scala:412)
    at java.lang.Thread.run(Thread.java:745)
[2016-12-12 18:20:20,180] WARN Unexpected error from /172.16.23.13; closing connection (org.apache.kafka.common.network.Selector)
java.lang.NullPointerException
    at org.apache.kafka.common.network.NetworkReceive.complete(NetworkReceive.java:67)
    at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:136)
    at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:323)
    at org.apache.kafka.common.network.Selector.poll(Selector.java:283)
    at kafka.network.Processor.poll(SocketServer.scala:472)
    at kafka.network.Processor.run(SocketServer.scala:412)
    at java.lang.Thread.run(Thread.java:745)

如果我增加内存，则会出现以下错误。

[2016-12-12 18:18:20,789] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager)
[2016-12-12 18:18:20,844] INFO Awaiting socket connections on 172.16.23.13:9092. (kafka.network.Acceptor)
[2016-12-12 18:18:20,848] INFO [Socket Server on Broker 0], Started 1 acceptor threads (kafka.network.SocketServer)
[2016-12-12 18:18:20,905] INFO [ExpirationReaper-0], Starting  (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2016-12-12 18:18:20,908] INFO [ExpirationReaper-0], Starting  (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2016-12-12 18:18:20,988] INFO Creating /controller (is it secure? false) (kafka.utils.ZKCheckedEphemeral)
[2016-12-12 18:18:20,999] INFO Result of znode creation is: OK (kafka.utils.ZKCheckedEphemeral)
[2016-12-12 18:18:21,000] INFO 0 successfully elected as leader (kafka.server.ZookeeperLeaderElector)
[2016-12-12 18:18:21,097] INFO [ExpirationReaper-0], Starting  (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2016-12-12 18:18:21,099] INFO [ExpirationReaper-0], Starting  (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2016-12-12 18:18:21,112] INFO [GroupCoordinator 0]: Starting up. (kafka.coordinator.GroupCoordinator)
[2016-12-12 18:18:21,113] INFO [GroupCoordinator 0]: Startup complete. (kafka.coordinator.GroupCoordinator)
[2016-12-12 18:18:21,117] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 10 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-12-12 18:18:21,133] INFO [ThrottledRequestReaper-Produce], Starting  (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2016-12-12 18:18:21,134] INFO [ThrottledRequestReaper-Fetch], Starting  (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2016-12-12 18:18:21,141] INFO Will not load MX4J, mx4j-tools.jar is not in the classpath (kafka.utils.Mx4jLoader$)
[2016-12-12 18:18:21,168] INFO Creating /brokers/ids/0 (is it secure? false) (kafka.utils.ZKCheckedEphemeral)
[2016-12-12 18:18:21,172] INFO Result of znode creation is: OK (kafka.utils.ZKCheckedEphemeral)
[2016-12-12 18:18:21,174] INFO Registered broker 0 at path /brokers/ids/0 with addresses: PLAINTEXT -> EndPoint(172.16.23.13,9092,PLAINTEXT)  (kafka.utils.ZkUtils)
[2016-12-12 18:18:21,194] INFO Kafka version : 0.10.0.0 (org.apache.kafka.common.utils.AppInfoParser)
[2016-12-12 18:18:21,194] INFO Kafka commitId : b8642491e78c5a13 (org.apache.kafka.common.utils.AppInfoParser)
[2016-12-12 18:18:21,195] INFO [Kafka Server 0], started (kafka.server.KafkaServer)
[2016-12-12 18:18:21,200] INFO New leader is 0 (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
[2016-12-12 18:18:30,013] WARN Unexpected error from /172.16.23.13; closing connection (org.apache.kafka.common.network.Selector)
org.apache.kafka.common.network.InvalidReceiveException: Invalid receive (size = 1347375956 larger than 104857600)
    at  org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:91)
    at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:71)
    at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:154)
    at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:135)
    at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:323)
    at org.apache.kafka.common.network.Selector.poll(Selector.java:283)
    at kafka.network.Processor.poll(SocketServer.scala:472)
    at kafka.network.Processor.run(SocketServer.scala:412)
    at java.lang.Thread.run(Thread.java:745)

【问题讨论】：

您是否为代理启用了安全性？
不，我没有明确启用任何安全功能

标签： java apache-kafka centos7

【解决方案1】：

其他一些进程正在向 kafka 发送数据。在我的情况下，influxdb 正在向 kafka 端口发送数据，因为 kapacitor 也侦听 9092 端口。所以influxdb正在向kafka发送数据。一旦我停止了 influxdb，就没有更多的例外。

【讨论】：