卡夫卡失败并挤压整个盒子的磁盘空间

Question

我正在使用 5 节点 kafka 和 5 节点 zookeeper 集群。动物园管理员工作正常。卡夫卡一直在工作，直到我在卡夫卡中创建了一个主题。所以，步骤是：

1. Setup Zookeeper cluster and start all zookeeper nodes
2. Setup Kafka cluster and start all kafka nodes
3. Everything is fine upto here.
4. Create a topic.
5. Following error starts to pop up continuously and squeezes all the disk space.

[2015-11-11 10:00:01,289] INFO Reconnect due to socket error: java.nio.channels.ClosedChannelException (kafka.consumer.SimpleConsumer)
[2015-11-11 10:00:01,289] WARN [ReplicaFetcherThread-1-1], Error in fetch Name: FetchRequest; Version: 0; CorrelationId: 287237122; ClientId: ReplicaFetcherThread-1-1; ReplicaId: 4; MaxWait: 500 ms; MinBytes: 1 bytes; RequestInfo: [DataPLFLogger,3] -> PartitionFetchInfo(0,1048576). Possible cause: java.nio.channels.ClosedChannelException (kafka.server.ReplicaFetcherThread)
[2015-11-11 10:00:01,413] INFO Reconnect due to socket error: java.nio.channels.ClosedChannelException (kafka.consumer.SimpleConsumer)
[2015-11-11 10:00:01,413] WARN [ReplicaFetcherThread-1-1], Error in fetch Name: FetchRequest; Version: 0; CorrelationId: 287237123; ClientId: ReplicaFetcherThread-1-1; ReplicaId: 4; MaxWait: 500 ms; MinBytes: 1 bytes; RequestInfo: [DataPLFLogger,3] -> PartitionFetchInfo(0,1048576). Possible cause: java.nio.channels.ClosedChannelException (kafka.server.ReplicaFetcherThread)
[2015-11-11 10:00:01,413] INFO Reconnect due to socket error: java.nio.channels.ClosedChannelException (kafka.consumer.SimpleConsumer)
[2015-11-11 10:00:01,413] WARN [ReplicaFetcherThread-1-1], Error in fetch Name: FetchRequest; Version: 0; CorrelationId: 287237124; ClientId: ReplicaFetcherThread-1-1; ReplicaId: 4; MaxWait: 500 ms; MinBytes: 1 bytes; RequestInfo: [DataPLFLogger,3] -> PartitionFetchInfo(0,1048576). Possible cause: java.nio.channels.ClosedChannelException (kafka.server.ReplicaFetcherThread)
[2015-11-11 10:00:01,413] INFO Reconnect due to socket error: java.nio.channels.ClosedChannelException (kafka.consumer.SimpleConsumer)
[2015-11-11 10:00:01,413] WARN [ReplicaFetcherThread-1-1], Error in fetch Name: FetchRequest; Version: 0; CorrelationId: 287237125; ClientId: ReplicaFetcherThread-1-1; ReplicaId: 4; MaxWait: 500 ms; MinBytes: 1 bytes; RequestInfo: [DataPLFLogger,3] -> PartitionFetchInfo(0,1048576). Possible cause: java.nio.channels.ClosedChannelException (kafka.server.ReplicaFetcherThread)
[2015-11-11 10:00:01,413] INFO Reconnect due to socket error: java.nio.channels.ClosedChannelException (kafka.consumer.SimpleConsumer)
[2015-11-11 10:00:01,414] WARN [ReplicaFetcherThread-1-1], Error in fetch Name: FetchRequest; Version: 0; CorrelationId: 287237126; ClientId: ReplicaFetcherThread-1-1; ReplicaId: 4; MaxWait: 500 ms; MinBytes: 1 bytes; RequestInfo: [DataPLFLogger,3] -> PartitionFetchInfo(0,1048576). Possible cause: java.nio.channels.ClosedChannelException (kafka.server.ReplicaFetcherThread)

我尝试同时使用 kafka-0.8.1.1 和 kafka-0.8.2.2 但我得到了同样的错误。 我正在使用以下配置文件：

broker.id=0
log.dirs=/data/kafka/node1/kafka-logs
host.name=xx-xxxxxxxxxxxxx-kafka1.xxx.com
port=10256
delete.topic.enable=true

# ZK configuration
zookeeper.connect=10.1.21.164:10240,10.1.26.104:10240,10.1.18.204:10240,10.1.22.65:10240,10.1.29.87:10240
zookeeper.connection.timeout.ms=2000
zookeeper.sync.time.ms=2000

# Replication configurations
num.replica.fetchers=4
default.replication.factor=3

# Log configuration
num.partitions=8
log.segment.bytes=67108864
auto.create.topics.enable=false
log.retention.hours=168
log.flush.interval.ms=500
log.flush.interval.messages=20000
log.flush.scheduler.interval.ms=2000
log.roll.hours=1

# Socket server configuration
num.io.threads=4
num.network.threads=8
queued.max.requests=40
fetch.purgatory.purge.interval.requests=1000
producer.purgatory.purge.interval.requests=1000

我还尝试从一个代理远程登录到其他 kafka 代理，并尝试远程登录 zookeeper 节点。每次连接都成功。

请帮助我如何解决这个问题。

Answer 1

创建 Kafka 主题

语法：

kafka-topics.sh --zookeeper <$ZOOKEEPER_HOST:2181> --create --topic <topic-name> --partitions <no of partitions> --replication-factor <no of replicas>

示例：

kafka-topics.sh --zookeeper node1:2181 --create --topic squid
--partitions 1 --replication-factor 1