Kafka 代理意外关闭

Question

在 Windows 子系统中的 kafka/zookeeper v2.4.1（二进制 kafka_2.13-2.4.1.tgz）的单个集群/实例安装中Linux (WSL) 与 Ubuntu 18.04，在清理日志文件期间，kafka 代理意外关闭，并显示以下错误消息：

ERROR Failed to clean up log for __consumer_offsets-11 in dir /tmp/kafka-logs due to IOException (kafka.server.LogDirFailureChannel)
java.io.IOException: Invalid argument
    at java.io.RandomAccessFile.setLength(Native Method)
    at kafka.log.AbstractIndex.$anonfun$resize$1(AbstractIndex.scala:188)
    at scala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.scala:17)
    at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:253)
    at kafka.log.AbstractIndex.resize(AbstractIndex.scala:174)
    at kafka.log.AbstractIndex.$anonfun$trimToValidSize$1(AbstractIndex.scala:240)
    at scala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.scala:17)
    at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:253)
    at kafka.log.AbstractIndex.trimToValidSize(AbstractIndex.scala:240)
    at kafka.log.LogSegment.onBecomeInactiveSegment(LogSegment.scala:508)
    at kafka.log.Cleaner.cleanSegments(LogCleaner.scala:595)
    at kafka.log.Cleaner.$anonfun$doClean$6(LogCleaner.scala:530)
    at kafka.log.Cleaner.$anonfun$doClean$6$adapted(LogCleaner.scala:529)
    at scala.collection.immutable.List.foreach(List.scala:305)
    at kafka.log.Cleaner.doClean(LogCleaner.scala:529)
    at kafka.log.Cleaner.clean(LogCleaner.scala:503)
    at kafka.log.LogCleaner$CleanerThread.cleanLog(LogCleaner.scala:372)
    at kafka.log.LogCleaner$CleanerThread.cleanFilthiestLog(LogCleaner.scala:345)
    at kafka.log.LogCleaner$CleanerThread.tryCleanFilthiestLog(LogCleaner.scala:325)
    at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:314)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)

存在清理失败的目录__consumer_offsets-11：

Kafka logs dir contents

我尝试了以下方法：

• 手动删除logs目录并重新运行kafka/zookeeper，但稍后仍会出现同样的错误。
• 将日志目录从服务器配置更改为嵌套在其他位置，而不是在 /tmp 中；没有解决问题。

此错误每天都会发生很多次，与日志保留配置无关。服务器配置属性 (server.properties) 是默认属性：

broker.id=0
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/tmp/kafka-logs
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=localhost:2181
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=0

Answer 1

问题似乎出在您的log.dirs 目前设置为tmp/kafka-logs。当您的机器关闭时，这可能会导致一些麻烦，因为tmp/ 的内容将被清除。

---

尝试将路径更改为永久位置而不是tmp/