Spark Streaming Kafka 集成直接方法 EOFException答案

【问题标题】：Spark Streaming Kafka Integration direct Approach EOFExceptionSpark Streaming Kafka 集成直接方法 EOFException
【发布时间】：2018-12-01 16:59:08
【问题描述】：

当我运行 spark 流示例 org.apache.spark.examples.streaming.JavaDirectKafkaWordCount 时，我发现了一个 EOFException 关注，我该如何解决它

Exception in thread "main" org.apache.spark.SparkException: java.io.EOFException: Received -1 when reading from channel, socket has likely been closed.
java.io.EOFException: Received -1 when reading from channel, socket has likely been closed.
java.io.EOFException: Received -1 when reading from channel, socket has likely been closed.
    at org.apache.spark.streaming.kafka.KafkaUtils$$anonfun$createDirectStream$2.apply(KafkaUtils.scala:413)
    at org.apache.spark.streaming.kafka.KafkaUtils$$anonfun$createDirectStream$2.apply(KafkaUtils.scala:413)
    at scala.util.Either.fold(Either.scala:97)
    at org.apache.spark.streaming.kafka.KafkaUtils$.createDirectStream(KafkaUtils.scala:412)
    at org.apache.spark.streaming.kafka.KafkaUtils$.createDirectStream(KafkaUtils.scala:528)
    at org.apache.spark.streaming.kafka.KafkaUtils.createDirectStream(KafkaUtils.scala)

【问题讨论】：

标签： apache-spark apache-kafka streaming eofexception spark-streaming-kafka

【解决方案1】：

直接流使用底层的 Kafka 消费者，因此需要提供代理列表。很可能您没有，因此您可能希望以bhost1:9092,bhost2:9092,...,bhostN:9092 的形式设置metadata.broker.list 属性。

另见Kafka SimpleConsumer cannot connect to zookeeper : Received -1 when reading from channel。

【讨论】：