Spark 1.4 缺少 Kafka 库答案

【问题标题】：Spark 1.4 missing Kafka librariesSpark 1.4 缺少 Kafka 库
【发布时间】：2015-09-25 21:09:38
【问题描述】：

我正在尝试运行在 spark 1.3.1 中完美运行的 Python spark 脚本。我已经下载了 spark 1.4 并尝试运行脚本，但它一直在说

在类路径中找不到 Spark Streaming 的 Kafka 库。试一试以下。
在 spark-submit 命令中包含 Kafka 库及其依赖项
$ bin/spark-submit --packages org.apache.spark:spark-streaming-kafka:1.4.0 ...
从 Maven Central http://search.maven.org/ 下载工件的 JAR，组 ID = org.apache.spark，工件 ID = spark-streaming-kafka-assembly，版本 = 1.4.0。然后，将 jar 包含在 spark-submit 命令中
$ bin/spark-submit --jars <spark-streaming-kafka-assembly.jar> ...

我在提交命令中明确引用了 jars 并将 jars 添加为

/opt/spark/spark-1.4.0-bin-hadoop2.6/bin/spark-submit --jars spark-streaming_2.10-1.4.0.jar,spark-core_2.10-1.4.0.jar,spark-streaming-kafka-assembly_2.10-1.4.0.jar,kafka_2.10-0.8.2.1.jar,kafka-clients-0.8.2.1.jar,spark-streaming-kafka-assembly_2.10-1.4.0.jar /root/SparkPySQLNew.py

它还说它在应用程序启动时添加了它们，为什么它没有找到它们？

15/07/08 05:44:37 INFO spark.SparkContext: Added JAR file:/root/spark-streaming_2.10-1.4.0.jar at http://192.168.134.138:49637/jars/spark-streaming_2.10-1.4.0.jar with timestamp 1436334277792
15/07/08 05:44:37 INFO spark.SparkContext: Added JAR file:/root/spark-core_2.10-1.4.0.jar at http://192.168.134.138:49637/jars/spark-core_2.10-1.4.0.jar with timestamp 1436334277919
15/07/08 05:44:38 INFO spark.SparkContext: Added JAR file:/root/spark-streaming-kafka-assembly_2.10-1.4.0.jar at http://192.168.134.138:49637/jars/spark-streaming-kafka-assembly_2.10-1.4.0.jar with timestamp 1436334278295
15/07/08 05:44:38 INFO spark.SparkContext: Added JAR file:/root/kafka_2.10-0.8.2.1.jar at http://192.168.134.138:49637/jars/kafka_2.10-0.8.2.1.jar with timestamp 1436334278353
15/07/08 05:44:38 INFO spark.SparkContext: Added JAR file:/root/kafka-clients-0.8.2.1.jar at http://192.168.134.138:49637/jars/kafka-clients-0.8.2.1.jar with timestamp 1436334278357
15/07/08 05:44:38 INFO spark.SparkContext: Added JAR file:/root/spark-streaming-kafka-assembly_2.10-1.4.0.jar at http://192.168.134.138:49637/jars/spark-streaming-kafka-assembly_2.10-1.4.0.jar with timestamp 1436334278665
15/07/08 05:44:38 INFO spark.SparkContext: Added JAR file:/root/spark-streaming-kafka-assembly_2.10-1.4.0-sources.jar at http://192.168.134.138:49637/jars/spark-streaming-kafka-assembly_2.10-1.4.0-sources.jar with timestamp 1436334278666

而且我知道我已经添加了很多，我从一个开始，然后到最后才全部添加。

【问题讨论】：

标签： hadoop apache-spark apache-kafka spark-streaming hortonworks-data-platform

【解决方案1】：

我怀疑每个版本的 spark 的确切答案各不相同，但基于 this HCC thread，以下内容似乎对其他人有用：

spark-submit --jars spark-assembly-1.5.2.2.3.4.7-4-hadoop2.7.1.2.3.4.7-4.jar,spark-streaming-kafka-assembly_2.10-1.6.1.jar

乍一看，不同之处在于它有 1 个 spark-streaming-kafka-assembly jar，而您提交的是两个。

【讨论】：