创建新的 StreamingContext 时出现 AbstractMethodError答案

【问题标题】：AbstractMethodError upon creation of new StreamingContext创建新的 StreamingContext 时出现 AbstractMethodError
【发布时间】：2019-12-09 10:33:35
【问题描述】：

我在尝试实例化 Spark Streaming 的新 StreamingContext 时遇到问题。

我正在尝试创建一个新的 StreamingContext，但是抛出了 AbstractMethodError 错误。我一直在调试堆栈跟踪，发现当在 StreamingListenerBus 中创建第三个 Spark ListenerBus 时，应用程序停止并抛出此错误。

下面是我要执行的代码

package platform.etl

import org.apache.spark.SparkConf
import org.apache.spark.streaming.{Seconds, StreamingContext}

object ClickGeneratorStreaming {
  def main(args: Array[String]): Unit = {

    val conf = new SparkConf().setAppName("ClickGeneratorStreaming").setMaster("local[*]")
    val ssc = new StreamingContext(conf, Seconds(10)

  }
}

这是堆栈跟踪

Exception in thread "main" java.lang.AbstractMethodError
    at org.apache.spark.util.ListenerBus$class.$init$(ListenerBus.scala:35)
    at org.apache.spark.streaming.scheduler.StreamingListenerBus.<init>(StreamingListenerBus.scala:30)
    at org.apache.spark.streaming.scheduler.JobScheduler.<init>(JobScheduler.scala:56)
    at org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.scala:183)
    at org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.scala:84)
    at platform.etl.ClickGeneratorStreaming$.main(ClickGeneratorStreaming.scala:10)
    at platform.etl.ClickGeneratorStreaming.main(ClickGeneratorStreaming.scala)

我的 build.sbt

name := "spark"

version := "0.1"

scalaVersion := "2.11.0"

val sparkVersion = "2.3.0.2.6.5.0-292"
val sparkKafkaVersion = "2.3.0"
val argonautVersion = "6.2"

resolvers += "jitpack" at "https://jitpack.io"
resolvers += "horton" at "http://repo.hortonworks.com/content/repositories/releases"
resolvers += "horton2" at "http://repo.hortonworks.com/content/groups/public"


libraryDependencies += "org.apache.hadoop" % "hadoop-aws" % "2.7.3.2.6.5.0-292" excludeAll ExclusionRule(organization = "javax.servlet")
libraryDependencies += "com.amazonaws" % "aws-java-sdk" % "1.7.4"
libraryDependencies += "com.softwaremill.sttp" %% "core" % "1.2.0-RC2"
libraryDependencies += "com.softwaremill.retry" %% "retry" % "0.3.0"
libraryDependencies += "org.scalatest" %% "scalatest" % "3.0.5" % Test
libraryDependencies += "com.github.scopt" %% "scopt" % "3.7.0"
libraryDependencies += "io.argonaut" %% "argonaut" % argonautVersion
libraryDependencies += "io.argonaut" %% "argonaut-monocle" % argonautVersion
libraryDependencies += "com.github.scopt" %% "scopt" % "3.7.0"
libraryDependencies += "com.github.mrpowers" % "spark-fast-tests" % "v2.3.0_0.11.0" % "test"
libraryDependencies += "org.scalactic" %% "scalactic" % "3.0.5"
libraryDependencies += "org.scalatest" %% "scalatest" % "3.0.5" % "test"
libraryDependencies += "com.datastax.spark" %% "spark-cassandra-connector" % "2.3.0"
libraryDependencies += "org.elasticsearch" % "elasticsearch-spark-20_2.11" % "5.2.2"
libraryDependencies += "com.redislabs" % "spark-redis" % "2.3.1-M2"
libraryDependencies +=  "org.scalaj" %% "scalaj-http" % "2.4.1"
libraryDependencies += "org.apache.spark" %% "spark-core" % sparkVersion
libraryDependencies += "org.apache.spark" %% "spark-sql" % sparkVersion 
libraryDependencies += "org.apache.spark" %% "spark-hive" % sparkVersion
libraryDependencies += "org.apache.spark" %% "spark-sql-kafka-0-10" % sparkVersion
libraryDependencies += "org.apache.spark" % "spark-streaming-kafka-0-10_2.11" % sparkVersion


assemblyMergeStrategy in assembly := {
  case PathList("javax", "servlet", xs @ _*)         => MergeStrategy.first
  case PathList("META-INF", xs @ _*) => MergeStrategy.discard
  case "application.conf"            => MergeStrategy.concat
  case "reference.conf"              => MergeStrategy.concat
  case _ => MergeStrategy.first
}

我的插件.sbt

addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.3")

【问题讨论】：

能否也包含 plugins.sbt ？它只是你使用的汇编插件吗？
经典依赖地狱。你真的需要那么多依赖吗？你不能从最新的火花流开始吗？
我已经包含了我的 plugins.sbt，这是一个相当大的项目，不幸的是，其中一部分必须有实时处理管道。
在尝试查找有关问题的更多信息时，我删除了我的 ivy 缓存并重新下载了依赖项，并且出现了此消息 [warn] * org.apache.spark:spark-tags_2。 11:2.3.0.2.6.5.0-292 超过 2.1.0 [警告] +- org.apache.spark:spark-streaming_2.11:2.1.0 (取决于 2.1.0)
您的项目中有很多驱逐。尝试运行 github.com/jrudolph/sbt-dependency-graph 找出攻击性库。

标签： scala apache-spark spark-streaming spark-streaming-kafka

【解决方案1】：

发现问题。看起来我忘记在我的 build.sbt 上添加 spark-streaming 依赖项，并且由于某种原因，它找到了一种方法来使用对我的导入的依赖项，使其使用不兼容的 spark-streaming 的不同版本使用我的 spark 版本。

为了解决这个问题，我刚刚在 build.sbt 中添加了一个换行符

libraryDependencies += "org.apache.spark" %% "spark-streaming" % sparkVersion

现在可以完美运行了。

【讨论】：