Cassandra Spark Connector 版本与 spark 2.2 冲突答案

【问题标题】：Cassandra Spark Connector version conflict with spark 2.2Cassandra Spark Connector 版本与 spark 2.2 冲突
【发布时间】：2019-06-21 07:10:57
【问题描述】：

运行 spark 作业时出现以下错误。请为 spark 和 Cassandra 连接器推荐正确的版本。

下面是我的 build.sbt

scalaVersion := "2.11.8"

 

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-streaming" % "2.2.0-cdh6.0.1" % "provided",
  "org.apache.spark" %% "spark-core" % "2.2.0-cdh6.0.1" % "provided", // excludeAll ExclusionRule(organization = "javax.servlet"),
  "org.apache.spark" %% "spark-sql" % "2.2.0-cdh6.0.1" % "provided",
  "org.apache.spark" %% "`enter code here`spark-streaming-kafka-0-10" % "2.2.0-cdh6.0.1",
  "org.apache.hbase" % "hbase-client" % "2.0.0-cdh6.0.1",
  "org.apache.hbase" % "hbase-common" % "2.0.0-cdh6.0.1",
  "com.datastax.spark" %% "spark-cassandra-connector" % "2.0.10",
  "net.liftweb" %% "lift-json" % "3.3.0",
  "com.typesafe" % "config" % "1.2.1"
)

在 spark 上提交作业后，我收到以下错误

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/configuration/ConfigurationException
    at com.datastax.spark.connector.streaming.DStreamFunctions.saveToCassandra$default$4(DStreamFunctions.scala:47)
    at com.StreamingPrerequisiteLoad$.main(StreamingPrerequisiteLoad.scala:72)
    at com.StreamingPrerequisiteLoad.main(StreamingPrerequisiteLoad.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.commons.configuration.ConfigurationException
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

【问题讨论】：

标签： apache-spark cassandra cassandra-3.0 spark-cassandra-connector

【解决方案1】：

我在 spark-cassandra-connector 遇到了类似的问题，所以看看它是如何工作的，对于 spark 2.2 版和 scala 11.8.0，spark-cassandra-connector 2.3.0 也可以使用。添加 commons-configuration 1.9 版本 jar ，因为它会抛出异常 NoClassDefFound:/org/apache/commons/configuration/ConfigurationException 。尝试以下依赖项：

    version := "0.1"

scalaVersion := "2.11.8"

libraryDependencies += "org.apache.spark" %% "spark-core" % "2.2.0" % "provided"
libraryDependencies += "org.apache.spark" %% "spark-streaming" % "2.2.0" % "provided"
libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.2.0" % "provided"
libraryDependencies += "net.liftweb" %% "lift-json" % "3.0.2"
libraryDependencies += "org.apache.spark" %% "spark-streaming-kafka-0-10" % "2.0.0" //% "provided"
libraryDependencies += "com.datastax.spark" %% "spark-cassandra-connector" % "2.3.0" //% "provided"
libraryDependencies += "commons-configuration" % "commons-configuration" % "1.9" //% "provided"

assemblyMergeStrategy in assembly := {
  case PathList("org", "apache", "spark", "unused", "UnusedStubClass.class") => MergeStrategy.first
  case x => (mergeStrategy in assembly).value(x)
}

【讨论】：