Spark Streaming 使用 Apache Spark 2.2.1-java.lang.NoClassDefFoundError: scala/xml/MetaData

Question

我在java eclipse中创建了一个maven项目，并在pom.xml文件中添加了依赖。

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>com.stream2</groupId>
  <artifactId>streamtest2</artifactId>
  <version>0.0.1-SNAPSHOT</version>
 <dependencies>

    <dependency>
        <groupId>org.scala-lang</groupId>
        <artifactId>scala-library</artifactId>
        <version>2.11.8</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.10</artifactId>
        <version>2.2.1</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-streaming_2.10</artifactId>
        <version>2.2.1</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-streaming-twitter_2.10</artifactId>
        <version>1.3.1</version>
     </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-examples_2.10</artifactId>
            <version>1.1.0</version>
        </dependency>
    </dependencies>  

    <properties>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>

    </properties>
</project>

然后我创建了一个 java 类并添加了以下代码以从我之前创建的 Twitter 应用程序中获取数据流here

当我安装 maven 然后作为 java 应用程序运行时，出现以下错误。

    Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
18/02/26 12:07:16 INFO SparkContext: Running Spark version 2.2.1
18/02/26 12:07:16 WARN NativeCodeLoader: Unable to load native-hadoop library 
for your platform... using builtin-java classes where applicable
18/02/26 12:07:16 INFO SparkContext: Submitted application: 
SparkTwitterHelloWorldExample
18/02/26 12:07:16 INFO SecurityManager: Changing view acls to: Pearlstone
18/02/26 12:07:16 INFO SecurityManager: Changing modify acls to: Pearlstone
18/02/26 12:07:16 INFO SecurityManager: Changing view acls groups to: 
18/02/26 12:07:16 INFO SecurityManager: Changing modify acls groups to: 
18/02/26 12:07:16 INFO SecurityManager: SecurityManager: authentication 
disabled; ui acls disabled; users  with view permissions: Set(Pearlstone); 
groups with view permissions: Set(); users  with modify permissions: 
Set(Pearlstone); groups with modify permissions: Set()
18/02/26 12:07:16 INFO Utils: Successfully started service 'sparkDriver' on port 
65136.
18/02/26 12:07:16 INFO SparkEnv: Registering MapOutputTracker
18/02/26 12:07:16 INFO SparkEnv: Registering BlockManagerMaster
18/02/26 12:07:16 INFO BlockManagerMasterEndpoint: Using 
org.apache.spark.storage.DefaultTopologyMapper for getting topology information
18/02/26 12:07:16 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
18/02/26 12:07:16 INFO DiskBlockManager: Created local directory at 
C:\Users\Pearlstone\AppData\Local\Temp\blockmgr-83f0290f-edf1-4656-b50a-
128eac23e165
18/02/26 12:07:16 INFO MemoryStore: MemoryStore started with capacity 898.5 MB
18/02/26 12:07:16 INFO SparkEnv: Registering OutputCommitCoordinator
Exception in thread "main" java.lang.NoClassDefFoundError: scala/xml/MetaData
    at org.apache.spark.ui.jobs.JobsTab.<init>(JobsTab.scala:38)
    at org.apache.spark.ui.SparkUI.initialize(SparkUI.scala:67)
    at org.apache.spark.ui.SparkUI.<init>(SparkUI.scala:84)
    at org.apache.spark.ui.SparkUI$.create(SparkUI.scala:221)
    at org.apache.spark.ui.SparkUI$.createLiveUI(SparkUI.scala:163)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:452)
    at org.apache.spark.streaming.StreamingContext$.createNewSparkContext(StreamingContext.scala:839)
    at org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.scala:85)
    at org.apache.spark.streaming.api.java.JavaStreamingContext.<init>
(JavaStreamingContext.scala:138)
    at streamtest2.stream2.main(stream2.java:20)
Caused by: java.lang.ClassNotFoundException: scala.xml.MetaData
    at java.net.URLClassLoader.findClass(Unknown Source)
    at java.lang.ClassLoader.loadClass(Unknown Source)
    at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
    at java.lang.ClassLoader.loadClass(Unknown Source)
    ... 10 more
18/02/26 12:07:16 INFO DiskBlockManager: Shutdown hook called
18/02/26 12:07:16 INFO ShutdownHookManager: Shutdown hook called
18/02/26 12:07:16 INFO ShutdownHookManager: Deleting directory 
C:\Users\Pearlstone\AppData\Local\Temp\spark-f566f6fe-b4a1-45cf-add2-
12b993e3f607
18/02/26 12:07:16 INFO ShutdownHookManager: Deleting directory 
C:\Users\Pearlstone\AppData\Local\Temp\spark-f566f6fe-b4a1-45cf-add2-
12b993e3f607\userFiles-20dc3f27-6857-4ec4-adb6-1231b8616959

Answer 1

您必须使用与以下相同版本的 scala 编译依赖项

<dependencies>
    <dependency>
        <groupId>org.scala-lang</groupId>
        <artifactId>scala-library</artifactId>
        <version>2.11.8</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.11</artifactId>
        <version>2.2.1</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-streaming_2.11</artifactId>
        <version>2.2.1</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-streaming-twitter_2.11</artifactId>
        <version>1.3.1</version>
    </dependency>
</dependencies> 

希望这会有所帮助！