找不到 SparkConf 类答案

【问题标题】：SparkConf Class not found找不到 SparkConf 类
【发布时间】：2021-06-23 00:10:04
【问题描述】：

我正在尝试使用 spark 分析 facebook 网络。当我创建 Spark Conf 的对象时，它给了我以下错误。线程“主”java.lang.NoClassDefFoundError 中的异常：org/apache/spark/SparkConf。

我正在使用 scala 2.11 版，它应该可以正常工作。在我的 intelliJ IDEA 中，还加载了 scala sdk 2.12。相同的代码是：-

import org.apache.spark.SparkConf
import org.apache.spark.graphx.GraphLoader
import org.apache.spark.sql.SparkSession

object Sample {

   def main(args: Array[String]): Unit = {
    print("Hello World")

    val conf = new SparkConf().setMaster("local[2]")
    val spark = SparkSession
      .builder
      .appName("SampleApp").config(conf)
      .getOrCreate()
    val sc = spark.sparkContext

    val graph = GraphLoader.edgeListFile(sc, "facebook_combined.txt")

    val ranks = graph.pageRank(1).vertices

    print(ranks.collect().mkString("\n"))

    spark.stop()

  }
}

我的 pom.xml 文件在之前说明：-

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
     xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>

<groupId>Sample</groupId>
<artifactId>CMPE256</artifactId>
<version>1.0-SNAPSHOT</version>



<properties>
    <spark.version>2.1.1</spark.version>
    <scala.dep.version>2.11</scala.dep.version>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>

<dependencies>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_${scala.dep.version}</artifactId>
        <version>${spark.version}</version>
        <scope>provided</scope>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_${scala.dep.version}</artifactId>
        <version>${spark.version}</version>
        <scope>provided</scope>
    </dependency>
    <dependency>
        <groupId>junit</groupId>
        <artifactId>junit</artifactId>
        <version>4.12</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-graphx_${scala.dep.version}</artifactId>
        <version>${spark.version}</version>
        <scope>provided</scope>
    </dependency>
    <dependency>
        <groupId>org.slf4j</groupId>
        <artifactId>slf4j-log4j12</artifactId>
        <version>1.6.6</version>
    </dependency>
    <dependency>
        <groupId>log4j</groupId>
        <artifactId>log4j</artifactId>
        <version>1.2.14</version>
    </dependency>
</dependencies>



<build>
    <plugins>
        <!-- mixed scala/java compile -->
        <plugin>
            <groupId>org.scala-tools</groupId>
            <artifactId>maven-scala-plugin</artifactId>
            <version>2.15.2</version>
            <executions>
                <execution>
                    <id>compile</id>
                    <goals>
                        <goal>compile</goal>
                    </goals>
                    <phase>compile</phase>
                </execution>
                <execution>
                    <id>test-compile</id>
                    <goals>
                        <goal>testCompile</goal>
                    </goals>
                    <phase>test-compile</phase>
                </execution>
                <execution>
                    <phase>process-resources</phase>
                    <goals>
                        <goal>compile</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
        <plugin>
            <artifactId>maven-compiler-plugin</artifactId>
            <version>3.6.1</version>
            <configuration>
                <source>1.8</source>
                <target>1.8</target>
            </configuration>
        </plugin>
    </plugins>
  </build>
</project>

【问题讨论】：

可能和这个有关：stackoverflow.com/questions/37849408/…
在我的 pom.xml 中注释了提供的部分后，我在线程“main”java.lang.NoSuchMethodError 中收到以下错误异常：scala.Predef$.refArrayOps([Ljava/lang/Object;) lscala/collection/mutable/ArrayOps;

标签： apache-spark

【解决方案1】：

在您的pom.xml 中注释掉<scope>provided</scope>，这应该可以解决问题。

更多关于为什么在这里：MavenDoc

【讨论】：

【解决方案2】：

你可以在 scala 中这样写。

    package Wordcount
    import org.apache.spark.streaming._
    import org.apache.spark.SparkContext
    import org.apache.spark._

    class FirstApp {
    val conf = new SparkConf().setMaster("local[2]").setAppName("wordcount");
    ...
    }

【讨论】：