SparkWordCount 问题 - java.lang.ClassNotFoundException

Question

这不是一个重复的问题，我已经尝试了很多方法来完成这项工作，但没有奏效

我正在尝试编写一个字数统计应用程序，以便我可以通过 spark-submit 运行

我正在使用 IntelliJ IDEA、spark - 2.1.1 和 scala - 2.11.8

我的字数统计代码如下所示::

package com.netflix.utilities

import org.apache.spark.SparkContext
import org.apache.spark.SparkConf

class SparkWordCount {
  def main(args: Array[String]) {
   // create Spark context with Spark configuration
   println("starting")
   val sc = new SparkContext(new SparkConf().setAppName("Spark Count"))

   // get threshold
   val threshold = args(1).toInt

   // read in text file and split each document into words
   val tokenized = sc.textFile(args(0)).flatMap(_.split(" "))

   // count the occurrence of each word
   val wordCounts = tokenized.map((_, 1)).reduceByKey(_ + _)

   // filter out words with fewer than threshold occurrences
   val filtered = wordCounts.filter(_._2 >= threshold)

   // count characters
   val charCounts = filtered.flatMap(_._1.toCharArray).map((_, 1)).reduceByKey(_ + _)

   System.out.println(charCounts.collect().mkString(", "))
  }
}

我的 pom.xml 看起来像这样 ::

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
     xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>

<groupId>com.netflix.utilities</groupId>
<artifactId>SparkWordCount</artifactId>
<version>1.0-SNAPSHOT</version>
<dependencies>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.11</artifactId>
        <version>2.1.1</version>
    </dependency>
    <dependency>
        <groupId>org.scala-lang</groupId>
        <artifactId>scala-library</artifactId>
        <version>2.11.8</version>
    </dependency>

</dependencies>

<distributionManagement>
    <repository>
        <id>internal.repo</id>
        <name>Internal repo</name>
        <url>file:///Users/sankar.biswas/Noah/</url>
    </repository>
</distributionManagement>


<build>
    <pluginManagement>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.7.0</version>
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                </configuration>
            </plugin>
        </plugins>
    </pluginManagement>
</build>

现在我已将我的 jar 放在与 spark-submit 相同的文件夹中

这是我正在运行的 spark-submit::

./spark-submit --class com.netflix.utilities.SparkWordCount --master local --deploy-mode client SparkWordCount-1.0-SNAPSHOT.jar "file:////Users/sankar.biswas/Desktop/你好.txt”

但我不断收到此错误 ::

java.lang.ClassNotFoundException: SparkWordCount

我没有得到我所缺少的东西。任何建议将不胜感激！

Answer 1

使用对象 SparkWordCount 代替类 SparkWordCount 并将 --class "SparkWordCount" 更改为 --class com.netflix.utilities.SparkWordCount 并运行它