【发布时间】：2022-08-23 00:34:02
【问题描述】：
import org.apache.spark.sql.SparkSession
object RDDBroadcast extends App {
val spark = SparkSession.builder()
.appName(\"SparkByExamples.com\")
.master(\"local\")
.getOrCreate()
val states = Map((\"NY\",\"New York\"),(\"CA\",\"California\"),(\"FL\",\"Florida\"))
val countries = Map((\"USA\",\"United States of America\"),(\"IN\",\"India\"))
val broadcastStates = spark.sparkContext.broadcast(states)
val broadcastCountries = spark.sparkContext.broadcast(countries)
val data = Seq((\"James\",\"Smith\",\"USA\",\"CA\"),
(\"Michael\",\"Rose\",\"USA\",\"NY\"),
(\"Robert\",\"Williams\",\"USA\",\"CA\"),
(\"Maria\",\"Jones\",\"USA\",\"FL\")
)
val rdd = spark.sparkContext.parallelize(data)
val rdd2 = rdd.map(f=>{
val country = f._3
val state = f._4
val fullCountry = broadcastCountries.value(country)
val fullState = broadcastStates.value(state)
(f._1,f._2,fullCountry,fullState)
})
println(rdd2.collect().mkString(\"\\n\"))
}
以上是获取国家和州名的 spark-scala 代码。在 InteliJ IDEA 中编译代码时，出现如下错误：
*Error: A JNI error has occurred, please check your installation and try again
Exception in thread \"main\" java.lang.NoClassDefFoundError: org/apache/spark/sql/SparkSession
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
at java.lang.Class.getMethod0(Class.java:3018)
at java.lang.Class.getMethod(Class.java:1784)
at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:650)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:632)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.SparkSession
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 7 more*
我尝试了多种解决方案，例如检查 java 和 javac 版本，但版本是同步的。 Click here to view the image.
我还检查了项目 File->project Structure...-> Module 中的 java 版本，并将其与 Run->Edit Configurations 进行了比较，这也是匹配的。
Build.sbt 代码：Click here to view the sbt code
我正在使用在 Linux 操作系统中安装 inteliJ 的 VMware 工作站 16 播放器。 java版本是1.8.0_301
标签： java scala apache-spark hadoop