【发布时间】:2020-08-04 06:17:08
【问题描述】:
首先,我使用 How to build jars from IntelliJ properly? 创建了一个 jar 文件。
我的 Jar 文件路径是
out/artifacts/sparkProgram_jar/sparkProgram.jar
一般来说,我的 spark 程序从 MongoDB 中读取一个表,使用 spark 的 mllib 对其进行转换,然后将其写入 MySQL。 这是我的 build.sbt 文件。
name := "sparkProgram"
version := "0.1"
scalaVersion := "2.12.4"
val sparkVersion = "3.0.0"
val postgresVersion = "42.2.2"
resolvers ++= Seq(
"bintray-spark-packages" at "https://dl.bintray.com/spark-packages/maven",
"Typesafe Simple Repository" at "https://repo.typesafe.com/typesafe/simple/maven-releases",
"MavenRepository" at "https://mvnrepository.com"
)
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % sparkVersion,
"org.apache.spark" %% "spark-sql" % sparkVersion,
"org.apache.spark" %% "spark-mllib" % sparkVersion,
// logging
"org.apache.logging.log4j" % "log4j-api" % "2.4.1",
"org.apache.logging.log4j" % "log4j-core" % "2.4.1",
"org.mongodb.spark" %% "mongo-spark-connector" % "2.4.1",
//"mysql" % "mysql-connector-java" % "5.1.12",
"mysql" % "mysql-connector-java" % "8.0.18"
).
我的主类位于名为 scala 对象的 com.testing 包中
mainObject
当我运行以下 spark-submit 命令时
spark-submit --master local --class com.testing.mainObject
--packages mysql:mysql-connector-java:8.0.18,org.mongodb.spark:mongo-spark-connector_2.12:2.4.1 out/artifacts/sparkProgram_jar/sparkProgram.jar
我收到此错误
Error: Missing application resource.
Usage: spark-submit [options] <app jar | python file | R file> [app arguments]
Usage: spark-submit --kill [submission ID] --master [spark://...]
Usage: spark-submit --status [submission ID] --master [spark://...]
Usage: spark-submit run-example [options] example-class [example args]
Options:
... zsh: command not found: --packages
然后当我尝试在没有 --packages 的情况下运行我的 spark-submit(只是为了检查会发生什么)时,我收到了这个错误。
命令:
spark-submit --master local --class com.testing.mainObject out/artifacts/sparkProgram_jar/sparkProgram.jar
错误: 错误:无法加载类 com.testing.mainObject
我以前使用过 spark-submit 并且它有效(几个月前)。我不确定为什么这仍然给我一个错误。我的 MANIFEST.MF 如下
Manifest-Version: 1.0
Main-Class: com.testing.mainObject
【问题讨论】:
标签: scala apache-spark spark-submit