spark-submit 在线程“main”java.lang.IllegalStateException 中抛出异常：找不到任何构建目录答案

【问题标题】：spark-submit throws Exception in thread "main" java.lang.IllegalStateException: Cannot find any build directoriesspark-submit 在线程“main”java.lang.IllegalStateException 中抛出异常：找不到任何构建目录
【发布时间】：2021-12-25 22:15:01
【问题描述】：

Spark 提交命令：

[root@d03db3cedc5a opt]# bash -x $SPARK_HOME/bin/spark-submit --master spark://analytics-seed:7077 --py-files $SPARK_HOME/hello_world.py
+ '[' -z /opt/spark ']'
+ export PYTHONHASHSEED=0
+ PYTHONHASHSEED=0
+ exec /opt/spark/bin/spark-class org.apache.spark.deploy.SparkSubmit --master spark://analytics-seed:7077 --py-files /opt/spark/hello_world.py  
Exception in thread "main" java.lang.IllegalArgumentException: Missing application resource.
        at org.apache.spark.launcher.CommandBuilderUtils.checkArgument(CommandBuilderUtils.java:241)
        at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildSparkSubmitArgs(SparkSubmitCommandBuilder.java:160)
        at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildSparkSubmitCommand(SparkSubmitCommandBuilder.java:276)
        at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildCommand(SparkSubmitCommandBuilder.java:151)
        at org.apache.spark.launcher.Main.main(Main.java:86)

当我尝试直接运行 java 命令时，它会抛出以下异常：

[root@d03db3cedc5a opt]# /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el8_4.x86_64/jre/bin/java -Xmx128m -cp '/opt/spark/jars/*' org.apache.spark.launcher.Main org.apache.spark.deploy.SparkSubmit --master spark://analytics-seed:7077 --py-files /opt/spark/hello_world.py
Exception in thread "main" java.lang.IllegalStateException: Cannot find any build directories.
        at org.apache.spark.launcher.CommandBuilderUtils.checkState(CommandBuilderUtils.java:248)
        at org.apache.spark.launcher.AbstractCommandBuilder.getScalaVersion(AbstractCommandBuilder.java:241)
        at org.apache.spark.launcher.AbstractCommandBuilder.buildClassPath(AbstractCommandBuilder.java:195)
        at org.apache.spark.launcher.AbstractCommandBuilder.buildJavaCommand(AbstractCommandBuilder.java:118)
        at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildSparkSubmitCommand(SparkSubmitCommandBuilder.java:237)
        at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildCommand(SparkSubmitCommandBuilder.java:151)
        at org.apache.spark.launcher.Main.main(Main.java:86)

我已导出以下 ENV：

export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el8_4.x86_64/jre
export SPARK_HOME=/opt/spark
export PATH=$PATH:$SPARK_HOME/bin
export PYTHONPATH=$SPARK_HOME/python/:$PYTHONPATH
export PYTHONPATH=$SPARK_HOME/python/lib/py4j-0.10.3-src.zip:$PYTHONPATH

【问题讨论】：

标签： java apache-spark pyspark spark-submit

【解决方案1】：

由于 --py-files 参数，命令失败。

$SPARK_HOME/bin/spark-submit \
--master spark://analytics-seed:7077 \
--py-files $SPARK_HOME/hello_world.py

删除参数后，命令应该按预期工作：

$SPARK_HOME/bin/spark-submit \
--master spark://analytics-seed:7077 \
hello_world.py

【讨论】：