Spark-Submit 命令需要一段时间才能运行答案

【问题标题】：Spark-Submit command takes a while to runSpark-Submit 命令需要一段时间才能运行
【发布时间】：2022-06-22 05:18:50
【问题描述】：

我们在运行 IBM AIX（版本 2）操作系统的服务器集群上安装了 Apache Hadoop 和 Spark。

Hadoop 版本 - hadoop-3.2.1 Spark 版本 - spark-3.0.1

我们正在通过运行 $SPARK_HOME/bin 文件夹下的 spark-submit --version 命令来测试 Spark 的整体安装。此命令间歇性地运行。当您第一次运行它时，该命令会立即运行。之后多次运行该命令需要很长时间才能执行（大约 30 - 40 分钟）。我们检查了服务器 CPU 和内存 - 内存不足或应用程序占用处理器资源没有问题。我们无法确定此命令运行时的延迟位置。

此 Hadoop/Spark 设置在运行 Red Hat 7.9 的集群中运行。在这种环境下，我们不会遇到这个问题。

这是我第一次在 Stack Overflow 上提问。如果我需要提供更多信息，请告诉我。

提前致谢。

============================ 5 月 11 日编辑：

日志运行成功（在 spark-submit 命令中添加了调试行）

bash-5.0$ spark-submit --version
Entered spark submit
About to execute spark submit command.....
About to load spark env.sh
Loaded spark env.sh
Entered statement to create RUNNER
searching spark_home/jars
Loaded spark jars DIR
Launching class path
Launched class path
Entering build command
Completed build command
About to enter while block
Entered while block for Entered build command
Entered build command
CMD is
build_command is  and org.apache.spark.deploy.SparkSubmit --version
Entered while block for
For  changing delim to blank
CMD is
build_command is  and org.apache.spark.deploy.SparkSubmit --version
Entered while block for /u01/app/java8_64/bin/java
Entered if condition for /u01/app/java8_64/bin/java
CMD is /u01/app/java8_64/bin/java
build_command is  and org.apache.spark.deploy.SparkSubmit --version
Entered while block for -cp
Entered if condition for -cp
CMD is /u01/app/java8_64/bin/java -cp
build_command is  and org.apache.spark.deploy.SparkSubmit --version
Entered while block for /u01/app/rmb/ria/AnthemSpark/spark-3.0.1-bin-hadoop3.2/conf/:/u01/app/rmb/ria/AnthemSpark/spark-3.0.1-bin-hadoop3.2/jars/*:/u01/app/rmb/ria/AnthemSpark/hadoop-3.2.1/etc/hadoop/
Entered if condition for /u01/app/rmb/ria/AnthemSpark/spark-3.0.1-bin-hadoop3.2/conf/:/u01/app/rmb/ria/AnthemSpark/spark-3.0.1-bin-hadoop3.2/jars/*:/u01/app/rmb/ria/AnthemSpark/hadoop-3.2.1/etc/hadoop/
CMD is /u01/app/java8_64/bin/java -cp /u01/app/rmb/ria/AnthemSpark/spark-3.0.1-bin-hadoop3.2/conf/:/u01/app/rmb/ria/AnthemSpark/spark-3.0.1-bin-hadoop3.2/jars/*:/u01/app/rmb/ria/AnthemSpark/hadoop-3.2.1/etc/hadoop/
build_command is  and org.apache.spark.deploy.SparkSubmit --version
Entered while block for -Xmx1g
Entered if condition for -Xmx1g
CMD is /u01/app/java8_64/bin/java -cp /u01/app/rmb/ria/AnthemSpark/spark-3.0.1-bin-hadoop3.2/conf/:/u01/app/rmb/ria/AnthemSpark/spark-3.0.1-bin-hadoop3.2/jars/*:/u01/app/rmb/ria/AnthemSpark/hadoop-3.2.1/etc/hadoop/ -Xmx1g
build_command is  and org.apache.spark.deploy.SparkSubmit --version
Entered while block for org.apache.spark.deploy.SparkSubmit
Entered if condition for org.apache.spark.deploy.SparkSubmit
CMD is /u01/app/java8_64/bin/java -cp /u01/app/rmb/ria/AnthemSpark/spark-3.0.1-bin-hadoop3.2/conf/:/u01/app/rmb/ria/AnthemSpark/spark-3.0.1-bin-hadoop3.2/jars/*:/u01/app/rmb/ria/AnthemSpark/hadoop-3.2.1/etc/hadoop/ -Xmx1g org.apache.spark.deploy.SparkSubmit
build_command is  and org.apache.spark.deploy.SparkSubmit --version
Entered while block for --version
Entered if condition for --version
CMD is /u01/app/java8_64/bin/java -cp /u01/app/rmb/ria/AnthemSpark/spark-3.0.1-bin-hadoop3.2/conf/:/u01/app/rmb/ria/AnthemSpark/spark-3.0.1-bin-hadoop3.2/jars/*:/u01/app/rmb/ria/AnthemSpark/hadoop-3.2.1/etc/hadoop/ -Xmx1g org.apache.spark.deploy.SparkSubmit --version
build_command is  and org.apache.spark.deploy.SparkSubmit --version
Entered while block for 0
Entered if condition for 0
CMD is /u01/app/java8_64/bin/java -cp /u01/app/rmb/ria/AnthemSpark/spark-3.0.1-bin-hadoop3.2/conf/:/u01/app/rmb/ria/AnthemSpark/spark-3.0.1-bin-hadoop3.2/jars/*:/u01/app/rmb/ria/AnthemSpark/hadoop-3.2.1/etc/hadoop/ -Xmx1g org.apache.spark.deploy.SparkSubmit --version 0
build_command is  and org.apache.spark.deploy.SparkSubmit --version
CMD is /u01/app/java8_64/bin/java -cp /u01/app/rmb/ria/AnthemSpark/spark-3.0.1-bin-hadoop3.2/conf/:/u01/app/rmb/ria/AnthemSpark/spark-3.0.1-bin-hadoop3.2/jars/*:/u01/app/rmb/ria/AnthemSpark/hadoop-3.2.1/etc/hadoop/ -Xmx1g org.apache.spark.deploy.SparkSubmit --version 0
completed while block
About to execute /u01/app/java8_64/bin/java -cp /u01/app/rmb/ria/AnthemSpark/spark-3.0.1-bin-hadoop3.2/conf/:/u01/app/rmb/ria/AnthemSpark/spark-3.0.1-bin-hadoop3.2/jars/*:/u01/app/rmb/ria/AnthemSpark/hadoop-3.2.1/etc/hadoop/ -Xmx1g 
   org.apache.spark.deploy.SparkSubmit --version
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 3.0.1
      /_/

Using Scala version 2.12.10, IBM J9 VM, 1.8.0_251
Branch HEAD
Compiled by user ubuntu on 2020-08-28T08:58:35Z
Revision 2b147c4cd50da32fe2b4167f97c8142102a0510d
Url https://gitbox.apache.org/repos/asf/spark.git
Type --help for more information.

================================================ ===============

运行失败：

bash-5.0$ spark-submit --version
Entered spark submit
About to execute spark submit command.....
About to load spark env.sh
Loaded spark env.sh
Entered statement to create RUNNER
searching spark_home/jars
Loaded spark jars DIR
Launching class path
Launched class path
Entering build command
Completed build command
About to enter while block
Entered while block for Entered build command
Entered build command
CMD is
build_command is  and org.apache.spark.deploy.SparkSubmit --version
Entered while block for
For  changing delim to blank
CMD is
build_command is  and org.apache.spark.deploy.SparkSubmit --version
Entered while block for /u01/app/java8_64/bin/java
Entered if condition for /u01/app/java8_64/bin/java
CMD is /u01/app/java8_64/bin/java
build_command is  and org.apache.spark.deploy.SparkSubmit --version
Entered while block for -cp
Entered if condition for -cp
CMD is /u01/app/java8_64/bin/java -cp
build_command is  and org.apache.spark.deploy.SparkSubmit --version
Entered while block for /u01/app/rmb/ria/AnthemSpark/spark-3.0.1-bin-hadoop3.2/conf/:/u01/app/rmb/ria/AnthemSpark/spark-3.0.1-bin-hadoop3.2/jars/*:/u01/app/rmb/ria/AnthemSpark/hadoop-3.2.1/etc/hadoop/
Entered if condition for /u01/app/rmb/ria/AnthemSpark/spark-3.0.1-bin-hadoop3.2/conf/:/u01/app/rmb/ria/AnthemSpark/spark-3.0.1-bin-hadoop3.2/jars/*:/u01/app/rmb/ria/AnthemSpark/hadoop-3.2.1/etc/hadoop/
CMD is /u01/app/java8_64/bin/java -cp /u01/app/rmb/ria/AnthemSpark/spark-3.0.1-bin-hadoop3.2/conf/:/u01/app/rmb/ria/AnthemSpark/spark-3.0.1-bin-hadoop3.2/jars/*:/u01/app/rmb/ria/AnthemSpark/hadoop-3.2.1/etc/hadoop/
build_command is  and org.apache.spark.deploy.SparkSubmit --version
Entered while block for -Xmx1g
Entered if condition for -Xmx1g
CMD is /u01/app/java8_64/bin/java -cp /u01/app/rmb/ria/AnthemSpark/spark-3.0.1-bin-hadoop3.2/conf/:/u01/app/rmb/ria/AnthemSpark/spark-3.0.1-bin-hadoop3.2/jars/*:/u01/app/rmb/ria/AnthemSpark/hadoop-3.2.1/etc/hadoop/ -Xmx1g
build_command is  and org.apache.spark.deploy.SparkSubmit --version
Entered while block for org.apache.spark.deploy.SparkSubmit
Entered if condition for org.apache.spark.deploy.SparkSubmit
CMD is /u01/app/java8_64/bin/java -cp /u01/app/rmb/ria/AnthemSpark/spark-3.0.1-bin-hadoop3.2/conf/:/u01/app/rmb/ria/AnthemSpark/spark-3.0.1-bin-hadoop3.2/jars/*:/u01/app/rmb/ria/AnthemSpark/hadoop-3.2.1/etc/hadoop/ -Xmx1g org.apache.spark.deploy.SparkSubmit
build_command is  and org.apache.spark.deploy.SparkSubmit --version
Entered while block for --version
Entered if condition for --version
CMD is /u01/app/java8_64/bin/java -cp /u01/app/rmb/ria/AnthemSpark/spark-3.0.1-bin-hadoop3.2/conf/:/u01/app/rmb/ria/AnthemSpark/spark-3.0.1-bin-hadoop3.2/jars/*:/u01/app/rmb/ria/AnthemSpark/hadoop-3.2.1/etc/hadoop/ -Xmx1g org.apache.spark.deploy.SparkSubmit --version
build_command is  and org.apache.spark.deploy.SparkSubmit --version
Entered while block for 0
Entered if condition for 0
CMD is /u01/app/java8_64/bin/java -cp /u01/app/rmb/ria/AnthemSpark/spark-3.0.1-bin-hadoop3.2/conf/:/u01/app/rmb/ria/AnthemSpark/spark-3.0.1-bin-hadoop3.2/jars/*:/u01/app/rmb/ria/AnthemSpark/hadoop-3.2.1/etc/hadoop/ -Xmx1g org.apache.spark.deploy.SparkSubmit --version 0
build_command is  and org.apache.spark.deploy.SparkSubmit --version

##########################

编辑 - 3 月 12 日

这是运行以下命令时的最后几行 - truss -d。最后的输出显示它进入“睡眠”状态。

0.9063:        lseek(0, 0, 1)                   Err#29 ESPIPE
0.9066:        fstatx(0, 0x0FFFFFFFFFFFE8F8, 176, 0) = 0
0.9068:        _sigaction(14, 0x0FFFFFFFFFFFE710, 0x0FFFFFFFFFFFE740) = 0
0.9071:        incinterval(0, 0x0FFFFFFFFFFFE640, 0x0FFFFFFFFFFFE660) = 0
0.9073:        kread(0, " o", 1)                = 1
0.9075:        kread(0, " r", 1)                = 1
0.9078:        kread(0, " g", 1)                = 1
0.9080:        kread(0, " .", 1)                = 1
0.9082:        kread(0, " a", 1)                = 1
0.9084:        kread(0, " p", 1)                = 1
0.9086:        kread(0, " a", 1)                = 1
0.9089:        kread(0, " c", 1)                = 1
0.9091:        kread(0, " h", 1)                = 1
0.9093:        kread(0, " e", 1)                = 1
0.9095:        kread(0, " .", 1)                = 1
0.9097:        kread(0, " s", 1)                = 1
0.9100:        kread(0, " p", 1)                = 1
0.9102:        kread(0, " a", 1)                = 1
0.9104:        kread(0, " r", 1)                = 1
0.9106:        kread(0, " k", 1)                = 1
0.9108:        kread(0, " .", 1)                = 1
0.9111:        kread(0, " d", 1)                = 1
0.9113:        kread(0, " e", 1)                = 1
0.9115:        kread(0, " p", 1)                = 1
0.9117:        kread(0, " l", 1)                = 1
0.9119:        kread(0, " o", 1)                = 1
0.9122:        kread(0, " y", 1)                = 1
0.9124:        kread(0, " .", 1)                = 1
0.9126:        kread(0, " S", 1)                = 1
0.9128:        kread(0, " p", 1)                = 1
0.9130:        kread(0, " a", 1)                = 1
0.9132:        kread(0, " r", 1)                = 1
0.9135:        kread(0, " k", 1)                = 1
0.9137:        kread(0, " S", 1)                = 1
0.9139:        kread(0, " u", 1)                = 1
0.9141:        kread(0, " b", 1)                = 1
0.9143:        kread(0, " m", 1)                = 1
0.9187:        kread(0, " i", 1)                = 1
0.9190:        kread(0, " t", 1)                = 1
0.9192:        kread(0, "\0", 1)                = 1
0.9195:        incinterval(0, 0x0FFFFFFFFFFFE5C0, 0x0FFFFFFFFFFFE5E0) = 0
0.9197:        _sigaction(14, 0x0FFFFFFFFFFFE690, 0x0FFFFFFFFFFFE6C0) = 0
0.9200:        kfcntl(1, F_GETFL, 0x0000000000000000) = 67110914
0.9204:        kfcntl(1, F_GETFL, 0x0000000000000000) = 67110914
0.9207:        kioctl(0, 22528, 0x0000000000000000, 0x0000000000000000) Err#25 ENOTTY
0.9211:        lseek(0, 0, 1)                   Err#29 ESPIPE
0.9214:        fstatx(0, 0x0FFFFFFFFFFFE8F8, 176, 0) = 0
0.9216:        _sigaction(14, 0x0FFFFFFFFFFFE710, 0x0FFFFFFFFFFFE740) = 0
0.9219:        incinterval(0, 0x0FFFFFFFFFFFE640, 0x0FFFFFFFFFFFE660) = 0
0.9222:        kread(0, " -", 1)                = 1
0.9224:        kread(0, " -", 1)                = 1
0.9227:        kread(0, " v", 1)                = 1
0.9229:        kread(0, " e", 1)                = 1
0.9231:        kread(0, " r", 1)                = 1
0.9234:        kread(0, " s", 1)                = 1
0.9236:        kread(0, " i", 1)                = 1
0.9238:        kread(0, " o", 1)                = 1
0.9241:        kread(0, " n", 1)                = 1
0.9243:        kread(0, "\0", 1)                = 1
0.9245:        incinterval(0, 0x0FFFFFFFFFFFE5C0, 0x0FFFFFFFFFFFE5E0) = 0
0.9248:        _sigaction(14, 0x0FFFFFFFFFFFE690, 0x0FFFFFFFFFFFE6C0) = 0
0.9251:        kfcntl(1, F_GETFL, 0x0000000000000000) = 67110914
0.9254:        kfcntl(1, F_GETFL, 0x0000000000000000) = 67110914
0.9257:        kioctl(0, 22528, 0x0000000000000000, 0x0000000000000000) Err#25 ENOTTY
0.9260:        lseek(0, 0, 1)                   Err#29 ESPIPE
0.9262:        fstatx(0, 0x0FFFFFFFFFFFE8F8, 176, 0) = 0
0.9265:        _sigaction(14, 0x0FFFFFFFFFFFE710, 0x0FFFFFFFFFFFE740) = 0
0.9268:        incinterval(0, 0x0FFFFFFFFFFFE640, 0x0FFFFFFFFFFFE660) = 0
0.9270:        kread(0, " 0", 1)                = 1
0.9273:        kread(0, "\0", 1)                = 1
0.9275:        incinterval(0, 0x0FFFFFFFFFFFE5C0, 0x0FFFFFFFFFFFE5E0) = 0
0.9278:        _sigaction(14, 0x0FFFFFFFFFFFE690, 0x0FFFFFFFFFFFE6C0) = 0
0.9281:        kfcntl(1, F_GETFL, 0x0000000000000000) = 67110914
0.9284:        kfcntl(1, F_GETFL, 0x0000000000000020) = 67110914
0.9287:        kioctl(0, 22528, 0x0000000000000000, 0x0000000000000000) Err#25 ENOTTY
0.9290:        lseek(0, 0, 1)                   Err#29 ESPIPE
0.9292:        fstatx(0, 0x0FFFFFFFFFFFE8F8, 176, 0) = 0
0.9295:        _sigaction(14, 0x0FFFFFFFFFFFE710, 0x0FFFFFFFFFFFE740) = 0
0.9297:        incinterval(0, 0x0FFFFFFFFFFFE640, 0x0FFFFFFFFFFFE660) = 0
2.9303:        kread(0, "\t", 1) (sleeping...)

【问题讨论】：

使用truss -d 调试可能是一个开始。
欢迎您！写得很好的问题，谢谢。请考虑添加一些运行 spark-submit --verbose --version 的日志，理想情况下来自慢速运行和快速运行。它可能会帮助某人回答具体问题，而不是笼统地回答！
嗨@ariels 我已经用成功和失败的日志编辑了帖子。我将在当天晚些时候运行 --verbose --version 命令并使用此命令更新帖子。
嗨@LorinczyZsigmond，我已经使用您提供的命令编辑了帖子。 spark-submit 命令正在读取同一 $SPARK_HOME/bin 文件夹中的 spark-class 文件。它需要一段时间来构建 spark-submit 命令。当它到达验证时它似乎挂起 - 它到达了 while 循环的结尾。

标签： apache-spark hadoop aix

【解决方案1】：

我没有足够的声望点来发表评论，所以我会在这里发布。 @LorinczyZsigmond 的建议最终将我们引导到了正确的位置。搜索最后一行 - kread(0, "\t", 1) (sleeping...) - 将我们引向 Issue with Bash read builtin in latest release，在那里有人讨论其他人与 IBM AIX Linux Toolbox 网站上发布的版本存在相同问题。更新到版本 5.1.4.2 后，问题消失了，我们能够完成脚本。谢谢大家的帮助。

【讨论】：