spark-submit python app with named argument via spark REST api答案

【问题标题】：spark-submit python app with named argument via spark REST apispark-submit python app with named argument via spark REST api
【发布时间】：2022-01-25 05:34:24
【问题描述】：

使用以下内容作为请求的主体来触发本机 REST api，该 API 执行带有命名参数 --filename 的 python 应用程序，通过 spark-submit 成功创建驱动程序，但执行失败，信息很少. --filename 参数是否导致了这种情况？有什么想法可以从哪里/如何获取有关失败的更多信息？

{ "action": "CreateSubmissionRequest",
  "appArgs": [
    "/opt/bitnami/spark/hostfiles/bronze.py --filename 'filename.json'"
  ],
 "appResource": "file:/opt/bitnami/spark/hostfiles/bronze.py",
 "clientSparkVersion": "3.2.0",
 "environmentVariables": {
    "SPARK_ENV_LOADED": "1"
  },
 "mainClass": "org.apache.spark.deploy.SparkSubmit",
  "sparkProperties": {
    "spark.driver.supervise": "false",
    "spark.app.name": "Spark REST API - Bronze Load",
    "spark.submit.deployMode": "client",
    "spark.master": "spark://spark:6066",
    "spark.eventLog.enabled":"true"
  }
}```

These are the logs from the worker container for the driver.

> 22/01/18 23:37:48 INFO Worker: Asked to launch driver driver-20220118233748-0006
> 22/01/18 23:37:48 INFO DriverRunner: Copying user jar file:/opt/bitnami/spark/hostfiles/bronze.py to /opt/bitnami/spark/work/driver-20220118233748-0006/bronze.py
> 22/01/18 23:37:48 INFO Utils: Copying /opt/bitnami/spark/hostfiles/bronze.py to /opt/bitnami/spark/work/driver-20220118233748-0006/bronze.py
> 22/01/18 23:37:48 INFO DriverRunner: Launch Command: "/opt/bitnami/java/bin/java" "-cp" "/opt/bitnami/spark/conf/:/opt/bitnami/spark/jars/*" "-Xmx1024M" "-Dspark.eventLog.enabled=true" "-Dspark.app.name=Spark REST API - Bronze Load" "-Dspark.driver.supervise=false" "-Dspark.master=spark://spark:7077" "-Dspark.submit.deployMode=client" "org.apache.spark.deploy.worker.DriverWrapper" "spark://Worker@192.168.32.4:35803" "/opt/bitnami/spark/work/driver-20220118233748-0006/bronze.py" "org.apache.spark.deploy.SparkSubmit" "/opt/bitnami/spark/hostfiles/bronze.py --filename 'filename.json'"
> 22/01/18 23:37:51 WARN Worker: Driver driver-20220118233748-0006 exited with failure

Meanwhile, the spark-submit successfully executes the app.
``` bin/spark-submit --deploy-mode "client" file:///opt/bitnami/spark/hostfiles/bronze.py --filename "filename.json"```

【问题讨论】：

--filename 是传递给 brown.py 的参数，以下 spark-submit 通过 CLI bin/spark-submit --deploy-mode "client" file:///opt/bitnami/spark/hostfiles/bronze.py --filename "filename.json" 成功

标签： apache-spark pyspark spark-submit

【解决方案1】：

以下对 api 请求正文的更改和处理 python 应用程序参数值而不需要 python 脚本中的“--filename”导致 python 应用程序正在执行。

更新了 appArgs 和 appResource 以使用 file:/// 来引用 python 应用程序。

  "appArgs": [
    "file:///opt/bitnami/spark/hostfiles/bronze.py", "filename.json"
  ],
 "appResource": "file:///opt/bitnami/spark/hostfiles/bronze.py",
 "clientSparkVersion": "3.2.0",
 "environmentVariables": {
    "SPARK_ENV_LOADED": "1"
  },
 "mainClass": "org.apache.spark.deploy.SparkSubmit",
  "sparkProperties": {
    "spark.driver.supervise": "false",
    "spark.app.name": "Spark REST API - Bronze Load",
    "spark.submit.deployMode": "client",
    "spark.master": "spark://spark:7077",
    "spark.eventLog.enabled":"true"
  }
}

【讨论】：