通过 NiFi 提交火花提交作业时 JSON 无效答案

【问题标题】：Invalid JSON while submitting spark submit job via NiFi通过 NiFi 提交火花提交作业时 JSON 无效
【发布时间】：2021-03-16 11:31:29
【问题描述】：

我正在尝试提交spark 作业，我在conf 属性中设置date 参数，并通过NiFi 中的脚本运行它。但是，当我运行脚本时，我遇到了错误。
Spark Submit 脚本中的代码：

aws emr add-steps --cluster-id "$1" --steps '[{"Args":["spark-submit","--deploy-mode","cluster","--jars","s3://tvsc-lumiq-edl/jars/ojdbc7.jar","--executor-memory","10g","--driver-memory","10g","--conf","spark.hadoop.yarn.timeline-service.enabled=false","--conf","currDate='\"$5\"'","--class",'\"$2\"','\"$3\"','\"$4\"'],"Type":"CUSTOM_JAR","ActionOnFailure":"CONTINUE","Jar":"command-runner.jar","Properties":"","Name":"Spark application"}]' --region "$6"

在我运行它之后，我得到以下错误：

ExecuteStreamCommand[id=5b08df5a-1f24-3958-30ca-2e27a6c4becf] Transferring flow file StandardFlowFileRecord[uuid=00f844ee-dbea-42a3-aba3-0edcabfc50a2,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1607082757752-507103, container=default, section=223], offset=29, length=-1],offset=0,name=6414901712887990,size=0] to nonzero status. Executable command /bin/bash ended in an error: 
Error parsing parameter '--steps': Invalid JSON:
[{"Args":["spark-submit","--deploy-mode","cluster","--jars","s3://tvsc-lumiq-edl/jars/ojdbc7.jar","--executor-memory","10g","--driver-memory","10g","--conf","spark.hadoop.yarn.timeline-service.enabled=false","--conf","currDate="Fri

我哪里出错了？

【问题讨论】：

标签： json apache-spark hadoop apache-nifi

【解决方案1】：

您可以使用JSONLint 来验证您的 JSON，这样可以更轻松地找出错误的原因。

在您的情况下，您将最后 3 个值括在单引号 ' 而不是双引号 "

您的steps JSON 应如下所示：

[{
"Args": [
    "spark-submit",
    "--deploy-mode",
    "cluster",
    "--jars",
    "s3://tvsc-lumiq-edl/jars/ojdbc7.jar",
    "--executor-memory",
    "10g",
    "--driver-memory",
    "10g",
    "--conf",
    "spark.hadoop.yarn.timeline-service.enabled=false",
    "--conf",
    "currDate='\"$5\"'",
    "--class",
    "\"$2\"",
    "\"$3\"",
    "\"$4\""
],
"Type": "CUSTOM_JAR",
"ActionOnFailure": "CONTINUE",
"Jar": "command-runner.jar",
"Properties": "",
"Name": "Spark application"
}]

具体来说，这3行：

"\"$2\"",
"\"$3\"",
"\"$4\""

代替原来的：

'\"$2\"',
'\"$3\"',
'\"$4\"'

【讨论】：