【发布时间】:2017-09-12 21:02:47
【问题描述】:
我在纱线集群中使用 oozie 配置协调器时遇到问题,这是一个 spark 作业,当我通过控制台运行工作流时,纱线会启动并正确执行该作业,但是当我从协调器调用相同的工作流时.xml 我有这个错误:
ERROR org.apache.spark.SparkContext - Error initializing SparkContext.
java.lang.IllegalArgumentException: Can not create a Path from an empty string
at org.apache.hadoop.fs.Path.checkPathArg(Path.java:127)
at org.apache.hadoop.fs.Path.<init>(Path.java:135)
at org.apache.hadoop.fs.Path.<init>(Path.java:94)
at org.apache.spark.deploy.yarn.Client.copyFileToRemote(Client.scala:337)
并且该作业永远不会在 yarn 集群中启动,看起来 yarn 无法从 oozie 接收 .jar 正确路径,知道吗?
这里的 coordinator.xml 和 workflow.xml 被简化了。
<coordinator-app name="Firebase acquisition process coordinator" frequency="${coord:days(1)}"
start="${startTime}" end="${endTime}" timezone="UTC" xmlns="uri:oozie:coordinator:0.5">
<controls>
...
</controls>
<action>
<workflow>
<app-path>hdfs://ip-111-11-11-111.us-west- 2.compute.internal:8020/user/hadoop/emr-spark/</app-path>
</workflow>
</action>
</coordinator-app>
<workflow-app name="bbbbbbbbbbbbbbb" xmlns="uri:oozie:workflow:0.5">
<start to="spark-0324"/>
<kill name="Kill">
<message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<action name="spark-0324">
<spark xmlns="uri:oozie:spark-action:0.2">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<master>yarn</master>
<mode>client</mode>
<class>classsxxx.Process</class>
<jar>hdfs://ip-111-11-11-111.us-west-2.compute.internal:8020/user/hadoop/emr-spark/lib/jarnamex.jar</jar>
<file>lib#lib</file>
</spark>
<ok to="End"/>
<error to="Kill"/>
</action>
<end name="End"/>
</workflow-app>
我的意思是,当我这样做时; oozie job -config ~/emr-spark/job.properties -run 它有效!!,但是当我尝试这个时; oozie job -run -config ~/emr-coordinator/coordinator.properties 不行。
职位属性
oozie.use.system.libpath=true
send_email=False
dryrun=False
nameNode=hdfs://ip-111-11-11-111.us-west-2.compute.internal:8020
jobTracker=ip-111-11-11-111.us-west-2.compute.internal:8032
oozie.wf.application.path=/user/hadoop/emr-spark
协调器属性
startTime=2017-09-08T19:46Z
endTime=2030-01-01T06:00Z
jobTracker=ip-111-11-11-111.us-west-2.compute.internal:8032
nameNode=hdfs://ip-111-11-11-111.us-west-2.compute.internal:8020
oozie.coord.application.path=hdfs://ip-111-11-11-111.us-west-2.compute.internal:8020/user/hadoop/emr-coordinator
oozie.use.system.libpath=true
【问题讨论】:
标签: apache-spark oozie emr oozie-coordinator