Spark Job 失败，退出状态为 15答案

【问题标题】：Spark Job failing with exit status 15Spark Job 失败，退出状态为 15
【发布时间】：2016-02-08 15:33:33
【问题描述】：

我正在尝试在 spark 中运行简单的字数统计作业，但在运行作业时遇到异常。

    For more detailed output, check application tracking page:http://quickstart.cloudera:8088/proxy/application_1446699275562_0006/Then, click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_1446699275562_0006_02_000001
Exit code: 15
Stack trace: ExitCodeException exitCode=15:
        at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
        at org.apache.hadoop.util.Shell.run(Shell.java:455)
        at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
        at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)


Container exited with a non-zero exit code 15
Failing this attempt. Failing the application.
         ApplicationMaster host: N/A
         ApplicationMaster RPC port: -1
         queue: root.cloudera
         start time: 1446910483956
         final status: FAILED
         tracking URL: http://quickstart.cloudera:8088/cluster/app/application_1446699275562_0006
         user: cloudera
Exception in thread "main" org.apache.spark.SparkException: Application finished with failed status
        at org.apache.spark.deploy.yarn.Client.run(Client.scala:626)
        at org.apache.spark.deploy.yarn.Client$.main(Client.scala:651)
        at org.apache.spark.deploy.yarn.Client.main(Client.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

我通过以下命令检查了日志

yarn logs -applicationId application_1446699275562_0006

这是日志

 15/11/07 07:35:09 ERROR yarn.ApplicationMaster: User class threw exception: Output directory hdfs://quickstart.cloudera:8020/user/cloudera/WordCountOutput already exists
org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory hdfs://quickstart.cloudera:8020/user/cloudera/WordCountOutput already exists
        at org.apache.hadoop.mapred.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:132)
        at org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopDataset(PairRDDFunctions.scala:1053)
        at org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:954)
        at org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:863)
        at org.apache.spark.rdd.RDD.saveAsTextFile(RDD.scala:1290)
        at org.com.td.sparkdemo.spark.WordCount$.main(WordCount.scala:23)
        at org.com.td.sparkdemo.spark.WordCount.main(WordCount.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:480)
15/11/07 07:35:09 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: Output directory hdfs://quickstart.cloudera:8020/user/cloudera/WordCountOutput already exists)
15/11/07 07:35:14 ERROR yarn.ApplicationMaster: SparkContext did not initialize after waiting for 100000 ms. Please check earlier log output for errors. Failing the application.

异常清楚地表明 WordCountOutput 目录已经存在，但我在运行作业之前确保该目录不存在。

为什么在运行我的作业之前目录不存在但我收到此错误？

【问题讨论】：

但是从堆栈跟踪来看，很明显 hdfs://quickstart.cloudera:8020/user/cloudera/WordCountOutput 已经存在。
是的......但是在运行我的字数统计作业之前检查过......目录不存在......并且在运行代码后我得到了这个异常......
你是如何删除目录的？似乎在 S3 上，至少总是添加某种元文件以及“目录”。因此，如果我添加一个目录“myFolder”，还会添加另一个名为“myFolder_$folder$”的元文件。如果我只删除“myFolder”（而不是“myFolder_$folder$”）并尝试再次运行我的程序，我将收到您看到的错误。
我使用 hdfs 中的 hadoop fs -rmr /path/to/dir 命令删除了我的目录
你能解决这个问题吗？

标签： scala hadoop apache-spark

【解决方案1】：

我遇到了同样的问题并通过添加下面突出显示的部分来解决它。

SparkConf sparkConf = new SparkConf().setAppName("Sentiment Scoring").set("spark.hadoop.validateOutputSpecs", "true");

谢谢，萨蒂什。

【讨论】：

最新版本默认为true，但还是会出现这个错误。参考：spark.apache.org/docs/latest/configuration.html

【解决方案2】：

对我们来说，这是由“超出物理内存限制”造成的。增加执行器内存后，问题已修复。

【讨论】：

【解决方案3】：

此错误主要发生在您在 spark-submit 命令中提交错误的 spark 参数时。请检查配置参数。就我而言，我未能提交需要从 HDFS 读取资源的 masternamenode 地址。

之前：纱线抛出错误的地方 - 15 dfs://masternamenode/TESTCGNATDATA/

之后：能够运行应用程序 hdfs://masternamenode/TESTCGNATDATA/

【讨论】：