YARN 集群模式上的 Spark 流式传输作业卡在接受状态，然后失败并出现超时异常答案

【问题标题】：Spark streaming job on YARN cluster mode stuck in accepted, then fails with a Timeout ExceptionYARN 集群模式上的 Spark 流式传输作业卡在接受状态，然后失败并出现超时异常
【发布时间】：2018-06-19 16:04:49
【问题描述】：

我正在运行一个 spark 流应用程序，它只是从 Kafka 主题读取消息，丰富它们，然后将丰富的消息写入另一个 kafka 主题。我已经在独立模式（客户端和集群部署模式）和 YARN 客户端模式下成功地尝试过它。当我以集群模式提交应用程序时，它会给我以下消息：

18/01/10 12:13:34 INFO Client: Submitting application application_1515582681419_0001 to ResourceManager
18/01/10 12:13:34 INFO YarnClientImpl: Submitted application application_1515582681419_0001
18/01/10 12:13:35 INFO Client: Application report for application_1515582681419_0001 (state: ACCEPTED)
18/01/10 12:13:35 INFO Client: 
     client token: N/A
     diagnostics: AM container is launched, waiting for AM container to Register with RM
     ApplicationMaster host: N/A
     ApplicationMaster RPC port: -1
     queue: default
     start time: 1515582814080
     final status: UNDEFINED
     tracking URL: http://ambari1.internal:8088/proxy/application_1515582681419_0001/
     user: root
18/01/10 12:13:36 INFO Client: Application report for application_1515582681419_0001 (state: ACCEPTED)
18/01/10 12:13:37 INFO Client: Application report for application_1515582681419_0001 (state: ACCEPTED)

并一直停留在接受状态，直到大约 4-5 分钟后，退出并显示以下错误消息：

18/01/10 12:17:00 INFO InputInfoTracker: remove old batch metadata: 1515583000000 ms
18/01/10 12:17:02 ERROR ApplicationMaster: Uncaught exception:
java.util.concurrent.TimeoutException: Futures timed out after [100000 milliseconds]
at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:201)
at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:423)
at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:282)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$main$1.apply$mcV$sp(ApplicationMaster.scala:768)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$2.run(SparkHadoopUtil.scala:67)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$2.run(SparkHadoopUtil.scala:66)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:66)
at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:766)
at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)
18/01/10 12:17:02 INFO ApplicationMaster: Final app status: FAILED, exitCode: 10, (reason: Uncaught exception: java.util.concurrent.TimeoutException: Futures timed out after [100000 milliseconds])
18/01/10 12:17:02 INFO StreamingContext: Invoking stop(stopGracefully=false) from shutdown hook
18/01/10 12:17:02 INFO ReceiverTracker: ReceiverTracker stopped
18/01/10 12:17:02 INFO JobGenerator: Stopping JobGenerator immediately

有趣的事实：如果我访问应用程序的年龄，我可以看到 Spark Context 已经启动并处理了一些消息。

有人可以帮我解决这个问题吗？ PS：这些是我的 YARN 集群的资源：

【问题讨论】：

ACCEPTED：等待 AM 容器被分配、启动并注册到 RM。即使我增加spark.yarn.am.waitTime... 运气不好
这有什么更新吗？我也有同样的问题。

标签： spark-streaming hadoop-yarn

【解决方案1】：

问题可能出在 Yarn“App Timeline Server”上。尝试重新启动它。

【讨论】：

【解决方案2】：

您是否正在使用本地 master 创建 spark 会话？请检查一下。

【讨论】：