上传资源文件时 createBlockOutputStream 中的 Spark HDFS 异常答案

【问题标题】：Spark HDFS Exception in createBlockOutputStream while uploading resource file上传资源文件时 createBlockOutputStream 中的 Spark HDFS 异常
【发布时间】：2016-01-21 19:14:05
【问题描述】：

我正在尝试使用 yarn-cluster 在集群中运行我的 JAR，但一段时间后我遇到了异常。失败前的最后一个INFO 是Uploading resource。我检查了所有安全组，hsdf ls 成功但仍然收到错误。

./bin/spark-submit --class MyMainClass --master yarn-cluster /tmp/myjar-1.0.jar myjarparameter

16/01/21 16:13:51 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/01/21 16:13:52 INFO client.RMProxy: Connecting to ResourceManager at yarn.myserver.com/publicip:publicport
16/01/21 16:13:53 INFO yarn.Client: Requesting a new application from cluster with 10 NodeManagers
16/01/21 16:13:53 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (13312 MB per container)
16/01/21 16:13:53 INFO yarn.Client: Will allocate AM container, with 896 MB memory including 384 MB overhead
16/01/21 16:13:53 INFO yarn.Client: Setting up container launch context for our AM
16/01/21 16:13:53 INFO yarn.Client: Preparing resources for our AM container
16/01/21 16:13:54 INFO yarn.Client: Uploading resource file:/opt/spark-1.2.0-bin-hadoop2.3/lib/spark-assembly-1.2.0-hadoop2.3.0.jar -> hdfs://hdfs.myserver.com/user/henrique/.sparkStaging/application_1452514285349_6427/spark-assembly-1.2.0-hadoop2.3.0.jar
16/01/21 16:14:55 INFO hdfs.DFSClient: Exception in createBlockOutputStream
org.apache.hadoop.net.ConnectTimeoutException: 60000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=/PRIVATE_IP:50010]
    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:532)
    at org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1341)
    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1167)
    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1122)
    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:522)
16/01/21 16:14:55 INFO hdfs.DFSClient: Abandoning BP-26920217-10.140.213.58-1440247331237:blk_1132201932_58466886
16/01/21 16:14:55 INFO hdfs.DFSClient: Excluding datanode 10.164.16.207:50010
16/01/21 16:15:55 INFO hdfs.DFSClient: Exception in createBlockOutputStream

./bin/hadoop fs -ls /user/henrique/.sparkStaging/

drwx------- henrique supergroup 0 2016-01-20 18:36 user/henrique/.sparkStaging/application_1452514285349_5868
drwx------ henrique supergroup 0 2016-01-21 16:13 user/henrique/.sparkStaging/application_1452514285349_6427
drwx------ henrique supergroup 0 2016-01-21 17:06 user/henrique/.sparkStaging/application_1452514285349_6443

【问题讨论】：

这个问题不太可能是代码问题，更像是 Hdoop 配置问题。检查 Yarn/Spark 使用的“用户”是否已经拥有访问 HDFS/文件系统所需的权限，并启用禁用防火墙或允许所需的 IP 和主机。您可能还需要查看this

标签： hadoop apache-spark hadoop-yarn

【解决方案1】：

已解决！ Hadoop 试图连接到私有 IP。通过将此配置添加到hsdf-site.xml

解决了问题

<property>
  <name>dfs.client.use.datanode.hostname</name>
  <value>true</value>
</property>

【讨论】：

您是否对所有节点上的 hdfs-site.xml 进行了编辑？