【问题标题】:org.apache.hadoop.fs.BlockMissingException in hadoop jobhadoop 作业中的 org.apache.hadoop.fs.BlockMissingException
【发布时间】:2015-09-06 13:10:26
【问题描述】:

我在 Hadoop 作业中遇到以下异常。我无法弄清楚这个异常的原因。你能帮我解决这个错误吗?为什么会出现此错误?

异常堆栈跟踪:

org.apache.hadoop.fs.BlockMissingException: Could not obtain block: blk_6488469967470272993_1110 file=/media/ephemeral0/hadoop-root/5614861051_themeToProductsPipe_F832FB7243E64F41892F20744AF6A0B7/part-00054
    at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:3158)
    at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:2940)
    at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:3107)
    at java.io.DataInputStream.readFully(DataInputStream.java:195)
    at java.io.DataInputStream.readFully(DataInputStream.java:169)
    at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1576)
    at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1554)
    at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1543)
    at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1538)
    at org.apache.hadoop.mapred.SequenceFileRecordReader.(SequenceFileRecordReader.java:47)
    at org.apache.hadoop.mapred.SequenceFileInputFormat.getRecordReader(SequenceFileInputFormat.java:64)
    at cascading.tap.hadoop.io.MultiInputFormat$1.operate(MultiInputFormat.java:253)
    at cascading.tap.hadoop.io.MultiInputFormat$1.operate(MultiInputFormat.java:248)
    at cascading.util.Util.retry(Util.java:762)
    at cascading.tap.hadoop.io.MultiInputFormat.getRecordReader(MultiInputFormat.java:247)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:346)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:312)
    at org.apache.hadoop.mapred.Child$3.run(Child.java:205)

【问题讨论】:

    标签: hadoop mapreduce hdfs bigdata cascading


    【解决方案1】:

    该错误意味着客户端无法按时命中包含该块的数据节点。这可能是由于网络问题或数据节点太忙而无法响应或已关闭。

    异常不足以追踪根本原因。

    【讨论】:

      猜你喜欢
      • 1970-01-01
      • 1970-01-01
      • 2014-10-13
      • 1970-01-01
      • 1970-01-01
      • 1970-01-01
      • 1970-01-01
      • 2015-05-23
      • 1970-01-01
      相关资源
      最近更新 更多