Hadoop Mapreduce：从不调用 Reducer

Question

我有一个简单的 Map/Reduce 任务。 Mapper 被正常调用和执行，但 reducer 从未被调用。

配置：

conf.setJobName("Index Builder");
conf.setSpeculativeExecution(false);

FileInputFormat.setInputPaths(conf, new Path(args[0].toString()));
FileOutputFormat.setOutputPath(conf, new Path(args[1].toString()));

conf.setInputFormat(TextInputFormat.class);
conf.setOutputFormat(NullOutputFormat.class);

conf.setMapperClass(IndexMapper.class);
conf.setReducerClass(IndexReducer.class);

conf.setMapOutputKeyClass(NullWritable.class);
conf.setMapOutputValueClass(NullWritable.class);
conf.setOutputValueClass(NullWritable.class);
conf.setOutputKeyClass(NullWritable.class);

映射器签名：

public class IndexMapper extends MapReduceBase implements
  Mapper<LongWritable, Text, NullWritable, NullWritable> {

   @Override
   public void map(LongWritable key, Text val,
     OutputCollector<NullWritable, NullWritable> output,
     Reporter reporter) throws IOException {
        // MAP FUNCTION
   }

}

ٌReducer 签名：

public class IndexReducer extends MapReduceBase implements
  Reducer<NullWritable, NullWritable, NullWritable, NullWritable> {

   @Override
   public void reduce(NullWritable arg0, Iterator<NullWritable> arg1,
     OutputCollector<NullWritable, NullWritable> arg2, Reporter reporter)
     throws IOException {
      // REDUCE CODE
   }
}

Answer 1

您将映射器的输出格式设置为 NullOutputFormat，因此它不会产生任何内容。因此，reducer 永远不会被调用，因为它需要一些数据才能运行。