MRUnit reducer 测试：值类不匹配答案

【问题标题】：MRUnit reducer test: Mismatch in value classMRUnit reducer 测试：值类不匹配
【发布时间】：2014-08-09 04:33:20
【问题描述】：

我的 reducer 单元测试在迁移到 MapReduce 2 后抛出了 "Mismatch in value class" 异常：

Mismatch in value class: expected: class org.apache.hadoop.io.IntWritable actual: class com.company.MyWritable

错误消息本身对我来说很清楚，但我不明白为什么 MRUnit 获得临时可写类而不是 IntWritable。

reducer 实现：

public static class TestCountReduce extends
        Reducer<Text, MyWritable, Text, IntWritable> {

    public void reduce(Text key, Iterator<MyWritable> values,
            Context context) throws IOException, InterruptedException {

        ...
        context.write(key, new IntWritable(s.size()));
    }
}

测试设置：

public void setUp() throws IOException {
    Mapper<Object, Text, Text, MyWritable> mapper = new MyMapper();
    Reducer<Text, MyWritable, Text, IntWritable> reducer = new MyReducer();

    mapDriver = new MapDriver<Object, Text, Text, MyWritable>();
    mapDriver.setMapper(mapper);

    reduceDriver = new ReduceDriver<Text, MyWritable, Text, IntWritable>();
    reduceDriver.setReducer(reducer);
}

最后是测试用例：

@Test
public void testReducer() throws IOException {
    List<MyWritable> values = new ArrayList<MyWritable>();
    values.add(new MyWritable("1"));
    values.add(new MyWritable("1"));
    reduceDriver.withInput(new Text("testkey"), values);
    reduceDriver.withOutput(new Text("testkey"), new IntWritable(1));
    reduceDriver.runTest();
}

【问题讨论】：

标签： java hadoop mapreduce reduce mrunit

【解决方案1】：

请检查 reducer 实现中的 reduce 方法签名

应该是

public void reduce(Text key, Iterable<MyWritable> values, Context context) throws IOException, InterruptedException {

而不是

public void reduce(Text key, Iterator<MyWritable> values, Context context) throws IOException, InterruptedException {

【讨论】：