在 Cloudera Hadoop 中构建 MapReduce 程序时出错

Question

在 Hadoop 中构建 MapReduce 文件时出现以下错误。 我正在使用 Cloudera hadoop 发行版。 testmr_classes 是一个文件夹，TestMR.java 是 MapReduce 文件

[cloudera@localhost ~]$ echo `hadoop classpath`
/etc/hadoop/conf:/usr/lib/hadoop/lib/*:/usr/lib/hadoop/.//*:/usr/lib/hadoop-hdfs/./:/usr/lib/hadoop-hdfs/lib/*:/usr/lib/hadoop-hdfs/.//*:/usr/lib/hadoop-yarn/lib/*:/usr/lib/hadoop-yarn/.//*:/usr/lib/hadoop-0.20-mapreduce/./:/usr/lib/hadoop-0.20-mapreduce/lib/*:/usr/lib/hadoop-0.20-mapreduce/.//*
[cloudera@localhost ~]$

[cloudera@localhost ~]$ javac -classpath `hadoop classpath`:. -d testmr_classes TestMR.java
TestMR.java:32: TestMR.Reduce is not abstract and does not override abstract method reduce(org.apache.hadoop.io.IntWritable,java.util.Iterator<org.apache.hadoop.io.Text>,org.apache.hadoop.mapred.OutputCollector<org.apache.hadoop.io.IntWritable,org.apache.hadoop.io.DoubleWritable>,org.apache.hadoop.mapred.Reporter) in org.apache.hadoop.mapred.Reducer
    public static class Reduce extends MapReduceBase implements Reducer<IntWritable,Text,IntWritable,DoubleWritable>
                  ^
1 error
[cloudera@localhost ~]$

下面是TestMR.java的内容，

import java.io.IOException;
import java.util.*;

import org.apache.hadoop.fs.Path;
import org.apache.hadoop.conf.*;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapred.*;
import org.apache.hadoop.mapred.Reducer;
import org.apache.hadoop.util.*;

public class TestMR 
{
    public static class Map extends MapReduceBase implements Mapper<IntWritable,Text,IntWritable,Text>
    {
        private IntWritable key = new IntWritable();
        private Text value = new Text();

        public void map(IntWritable key, Text line, OutputCollector<IntWritable, Text> output, Reporter reporter) throws IOException
        {
            String [] split = line.toString().split(",");
            key.set(Integer.parseInt(split[0]));

            if(split[2] == "Test")
            {
                value.set(split[4] + "," + split[7]);
                output.collect(key, value);
            }
        }
    }

    public static class Reduce extends MapReduceBase implements Reducer<IntWritable,Text,IntWritable,DoubleWritable>
    {
        public void reduce(IntWritable key, Iterable<Text> v, OutputCollector<IntWritable, DoubleWritable> output, Reporter reporter) throws IOException
        {
            Iterator values = v.iterator();
            while(values.hasNext())
            {
                String [] tmp_buf_1 = values.next().toString().split(",");
                String V1 = tmp_buf_1[0];
                String T1 = tmp_buf_1[1];

                if(!values.hasNext())   
                    break;  

                String [] tmp_buf_2 = values.next().toString().split(",");       
                String V2 = tmp_buf_2[0];
                String T2 = tmp_buf_2[1];           

                double dResult = (Double.parseDouble(V2) - Double.parseDouble(V1)) / (Double.parseDouble(T2) - Double.parseDouble(T1));

                output.collect(key, new DoubleWritable(dResult));
            }
        }
    }

    public static void main(String[] args) throws Exception
    {
        JobConf conf = new JobConf(TestMR.class);
        conf.setJobName("TestMapReduce");

        conf.setOutputKeyClass(IntWritable.class);
        conf.setOutputValueClass(DoubleWritable.class);

        conf.setMapperClass(Map.class);
        conf.setCombinerClass(Reduce.class);
        conf.setReducerClass(Reduce.class);

        conf.setInputFormat(TextInputFormat.class);
        conf.setOutputFormat(TextOutputFormat.class);

        FileInputFormat.setInputPaths(conf, new Path(args[0]));
        FileOutputFormat.setOutputPath(conf, new Path(args[1]));

        JobClient.runJob(conf);
    }
}

这是我第一次尝试 MapReduce，很高兴知道我是否在这里遗漏了什么。

Answer 1

仔细查看 reduce() 的第二个参数和错误。你写了 Iterable 但它给了你一个迭代器。

Answer 2

你需要指定你的类路径来代替'hadoop classpath'，例如：

javac -classpath /opt/hadoop-1.1.2/hadoop-core-1.1.2.jar:/opt/hadoop-1.1.2/lib/commons-cli-1.2.jar -d testmr_classes TestMR.java 

具体路径取决于您安装 hadoop 的位置。