在Windows下面运行hadoop的MapReduce程序的方法:
1.下载hadoop的安装包,这里使用的是"hadoop-2.6.4.tar.gz":
2.将安装包直接解压到D盘根目录:
3.配置环境变量:
4.下载hadoop的eclipse插件,并将插件放到eclipse的plugins目录下:
5.打开Eclipse,选择菜单"Window"-->"Preferences",在左侧找到"Hadoop Map/Reduce",
在右侧选择hadoop的目录:
6.打开菜单"Window"中的"Show View"窗口,选择"Map/Reduce Locations":
7:在打开的"Map/Reduce Locations"面板中,点击小象图标,打开新建配置窗口:
8.按照下图所示,填写hadoop集群的主机地址和端口:
9.新创建的hadoop集群连接配置,右上角的齿轮可以修改配置信息:
10.打开菜单"Window"中的"Show View"窗口,找到"Project Explorer":
11.在"Project Explorer"面板中找到"DFS Locations",展开下面的菜单就可以连接上HDFS,
可以直接看到HDFS中的目录和文件:
12.在"Project Explorer"面板中点击鼠标右键,选择新建,就可以创建"Map/Reduce"项目了:
13.下面我们创建了一个名为"hadoop-test"的项目,可以看到它自动帮我们导入了很多的jar包:
14.在项目的src下面创建log4j.properties文件,内容如下:
log4j.rootLogger=debug,stdout,R log4j.appender.stdout=org.apache.log4j.ConsoleAppender log4j.appender.stdout.layout=org.apache.log4j.PatternLayout log4j.appender.stdout.layout.ConversionPattern=%5p - %m%n log4j.appender.R=org.apache.log4j.RollingFileAppender log4j.appender.R.File=mapreduce_test.log log4j.appender.R.MaxFileSize=1MB log4j.appender.R.MaxBackupIndex=1 log4j.appender.R.layout=org.apache.log4j.PatternLayout log4j.appender.R.layout.ConversionPattern=%p %t %c - %m%n log4j.logger.com.codefutures=DEBUG
15.在src下面创建org.apache.hadoop.io.nativeio包,并将hadoop源码中的NativeIO.java类拷贝到该包下(这一步是从网上看的,不知道有什么用)。
NativeIO.java类所在路径为:hadoop-2.6.4-src\hadoop-common-project\hadoop-common\src\main\java\org\apache\hadoop\io\nativeio
16.修改NativeIO.java类中的如下部分,返回"true"即可:
17.下面来写一个hadoop的MapReduce程序,经典的wordcount(单词计数)程序,代码如下(共3个java类):
(1)WordCountDriver.java类:
1 package com.xuebusi.hadoop.mr.windows; 2 3 import org.apache.hadoop.conf.Configuration; 4 import org.apache.hadoop.fs.Path; 5 import org.apache.hadoop.io.IntWritable; 6 import org.apache.hadoop.io.Text; 7 import org.apache.hadoop.mapreduce.Job; 8 import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; 9 import org.apache.hadoop.mapreduce.lib.input.TextInputFormat; 10 import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; 11 12 public class WordCountDriver { 13 14 public static void main(String[] args) throws Exception { 15 16 Configuration conf = new Configuration(); 17 Job job = Job.getInstance(conf); 18 19 job.setJarByClass(WordCountDriver.class); 20 21 job.setMapperClass(WordCountMapper.class); 22 job.setReducerClass(WordCountReducer.class); 23 24 job.setMapOutputKeyClass(Text.class); 25 job.setMapOutputValueClass(IntWritable.class); 26 27 job.setOutputKeyClass(Text.class); 28 job.setOutputValueClass(IntWritable.class); 29 30 job.setInputFormatClass(TextInputFormat.class); 31 32 FileInputFormat.setInputPaths(job, new Path("c:/wordcount/input")); 33 FileOutputFormat.setOutputPath(job, new Path("c:/wordcount/output")); 34 35 boolean res = job.waitForCompletion(true); 36 System.exit(res ? 0 : 1); 37 38 } 39 40 }