为了做mapreduce开发,要使用eclipse,并且需要对应的Hadoop插件hadoop-eclipse-plugin-2.7.3.jar,首先说明一下,在hadoop1.x之前官方hadoop安装包中都自带有eclipse的插件,而如今随着程序员的开发工具eclipse版本的增多和差异,hadoop插件也必须要和开发工具匹配,hadoop的插件包也不可能全部兼容.为了简化,如今的hadoop安装包内不会含有eclipse的插件.需要各自根据自己的eclipse自行编译.
1. 环境准备
使用ant制作自己的eclipse插件,介绍一下我的环境和工具 ( 安装路径根据自己 )
系统: 64bit Ubuntu 14.04,(系统不重要Win也可以,方法都一样)
JDK 版本: jdk-7u80-linux-x64.tar.gz 安装路径： /usr/lib/jvm
eclipse 版本: ide工具eclipse-jee-mars-2-linux-gtk-x86_64.tar.gz 安装路径： /home/hadoop/
hadoop 版本: hadoop-2.7.3.tar.gz 安装路径：/usr/local
ant(这个也随意,二进制安装或者apt-get安装都可以,配置好环境变量) , 我的 ant 版本是1.9.3 , 有的是 1.9.7
export ANT_HOME=/usr/local/apache-ant-1.9.3
export PATH=$PATH:$ANT_HOME/bin
如果提示找不到ant的launcher.ja包,添加环境变量
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/jre/lib:$JAVA_HOME/lib/toos.jar:$ANT_HOME/lib/ant-launcher.jar
$ ant -version
Apache Ant(TM) version 1.9.3 compiled on April 8 2014
我使用的是GitHub 上的 hadoop2x-eclipse-plugin （https://github.com/winghc/hadoop2x-eclipse-plugin）提供的 hadoop2x-eclipse-plugin-master.zip ( 这是最新的 ,它对应 hadoop2x-eclipse-plugin-2.6.0.zip ),
以zip格式下载,然后解压到一个合适的路径下.注意路径的权限和目录所有者是当前用户下的.
三个编译工具和资源的路径如下:
eclipse : /home/hadoop/eclipse hadoop : /usr/local/hadoop-2.7.3 hadoop2x-eclipse-plugin-master : /home/hadoop/hadoop2x-eclipse-plugin-master
2 制作 hadoop-eclipse-plugin-2.7.3.jar
但是我的hadoop版本是2.7.3,所以需要另外制作插件，好在这篇Github文章提供了怎么build插件的方法。
解压下载过来的hadoop2x-eclipse-plugin,进入其中目录hadoop2x-eclipse-plugin-master/src/contrib/eclipse-plugin/执行操作, Github文章提供了怎么build插件的方法如下所示:
eclipse plugin for hadoop 2.x.x How to build [hdpusr@demo hadoop2x-eclipse-plugin]$ cd src/contrib/eclipse-plugin # Assume hadoop installation directory is /usr/share/hadoop [hdpusr@apclt eclipse-plugin]$ ant jar -Dversion=2.4.1 -Dhadoop.version=2.4.1 -Declipse.home=/opt/eclipse -Dhadoop.home=/usr/share/hadoop final jar will be generated at directory ${hadoop2x-eclipse-plugin}/build/contrib/eclipse-plugin/hadoop-eclipse-plugin-2.4.1.jar
2.1 修改相关文件, 主要有两个,一个是 hadoop2x-eclipse-plugin-master/src/contrib/eclipse-plugin/build.xml , 第二个是 hadoop2x-eclipse-plugin-master/ivy/libraries.properties
但是此时我需要的是 hadoop 2.7.3的 eclilpse 插件,而 github 下载过来的 hadoop2x-eclipse-plugin 配置是 hadoop2.6 的编译环境,所以执行 ant 之前需要需要修改 ant 的 build.xml 配置文件以及相关文件
第一个文件: hadoop2x-eclipse-plugin-master/src/contrib/eclipse-plugin/build.xml , 如下所示:
1 <?xml version="1.0" encoding="UTF-8" standalone="no"?> 2 3 <!-- 4 Licensed to the Apache Software Foundation (ASF) under one or more 5 contributor license agreements. See the NOTICE file distributed with 6 this work for additional information regarding copyright ownership. 7 The ASF licenses this file to You under the Apache License, Version 2.0 8 (the "License"); you may not use this file except in compliance with 9 the License. You may obtain a copy of the License at 10 11 http://www.apache.org/licenses/LICENSE-2.0 12 13 Unless required by applicable law or agreed to in writing, software 14 distributed under the License is distributed on an "AS IS" BASIS, 15 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 See the License for the specific language governing permissions and 17 limitations under the License. 18 --> 19 20 <project default="jar" name="eclipse-plugin"> 21 22 <import file="../build-contrib.xml"/> 23 24 <path id="eclipse-sdk-jars"> 25 <fileset dir="${eclipse.home}/plugins/"> 26 <include name="org.eclipse.ui*.jar"/> 27 <include name="org.eclipse.jdt*.jar"/> 28 <include name="org.eclipse.core*.jar"/> 29 <include name="org.eclipse.equinox*.jar"/> 30 <include name="org.eclipse.debug*.jar"/> 31 <include name="org.eclipse.osgi*.jar"/> 32 <include name="org.eclipse.swt*.jar"/> 33 <include name="org.eclipse.jface*.jar"/> 34 35 <include name="org.eclipse.team.cvs.ssh2*.jar"/> 36 <include name="com.jcraft.jsch*.jar"/> 37 </fileset> 38 </path> 39 40 <path id="hadoop-sdk-jars"> 41 <fileset dir="${hadoop.home}/share/hadoop/mapreduce"> 42 <include name="hadoop*.jar"/> 43 </fileset> 44 <fileset dir="${hadoop.home}/share/hadoop/hdfs"> 45 <include name="hadoop*.jar"/> 46 </fileset> 47 <fileset dir="${hadoop.home}/share/hadoop/common"> 48 <include name="hadoop*.jar"/> 49 </fileset> 50 </path> 51 52 53 54 <!-- Override classpath to include Eclipse SDK jars --> 55 <path id="classpath"> 56 <pathelement location="${build.classes}"/> 57 <!--pathelement location="${hadoop.root}/build/classes"/--> 58 <path refid="eclipse-sdk-jars"/> 59 <path refid="hadoop-sdk-jars"/> 60 </path> 61 62 <!-- Skip building if eclipse.home is unset. --> 63 <target name="check-contrib" unless="eclipse.home"> 64 <property name="skip.contrib" value="yes"/> 65 <echo message="eclipse.home unset: skipping eclipse plugin"/> 66 </target> 67 68 <target name="compile" depends="init, ivy-retrieve-common" unless="skip.contrib"> 69 <echo message="contrib: ${name}"/> 70 <javac 71 encoding="${build.encoding}" 72 srcdir="${src.dir}" 73 includes="**/*.java" 74 destdir="${build.classes}" 75 debug="${javac.debug}" 76 deprecation="${javac.deprecation}"> 77 <classpath refid="classpath"/> 78 </javac> 79 </target> 80 81 <!-- Override jar target to specify manifest --> 82 <target name="jar" depends="compile" unless="skip.contrib"> 83 <mkdir dir="${build.dir}/lib"/> 84 <copy todir="${build.dir}/lib/" verbose="true"> 85 <fileset dir="${hadoop.home}/share/hadoop/mapreduce"> 86 <include name="hadoop*.jar"/> 87 </fileset> 88 </copy> 89 <copy todir="${build.dir}/lib/" verbose="true"> 90 <fileset dir="${hadoop.home}/share/hadoop/common"> 91 <include name="hadoop*.jar"/> 92 </fileset> 93 </copy> 94 <copy todir="${build.dir}/lib/" verbose="true"> 95 <fileset dir="${hadoop.home}/share/hadoop/hdfs"> 96 <include name="hadoop*.jar"/> 97 </fileset> 98 </copy> 99 <copy todir="${build.dir}/lib/" verbose="true"> 100 <fileset dir="${hadoop.home}/share/hadoop/yarn"> 101 <include name="hadoop*.jar"/> 102 </fileset> 103 </copy> 104 105 <copy todir="${build.dir}/classes" verbose="true"> 106 <fileset dir="${root}/src/java"> 107 <include name="*.xml"/> 108 </fileset> 109 </copy> 110 111 112 113 <copy file="${hadoop.home}/share/hadoop/common/lib/protobuf-java-${protobuf.version}.jar" todir="${build.dir}/lib" verbose="true"/> 114 <copy file="${hadoop.home}/share/hadoop/common/lib/log4j-${log4j.version}.jar" todir="${build.dir}/lib" verbose="true"/> 115 <copy file="${hadoop.home}/share/hadoop/common/lib/commons-cli-${commons-cli.version}.jar" todir="${build.dir}/lib" verbose="true"/> 116 <copy file="${hadoop.home}/share/hadoop/common/lib/commons-configuration-${commons-configuration.version}.jar" todir="${build.dir}/lib" verbose="true"/> 117 <copy file="${hadoop.home}/share/hadoop/common/lib/commons-lang-${commons-lang.version}.jar" todir="${build.dir}/lib" verbose="true"/> 118 <copy file="${hadoop.home}/share/hadoop/common/lib/commons-collections-${commons-collections.version}.jar" todir="${build.dir}/lib" verbose="true"/> 119 <copy file="${hadoop.home}/share/hadoop/common/lib/jackson-core-asl-${jackson.version}.jar" todir="${build.dir}/lib" verbose="true"/> 120 <copy file="${hadoop.home}/share/hadoop/common/lib/jackson-mapper-asl-${jackson.version}.jar" todir="${build.dir}/lib" verbose="true"/> 121 <copy file="${hadoop.home}/share/hadoop/common/lib/slf4j-log4j12-${slf4j-log4j12.version}.jar" todir="${build.dir}/lib" verbose="true"/> 122 <copy file="${hadoop.home}/share/hadoop/common/lib/slf4j-api-${slf4j-api.version}.jar" todir="${build.dir}/lib" verbose="true"/> 123 <copy file="${hadoop.home}/share/hadoop/common/lib/guava-${guava.version}.jar" todir="${build.dir}/lib" verbose="true"/> 124 <copy file="${hadoop.home}/share/hadoop/common/lib/hadoop-auth-${hadoop.version}.jar" todir="${build.dir}/lib" verbose="true"/> 125 <copy file="${hadoop.home}/share/hadoop/common/lib/commons-cli-${commons-cli.version}.jar" todir="${build.dir}/lib" verbose="true"/> 126 <copy file="${hadoop.home}/share/hadoop/common/lib/netty-${netty.version}.jar" todir="${build.dir}/lib" verbose="true"/> 127 <copy file="${hadoop.home}/share/hadoop/common/lib/htrace-core-${htrace.version}.jar" todir="${build.dir}/lib" verbose="true"/> 128 129 <jar 130 jarfile="${build.dir}/hadoop-${name}-${hadoop.version}.jar" 131 manifest="${root}/META-INF/MANIFEST.MF"> 132 <manifest> 133 <attribute name="Bundle-ClassPath" 134 value="classes/, 135 lib/hadoop-mapreduce-client-core-${hadoop.version}.jar, 136 lib/hadoop-mapreduce-client-common-${hadoop.version}.jar, 137 lib/hadoop-mapreduce-client-jobclient-${hadoop.version}.jar, 138 lib/hadoop-auth-${hadoop.version}.jar, 139 lib/hadoop-common-${hadoop.version}.jar, 140 lib/hadoop-hdfs-${hadoop.version}.jar, 141 lib/protobuf-java-${protobuf.version}.jar, 142 lib/log4j-${log4j.version}.jar, 143 lib/commons-cli-${commons-cli.version}.jar, 144 lib/commons-configuration-${commons-configuration.version}.jar, 145 lib/commons-httpclient-${commons-httpclient.version}.jar, 146 lib/commons-lang-${commons-lang.version}.jar, 147 lib/commons-collections-${commons-collections.version}.jar, 148 lib/jackson-core-asl-${jackson.version}.jar, 149 lib/jackson-mapper-asl-${jackson.version}.jar, 150 lib/slf4j-log4j12-${slf4j-log4j12.version}.jar, 151 lib/slf4j-api-${slf4j-api.version}.jar, 152 lib/guava-${guava.version}.jar, 153 lib/netty-${netty.version}.jar, 154 lib/htrace-core-${htrace.version}.jar"/> 155 </manifest> 156 <fileset dir="${build.dir}" includes="classes/ lib/"/> 157 <!--fileset dir="${build.dir}" includes="*.xml"/--> 158 <fileset dir="${root}" includes="resources/ plugin.xml"/> 159 </jar> 160 </target> 161 162 </project>