简介:

  sqoop是一款用于hadoop和关系型数据库之间数据导入导出的工具。你可以通过sqoop把数据从数据库(比如mysql,oracle)导入到hdfs中;也可以把数据从hdfs中导出到关系型数据库中。通过将sqoop的操作命令转化为Hadoop的MapReduce作业进行导入导出,(通常只涉及到Map任务)即sqoop生成的Job主要是并发运行MapTask实现数据并行传输以提升数据传送速度和效率,如果使用Shell脚本来实现多线程数据传送则存在很大的难度Sqoop2(sqoop1.99.7)需要在Hadoop安装目录下的配置文件中设置代理,属于重量级嵌入安装,文中我们使用qoop1(Sqoop1.4.6)。

 

前提:(若不知道如何安装请看我前面写的hadoop分类的文章

CloudDeskTop上安装了: hadoop-2.7.3  jdk1.7.0_79  mysql-5.5.32 sqoop-1.4.6 hive-1.2.2
master01和master02安装了: hadoop-2.7.3 jdk1.7.0_79
slave01、slave02、slave03安装了: hadoop-2.7.3 jdk1.7.0_79 zookeeper-3.4.10

一、安装:

 1、上传安装包sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz到/install/目录下

2、解压:

[hadoop@CloudDeskTop install]$ tar -zxvf sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz -C /software/

3、配置环境:

[hadoop@CloudDeskTop software]$ su -lc "vi /etc/profile"

JAVA_HOME=/software/jdk1.7.0_79
HADOOP_HOME=/software/hadoop-2.7.3
SQOOP_HOME=/software/sqoop-1.4.6
PATH=$PATH:$JAVA_HOME/bin:$JAVA_HOME/lib:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$SQOOP_HOME/bin
export PATH JAVA_HOME HADOOP_HOME SQOOP_HOME

4、配置完环境后,执行如下语句,立即生效配置文件:

[hadoop@CloudDeskTop software]$ source /etc/profile

5、进入/software/sqoop-1.4.6/lib/目录,上传mysql-connector-java-5.1.43-bin.jar

这个地方的数据库驱动包必须选择该版本(5.1.43),因为Sqoop需要对接MySql数据库,如果选择的数据库驱动包不是这个版本,很容易出错。

6、配置sqoop

[hadoop@CloudDeskTop software]$ cd /software/sqoop-1.4.6/bin/

[hadoop@CloudDeskTop bin]$ vi configure-sqoop

注释掉如下代码:用这个符号“:<<COMMENT”作为起始符,“COMMENT”作为结束符;

127 :<<COMMENT
128 ## Moved to be a runtime check in sqoop.
129 if [ ! -d "${HBASE_HOME}" ]; then
130   echo "Warning: $HBASE_HOME does not exist! HBase imports will fail."
131   echo 'Please set $HBASE_HOME to the root of your HBase installation.'
132 fi
133 
134 ## Moved to be a runtime check in sqoop.
135 if [ ! -d "${HCAT_HOME}" ]; then
136   echo "Warning: $HCAT_HOME does not exist! HCatalog jobs will fail."
137   echo 'Please set $HCAT_HOME to the root of your HCatalog installation.'
138 fi
139 
140 if [ ! -d "${ACCUMULO_HOME}" ]; then
141   echo "Warning: $ACCUMULO_HOME does not exist! Accumulo imports will fail."
142   echo 'Please set $ACCUMULO_HOME to the root of your Accumulo installation.'
143 fi
144 if [ ! -d "${ZOOKEEPER_HOME}" ]; then
145   echo "Warning: $ZOOKEEPER_HOME does not exist! Accumulo imports will fail."
146   echo 'Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.'
147 fi
148 COMMENT
View Code

相关文章: