简介:
sqoop是一款用于hadoop和关系型数据库之间数据导入导出的工具。你可以通过sqoop把数据从数据库(比如mysql,oracle)导入到hdfs中;也可以把数据从hdfs中导出到关系型数据库中。通过将sqoop的操作命令转化为Hadoop的MapReduce作业进行导入导出,(通常只涉及到Map任务)即sqoop生成的Job主要是并发运行MapTask实现数据并行传输以提升数据传送速度和效率,如果使用Shell脚本来实现多线程数据传送则存在很大的难度Sqoop2(sqoop1.99.7)需要在Hadoop安装目录下的配置文件中设置代理,属于重量级嵌入安装,文中我们使用qoop1(Sqoop1.4.6)。
前提:(若不知道如何安装请看我前面写的hadoop分类的文章)
CloudDeskTop上安装了: hadoop-2.7.3 jdk1.7.0_79 mysql-5.5.32 sqoop-1.4.6 hive-1.2.2 master01和master02安装了: hadoop-2.7.3 jdk1.7.0_79 slave01、slave02、slave03安装了: hadoop-2.7.3 jdk1.7.0_79 zookeeper-3.4.10
一、安装:
1、上传安装包sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz到/install/目录下
2、解压:
[hadoop@CloudDeskTop install]$ tar -zxvf sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz -C /software/
3、配置环境:
[hadoop@CloudDeskTop software]$ su -lc "vi /etc/profile"
JAVA_HOME=/software/jdk1.7.0_79 HADOOP_HOME=/software/hadoop-2.7.3 SQOOP_HOME=/software/sqoop-1.4.6 PATH=$PATH:$JAVA_HOME/bin:$JAVA_HOME/lib:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$SQOOP_HOME/bin export PATH JAVA_HOME HADOOP_HOME SQOOP_HOME
4、配置完环境后,执行如下语句,立即生效配置文件:
[hadoop@CloudDeskTop software]$ source /etc/profile
5、进入/software/sqoop-1.4.6/lib/目录,上传mysql-connector-java-5.1.43-bin.jar包
这个地方的数据库驱动包必须选择该版本(5.1.43),因为Sqoop需要对接MySql数据库,如果选择的数据库驱动包不是这个版本,很容易出错。
6、配置sqoop
[hadoop@CloudDeskTop software]$ cd /software/sqoop-1.4.6/bin/
[hadoop@CloudDeskTop bin]$ vi configure-sqoop
注释掉如下代码:用这个符号“:<<COMMENT”作为起始符,“COMMENT”作为结束符;
127 :<<COMMENT 128 ## Moved to be a runtime check in sqoop. 129 if [ ! -d "${HBASE_HOME}" ]; then 130 echo "Warning: $HBASE_HOME does not exist! HBase imports will fail." 131 echo 'Please set $HBASE_HOME to the root of your HBase installation.' 132 fi 133 134 ## Moved to be a runtime check in sqoop. 135 if [ ! -d "${HCAT_HOME}" ]; then 136 echo "Warning: $HCAT_HOME does not exist! HCatalog jobs will fail." 137 echo 'Please set $HCAT_HOME to the root of your HCatalog installation.' 138 fi 139 140 if [ ! -d "${ACCUMULO_HOME}" ]; then 141 echo "Warning: $ACCUMULO_HOME does not exist! Accumulo imports will fail." 142 echo 'Please set $ACCUMULO_HOME to the root of your Accumulo installation.' 143 fi 144 if [ ! -d "${ZOOKEEPER_HOME}" ]; then 145 echo "Warning: $ZOOKEEPER_HOME does not exist! Accumulo imports will fail." 146 echo 'Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.' 147 fi 148 COMMENT