hadoop2.6.5安装(一)应该算安装前期的踩坑记录,该文是针对完全分布式的基本安装配置。
系统:CentOS 7.2
软件:
jdk 8 (链接:http://www.oracle.com/technetwork/cn/java/javase/downloads/jdk8-downloads-2133151-zhs.html)
hadoop 2.6.5 (链接:http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.6.5/)
前期理论:
Hadoop的核心模块就是HDFS (hadoop分布式文件系统,能提供高吞吐量的数据访问,适用于超大数据集的应用) 以及MapReduce(支持java语言的一套从海量数据中提取分析元素返回结果集的编程模型);hadoop旗下的子项目例如:HBase、Hive 等都是基于前面两者发展实现的。
一、准备工作
1.至少3台机器,分别部署1个主节点和2个从节点(三个节点最好都是 同一网段的内网地址,因为我使用的是不同网段的外网地址,后续踩坑无数)
HDmaster 118.25.50.236 HDslave1 118.25.8.59 HDslave2 111.231.92.235
2.首先在3台机器的/etc/hosts 分别配置好以下信息
[root@HDmaster ~]# vim /etc/hosts #添加好如下域名,方便后续解析 118.25.50.236 HDmaster 118.25.8.59 HDslave1 111.231.92.235 HDslave2 #然后可以选择scp到另外的机器或者直接复制粘贴也ok
设置好之后测试一下,域名解析是否ok,排除防火墙、网关的网络影响
1 [root@HDmaster ~]# ping HDmaster -c2 2 PING HDmaster (118.25.50.236) 56(84) bytes of data. 3 64 bytes from HDmaster (118.25.50.236): icmp_seq=1 ttl=63 time=0.336 ms 4 64 bytes from HDmaster (118.25.50.236): icmp_seq=2 ttl=63 time=0.360 ms 5 --- HDmaster ping statistics --- 6 2 packets transmitted, 2 received, 0% packet loss, time 1000ms 7 rtt min/avg/max/mdev = 0.336/0.348/0.360/0.012 ms 8 [root@HDmaster ~]# ping HDslave1 -c2 9 PING HDslave1 (118.25.8.59) 56(84) bytes of data. 10 64 bytes from HDslave1 (118.25.8.59): icmp_seq=1 ttl=61 time=0.338 ms 11 64 bytes from HDslave1 (118.25.8.59): icmp_seq=2 ttl=61 time=0.403 ms 12 --- HDslave1 ping statistics --- 13 2 packets transmitted, 2 received, 0% packet loss, time 999ms 14 rtt min/avg/max/mdev = 0.338/0.370/0.403/0.037 ms 15 [root@HDmaster ~]# ping HDslave2 -c2 16 PING HDslave2 (111.231.92.235) 56(84) bytes of data. 17 64 bytes from HDslave2 (111.231.92.235): icmp_seq=1 ttl=63 time=0.371 ms 18 64 bytes from HDslave2 (111.231.92.235): icmp_seq=2 ttl=63 time=0.393 ms 19 --- HDslave2 ping statistics --- 20 2 packets transmitted, 2 received, 0% packet loss, time 1000ms 21 rtt min/avg/max/mdev = 0.371/0.382/0.393/0.011 ms
3.配置三台机器之间的ssh免密登录