大数据技术之Hadoop3.1.2版本HA模式
作者:尹正杰
版权声明:原创作品,谢绝转载!否则将追究法律责任。
一.Hadoop的HA特点
1>.主备NameNode
2>.解决单点故障(属性,位置)
1.主NameNode对外提供服务,备NameNode同步主NameNode元数据一带切换; 2.所有DataNode同时向两个NameNode汇报数据块信息(位置); 3.JNN:集群(属性); 4.Standby:备,完成了edits.log文件的合并产生新的image,推送回ANN;
3>.两种切换选择
1.手动切换 通过命令实现主备之间的切换,可以用HDFS升级等场合。 2.自动切换 基于Zookeeper实现。
4>.基于zookeeper自动切换方案
1.Zookeeper Failover Controller进程用于监控NameNode监控状态; 2.并向zookeeper注册NameNode; 3.NameNode挂掉后,ZKFC为NameNode竞争锁,获得ZKFC锁的NameNode变为active;
5>.配置Hadoop3.1.2的官方文档(http://hadoop.apache.org/docs/r3.1.2/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html)
二.zookeeper完全分布式集群搭建
1>.下载zookeeper软件(下载地址:https://www.apache.org/dyn/closer.cgi/zookeeper/)
[root@node101.yinzhengjie.org.cn ~]# wget https://mirrors.tuna.tsinghua.edu.cn/apache/zookeeper/zookeeper-3.4.14/zookeeper-3.4.14.tar.gz --2019-04-12 08:48:40-- https://mirrors.tuna.tsinghua.edu.cn/apache/zookeeper/zookeeper-3.4.14/zookeeper-3.4.14.tar.gz Resolving mirrors.tuna.tsinghua.edu.cn (mirrors.tuna.tsinghua.edu.cn)... 101.6.8.193, 2402:f000:1:408:8100::1 Connecting to mirrors.tuna.tsinghua.edu.cn (mirrors.tuna.tsinghua.edu.cn)|101.6.8.193|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 37676320 (36M) [application/x-gzip] Saving to: ‘zookeeper-3.4.14.tar.gz’ 100%[============================================================================================================================================>] 37,676,320 7.57MB/s in 4.5s 2019-04-12 08:48:45 (7.94 MB/s) - ‘zookeeper-3.4.14.tar.gz’ saved [37676320/37676320] [root@node101.yinzhengjie.org.cn ~]#