1、概念
集群部署完了，但是有一个很大的问题，那就是Master节点存在单点故障，要解决此问题，就要借助zookeeper，并且启动至少两个Master节点来实现高可靠，配置方式比较简单
2、集群规划
| node1 | node2 | node3 |
|---|---|---|
| zookeeper | zookeeper | zookeeper |
| master | master | |
| worker | worker |
3、停止spark所有服务，修改配置文件spark-env.sh
[[email protected] spark]$ vim conf/spark-env.sh
export SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url=node1,node2,node3 -Dspark.deploy.zookeeper.dir=/spark"
# SPARK_MASTER_HOST=node1
SPARK_MASTER_PORT=7077
尖叫提示：SPARK_MASTER_HOST 注释掉
4、将配置好的Spark文件同步到其他节点上
[[email protected] spark]$ rsync -rvl conf/ [email protected]:/opt/module/conf/
[[email protected] spark]$ rsync -rvl conf/ [email protected]:/opt/module/conf/
5、启动Spark HA
启动zookeeper
[[email protected] zookeeper-3.4.10]$ bin/zkServer.sh start
[[email protected] zookeeper-3.4.10]$ bin/zkServer.sh start
[[email protected] zookeeper-3.4.10]$ bin/zkServer.sh start
启动HDFS
[[email protected] hadoop-2.8.2]$ sbin/start-dfs.sh
尖叫提示：如果配置了Job History Server，需要在Job History Server之前启动HDFS
启动spark
[[email protected] spark]$ sbin/start-all.sh
启动第二个master
[[email protected] spark]$ sbin/start-master.sh
尖叫提示：程序中spark集群的访问地址需要改成 spark://master01:port1,master02:port2
[[email protected] spark]$ bin/spark-shell --master spark://node1:7077,node2:7077
启动Job History Server
[[email protected] spark]$ sbin/start-history-server.sh
尖叫提示：如果没有配置，可以不用启动
6、zookeeper 中保存了什么呢？
[[email protected] zookeeper-3.4.10]$ bin/zkCli.sh -server node1:2181,node2:2181,node3:2181
[zk: node1:2181,node2:2181,node3:2181(CONNECTED) 0] ls /spark
[leader_election, master_status]
[zk: node1:2181,node2:2181,node3:2181(CONNECTED) 1] get /spark/
leader_election master_status
[zk: node1:2181,node2:2181,node3:2181(CONNECTED) 1] get /spark/master_status
192.168.2.102
cZxid = 0x1d00000004
ctime = Tue Apr 03 17:18:14 CST 201*
mZxid = 0x1d00000004
mtime = Tue Apr 03 17:18:14 CST 201*
pZxid = 0x1d0000000a
cversion = 2
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 13
numChildren = 2
7、执行一个spark程序（HA）
[[email protected] spark]$ bin/spark-submit \
--class org.apache.spark.examples.SparkPi \
--master spark://node1:7077, node2:7077 \
--executor-memory 1G \
--total-executor-cores 2 \
/opt/module/spark/examples/jars/spark-examples_2.11-2.1.1.jar \
100