Flink1.7.2 HA模式安装(Standalone和Yarn)

Flink HA模式安装(Standalone和Yarn)

flink是一个大数据开发人员必须会的组件，现在无论大小公司都在搞，所以安装是第一步。

0.安装Java1.8的环境(不赘述)
1.下载安装包
下载官网：https://flink.apache.org/downloads.html
也可以直接下面点击下面链接下载:
http://mirrors.tuna.tsinghua.edu.cn/apache/flink/flink-1.7.2/flink-1.7.2-bin-scala_2.12.tgz

2.配置环境变量
sudo vi /etc/profile

#flink
export FLINK_HOME=/opt/modules/flink-1.7.2
export PATH=${FLINK_HOME}/bin:$PATH

安装Standalone 模式看第三步就够了，安装on yarn 模式只看第四步就够了

3.Standalone HA模式安装
a.vi ${FLINK_HOME}/conf/flink-conf.yaml

state.backend: filesystem
state.checkpoints.dir: hdfs://ns/flink-checkpoints
taskmanager.numberOfTaskSlots: 2
high-availability: zookeeper
high-availability.storageDir: hdfs://ns/flink/ha/
high-availability.zookeeper.quorum: cdh01.com:2181,cdh02.com:2181,cdh03.com:2181
high-availability.zookeeper.client.acl: open

b.vi ${FLINK_HOME}/conf/zoo.cfg(此步骤不需要)
server对应的1、2、3要和zookeeper中dataDir参数的myid的值保持一致
server.1=cdh01.com:2888:3888
server.2=cdh02.com:2888:3888
server.3=cdh03.com:2888:3888

c.vi ${FLINK_HOME}/conf/masters

cdh02.com:8081
cdh03.com:8081

d.vi ${FLINK_HOME}/conf/slaves

cdh01.com
cdh02.com
cdh03.com

e.将安装包分发

scp -r flink-1.7.2/ [email protected]:/opt/modules/
scp -r flink-1.7.2/ [email protected]:/opt/modules/

f.启动命令start-cluster.sh，如果显示如下说明成功(特别是第一行 Starting HA cluster)

[[email protected] ~]# start-cluster.sh
Starting HA cluster with 2 masters.
Starting standalonesession daemon on host cdh02.com.
Starting standalonesession daemon on host cdh03.com.
Starting taskexecutor daemon on host cdh01.com.
Starting taskexecutor daemon on host cdh02.com.
Starting taskexecutor daemon on host cdh03.com.

g.测试命令

flink run ${FLINK_HOME}/examples/batch/WordCount.jar

安装Standalone 模式看第三步就够了，安装on yarn 模式只看第四步就够了

4.Yarn HA模式安装
a.添加环境变量sudo vi /etc/profile(为了能读取到hadoop的yarn-site.xml等文件)

export HADOOP_CONF_DIR=/etc/hadoop/conf

b.flink on yarn 模式有两种提交任务方式
第一种，直接通过 -m yarn-cluster提交到yarn上运行，测试命令如下

flink run -m yarn-cluster -yn 2 ${FLINK_HOME}/examples/batch/WordCount.jar

第二种，先通过yarn-session开辟一块资源，再指定jobmanager提交任务
开辟资源命令

yarn-session.sh -d -s 2 -tm 800 -n 3

再提交

flink run -m ml20.com:34562 ${FLINK_HOME}/examples/batch/WordCount.jar

Flink1.7.2 HA模式安装(Standalone和Yarn)

线上推荐使用第一种方式

Flink1.7.2 HA模式安装(Standalone和Yarn)
举例

flink run -d \
   -c com.FlinkCheckpointTest \
   -s hdfs://ns/flink/flink-checkpoints/***/chk-13 \
   -m yarn-cluster \
   -yjm 4096m -ytm 4096m -ynm FlinkCheckpointTest \
   /home/jar/flink_test.jar