Hadoop-YARN 分布式安装

一、概述

hadoop-3.1.1.tar.gz
172.16.233.137(主节点：ResourseManager)
172.16.233.138(从节点：NodeManager)
172.16.233.139(从节点：NodeManager)

二、流程

Hadoop-YARN 分布式安装

Client提交作业到ResourseManager。
Resourse Manager随机挑选挑选一台空闲的机器创建App Mstr。
App Mstr 获取切片清单向Resourse Manager申请资源Container，Container由Node Manager来管理, 所以Node Manager在DataNode节点上，一对一的关系。
返回Container给App Mstr后，由App Mstr调度作业在Container执行。同时Container会返回作业的状态和进度等信息给App Mstr。

三、安装(三台机器一样)

1、配置hadoop-env.sh，添加

export YARN_RESOURCEMANAGER_USER=root
export YARN_NODEMANAGER_USER=root

2、配置mapred-site.xml，添加

<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
    <property>
        <name>yarn.app.mapreduce.am.env</name>
	<value>HADOOP_MAPRED_HOME=$HADOOP_HOME</value>
    </property>
    <property>
        <name>mapreduce.application.classpath</name>
        <value>$HADOOP_HOME/share/hadoop/mapreduce/*:$HADOOP_HOME/share/hadoop/mapreduce/lib/*</value>
    </property>
    # 提示Container killed on request. Exit code is 143
    # 'VIRTUAL' memory limit. Current usage: 35.1 MB of 1 GB physical memory used; 2.3 GB of 2.1 GB virtual memory used. Killing container.
    # 报错时：修改map和reduce的物理内存
    <property>
        <name>mapreduce.map.memory.mb</name>
        <value>200</value>
    </property>
    <property>
        <name>mapreduce.reduce.memory.mb</name>
        <value>200</value>
    </property>
</configuration>

3、配置yarn-site.xml，添加

<configuration>
    # ResourseManager的节点主机名
    <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>node1</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services</name>
	<value>mapreduce_shuffle</value>
    </property>
    # 白名单
    <property>
        <name>yarn.nodemanager.env-whitelist</name>
      <value>JAVA_HOME,HADOOP_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
    </property>
</configuration>

4、进入到主节点启动yarn：

[[email protected] hadoop-3.1.1]# ./sbin/start-yarn.sh
Starting resourcemanager
上一次登录：五 3月  8 23:25:33 EST 2019pts/0 上
Starting nodemanagers
上一次登录：五 3月  8 23:33:34 EST 2019pts/0 上
[[email protected] hadoop-3.1.1]#


# resourse节点
[[email protected] hadoop-3.1.1]# jps
9461 Jps
9143 ResourceManager
6046 NameNode
[[email protected] hadoop-3.1.1]#

# node节点
[[email protected] hadoop]# jps
4720 NodeManager
5058 Jps
2851 DataNode
2925 SecondaryNameNode
[[email protected] hadoop]#

# node节点
[[email protected] hadoop]# jps
2896 DataNode
4953 Jps
4476 NodeManager
[[email protected] hadoop]#

5、访问：node1:8088

Hadoop-YARN 分布式安装

三、测试

./bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.1.jar wordcount /data/wordcount.txt /data/test

提交的作业jar文件：adoop-mapreduce-examples-3.1.1.jar
提交的作业：wordcount，属于adoop-mapreduce-examples-3.1.1.jar里面的。
输入参数：/data/wordcount.txt
输出参数：/data/test，这个目录必须在hdfs上不存在。

Hadoop-YARN 分布式安装