Mac安装Hadoop - 爱码网

软件版本

hadoop3.2.1

一、打开本地ssh登录

# 生成公钥默认
$ ssh-keygen -t rsa -C "robots_wang@163.com" -b 4096
# 拷贝
$ cat ~/.ssh/id_rsa.pub
# 拷贝至电脑信任列表
$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
测试：ssh localhost

二、安装Hadoop

【1】下载地址：

　　https://archive.apache.org/dist/hadoop/common/

　　解压：tar -zxvf hadoop-3.2.1.tar.gz -C /Users/mac/SoftWare

【2】配置环境变量

vim ~/.bash_profile

export JAVA_HOME="/Library/Java/JavaVirtualMachines/jdk1.8.0_201.jdk/Contents/Home"

export CLASSPAHT=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

export PATH="${JAVA_HOME}:${PATH}"

export HADOOP_HOME=/Users/mac/SoftWare/hadoop-3.2.1

export HADOOP_COMMON_HOME=$HADOOP_HOME

export HADOOP_OPTS="-Djava.library.path=${HADOOP_HOME}/lib/native"

export PATH=${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin:$PATH

source ~/.bash_profile

【3】修改配置文件(cd /Users/mac/SoftWare/hadoop-3.2.1/etc/hadoop)

hadoop-env.sh

修改JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_201.jdk/Contents/Home

core-site.xml

新加

<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>file:/Users/mac/SoftWare/Cache/hadoop/tmp</value>
</property>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value>
</property>

<property>
<name>hadoop.native.lib</name>
<value>false</value>
<description>Should native hadoop libraries, if present, be used.</description>
</property>

</configuration>

hdfs-site.xml

新加

<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/Users/mac/SoftWare/Cache/hadoop/tmp/dfs/name</value>
</property>
<property>
<name>dfs.namenode.data.dir</name>
<value>file:/Users/mac/SoftWare/Cache/hadoop/tmp/dfs/data</value>
</property>
</configuration>

mapred-site.xml

添加

<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>

<name>mapreduce.application.classpath</name>

<value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value>

</property>

</configuration>

yarn-site.xml

新增

<configuration>

<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.env-whitelist</name>
<value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
</property>
</configuration>

三、启动Hadoop

【1】初始化Hadoop仅一次

cd /Users/mac/SoftWare/hadoop-3.2.1/bin

hdfs namenode -format

【2】启动dfs

cd /Users/mac/SoftWare/hadoop-3.2.1/sbin

./start-dfs.sh

域名：http://localhost:9870

HDFS文件系统地址：http://localhost:9870/explorer.html#

Starting namenodes on [localhost]

Starting datanodes

Starting secondary namenodes [account.jetbrains.com]

account.jetbrains.com: Warning: Permanently added \'account.jetbrains.com,0.0.0.0\' (ECDSA) to the list of known hosts.

2021-07-29 11:42:41,550 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

【3】启动yarn

./start-yarn.sh

http://localhost:8088

参考文档

https://blog.csdn.net/pgs1004151212/article/details/104391391