HDFS文件系统搭建

HDFS文件系统

软件下载

hadoop-3.3.0.tar.gz

JDK环境

hadoop java编写，运行需依赖jvm

jdk-8u161-linux-x64.tar.gz

配置系统环境变量

#JAVA
export JAVA_HOME=/home/hadoop/jdk1.8.0_161
export CLASSPATH=$JAVA_HOME/libdt.jar:$JAVA_HOME/tools.jar
#hadoop
export HADOOP_HOME=/home/hadoop/hadoop-3.3.0
export PATH=$JAVA_HOME/bin:$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

hadoop配置

hadoop-3.3.0/etc/hadoop 的目录下

配置JAVA_HOME

hadoop-env.sh

export JAVA_HOME=/home/hadoop/jdk1.8.0_161

配置核心组件文件

core-site.xml需要在和之间添加

<property>
        <name>fs.defaultFS</name>
        <value>hdfs://172.16.176.103:9000</value>
</property>
<property>
        <name>hadoop.tmp.dir</name>
        <value>/data</value>
</property>

配置文件系统

hdfs-site.xml需要在和之间添加

<!--namenode-->
<property>
        <name>dfs.namenode.name.dir</name>
        <value>/data/namenode</value>
</property>
<!--datanode-->
<property>
        <name>dfs.datanode.data.dir</name>
        <value>/data/datanode</value>
</property>
<!--副本数,默认1-->
<property>
   		<name>dfs.replication</name>
        <value>1</value>
</property>
<!--禁用权限检查,配合es-->
<property>
  <name>dfs.permissions</name>
  <value>false</value>
</property>

配置mapred

mapred-site.xml

<property>
  <name>mapreduce.framework.name</name>
  <value>yarn</value>
</property>

配置 yarn-site.xml

yarn-site.xml

<property>
  <name>yarn.resourcemanager.hostname</name>
  <value>elasticsearch01</value>
</property>

格式化文件系统

hdfs namenode -format

启动hdfs

start-dfs.sh

$ start-dfs.sh 
WARNING: HADOOP_SECURE_DN_USER has been replaced by HDFS_DATANODE_SECURE_USER. Using value of HADOOP_SECURE_DN_USER.
Starting namenodes on [host103]
Starting datanodes
Starting secondary namenodes [host103]

访问

http://localhost:9870/