小Q大数据-hadoop环境搭建

系统环境Centos 7

一、服务器环境配置

1、服务器列表

名称	地址	操作系统	root密码
master	x.x.x.1	Centos 7	root
slave1	x.x.x.2	Centos 7	root
slave2	x.x.x.3	Centos 7	root

2、配置ssh信任

    1）在主节点生成公私秘钥对：

         进入/.ssh/目录下，

            ssh-****** -t rsa -P ''

           cat .ssh/*.pub > .ssh/authorized_keys

          chmod 600 ~/.ssh/authorized_keys

         scp -r .ssh root@x.x.x.2:/root/

         scp -r .sshroot@x.x.x.3:/root/

3、更改hosts

    在master 机器上执行如下命令:

    vi  /etc/hosts

    x.x.x.1  master.bigdata.com master

    x.x.x.2  slave1.bigdata.com slave1

    x.x.x.3  slave2.bigdata.com slave2

    

    执行以下命令：

    scp /etc/hosts root@slave1:/etc/hosts

    scp /etc/hostsroot@slave2:/etc/hosts

4、配置防火墙

    iptables –F

    systemctldisable firewalld

    servicefirewalld stop

    setenforce 0

5、配置阿里yum源（可选）

    mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.bak

    wget -O /etc/yum.repos.d/CentOS-Base.repohttp://mirrors.aliyun.com/repo/Centos-7.repo

    yum makecache

6、安装mysql数据库   

    在mysql服务安装在了master节点，也可选择其他节点或机器。具体步骤如下：          

wget http://repo.mysql.com/mysql57-community-release-el7-8.noarch.rpm rpm -ivh mysql57-community-release-el7-8.noarch.rpm yum install mysql-server [root@master ~]# service mysqld  start [root@master ~]# grep "password" /var/log/mysqld.log [root@master ~]# mysql -u root -p Enter password: mysql> set password = password("BigData123!@#"); mysql> grant all privileges on *.* to root@"%" identified by "BigData123!@#" ; mysql> flush privileges;

7、时钟同步

yum install -y ntp systemctl enable ntpd

8、关闭SELinux、配置umask值

setenforce 0 echo umask 0022 >> /etc/profile

9、安装mysql-java驱动

    yum install mysql-connector-java.noarch

二、配置本地yum库

1、服务安装配置

yum install yum-utils createrepo yum install httpd service httpd start

2、下载镜像包，解压到指定目录

操作系统	HDP/Ambari 版本	库名称	下载链接
CentOS 7	2.6.4.0	HDP	http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.6.4.0/HDP-2.6.4.0-centos7-rpm.tar.gz
		HDP-UTILS	http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.22/repos/centos7/HDP-UTILS-1.1.0.22-centos7.tar.gz
		HDP-GPL	http://public-repo-1.hortonworks.com/HDP-GPL/centos7/2.x/updates/2.6.4.0/HDP-GPL-2.6.4.0-centos7-rpm.tar.gz
CentOS 7	2.6.1.5	Ambari	http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.6.1.5/ambari-2.6.1.5-centos7.tar.gz

    下载后把镜像包解压缩，并且移动到/var/www/html/hdp 目录下；

tar -zxvf ambari-2.6.1.5-centos7.tar.gz

    完成后目录结构如下：

三、安装配置ambari Server

1、配置ambari 的yum源（在master节点）

wget  -nv  http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.6.1.5/ ambari.repo -O /etc/yum.repos.d/ambari.repo

2、修改ambari.repo配置信息

3、执行ambari server安装

yum install ambari-server

4、配置ambari server

[root@master yum.repos.d]# ambari-server setup Checking SELinux... SELinux status is 'enabled' SELinux mode is 'permissive' WARNING: SELinux is set to 'permissive' mode and temporarily disabled. OK to continue [y/n] (y)? y Customize user account for ambari-server daemon [y/n] (n)? Adjusting ambari-server permissions and ownership... Checking firewall status... Checking JDK... [1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8 [2] Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7 [3] Custom JDK ============================================================================== Enter choice (1): 1 ………… Successfully downloaded JDK distribution to /var/lib/ambari-server/resources/jdk-8u112-linux-x64.tar.gz Installing JDK to /usr/jdk64/ Successfully installed JDK to /usr/jdk64/ Successfully downloaded JCE Policy archive to /var/lib/ambari-server/resources/jce_policy-8.zip Installing JCE policy... Checking GPL software agreement... GPL License for LZO: https://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html Enable Ambari Server to download and install GPL Licensed LZO packages [y/n] (n)? Completing setup... Configuring database... Enter advanced database configuration [y/n] (n)? y Configuring database... ============================================================================== Choose one of the following options: [1] - PostgreSQL (Embedded) [2] - Oracle [3] - MySQL / MariaDB [4] - PostgreSQL [5] - Microsoft SQL Server (Tech Preview) [6] - SQL Anywhere [7] - BDB ============================================================================== Enter choice (1): 3 Hostname (localhost): master.bigdata.com Port (3306): Database name (ambari): Username (ambari): Enter Database Password (bigdata): BigData123_ Configuring ambari database... Configuring remote database connection properties... WARNING: Before starting Ambari Server, you must run the following DDL against the database to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql (此处转下表的数据库操作) Proceed with configuring remote database connection properties [y/n] (y)? y Extracting system views... ambari-admin-2.6.1.5.3.jar Ambari Server 'setup' completed successfully.

5、数据库操作

[root@master ~]# mysql -u root -p Enter password: mysql> create database ambari; mysql> grant all privileges on ambari.* to ambari@"%" identified by "BigData123_"; mysql> use ambari; Database changed mysql> source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql Query OK, 0 rows affected (0.00 sec)

6、服务启停

    启动：ambari-server start

    停止：ambari-server stop

    启动成功后可以通过浏览器访问管理界面。

    URL: x.x.x.1:8080

    用户名：admin

    密码：admin

四、HDP部署

1、登录ambari server管理界面

    在浏览器打开http://x.x.x.1:8080

    输入用户名密码  默认admin/admin

2、进入集群安装向导

3、输入集群名称

4、自定义hdp版本（可选）

在HDP/centos7/2.6.4.0-91/HDP-2.6.4.0-91.xml文件基础上，修改baseurl。

选择增加版本、上传修改后的版本文件。

增加版本后，直接下一步

5、选择版本

6、确认主机列表（并上传id_rsa文件）

7、主机注册、主机环境检查

8、选择安装部署组件

9、分配服务器角色

10、设置slave节点和客户端

11、配置hive组件

选择已有的mysql数据库，填入相关的信息（用户名、密码、数据库地址）

数据库操作

[root@master ~]# mysql -u root -p Enter password: mysql> create database hive; mysql> grant all privileges on hive.* to hive@"%" identified by "BigData123_"; mysql> flush privileges;

Ambari-server设置

[root@master bin]# rpm -qa |grep mysql-connector mysql-connector-java-5.1.25-3.el7.noarch [root@master bin]# rpm -ql mysql-connector-java-5.1.25-3.el7.noarch …… /usr/share/java/mysql-connector-java.jar …… [root@master java]# ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar Using python  /usr/bin/python Setup ambari-server Copying /usr/share/java/mysql-connector-java.jar to /var/lib/ambari-server/resources If you are updating existing jdbc driver jar for mysql with mysql-connector-java.jar. Please remove the old driver jar, from all hosts. Restarting services that need the driver, will automatically copy the new jar to the hosts. JDBC driver was successfully initialized. Ambari Server 'setup' completed successfully.

测试数据库连接

12、设置Ambari Metrics的admin密码

13、设置smartsence的admin密码

14、确认安装信息，点击下一步

15、执行安装

16、安装成功，点击完成进入图形界面