B2-CentOS7安装部署CDH6.1.0（未完成）

  1. 下载相关安装包

    Cloudera Manager 下载地址：

    https://archive.cloudera.com/cm6/6.1.0/redhat7/yum/RPMS/x86_64/cloudera-manager-agent-6.1.0-769885.el7.x86_64.rpm

    https://archive.cloudera.com/cm6/6.1.0/redhat7/yum/RPMS/x86_64/cloudera-manager-daemons-6.1.0-769885.el7.x86_64.rpm

    https://archive.cloudera.com/cm6/6.1.0/redhat7/yum/RPMS/x86_64/cloudera-manager-server-6.1.0-769885.el7.x86_64.rpm

    jdk 下载地址：

    https://archive.cloudera.com/cm6/6.1.0/redhat7/yum/RPMS/x86_64/oracle-j2sdk1.8-1.8.0+update141-1.x86_64.rpm

    CDH 安装包下载地址：https://archive.cloudera.com/cdh6/6.1.0/parcels/，需要下载以下文件：

    CDH-6.1.0-1.cdh6.1.0.p0.770702-el7.parcel

    manifest.json

  2. 安装三台 visualbox 虚拟机

    参考：https://blog.csdn.net/chirs_chen/article/details/84978941

    虚拟机管理软件：Oracle VM VisualBox

    虚拟机系统：CentOS-7-x86_64-Minimal-1810.iso

    IP地址 主机名
    192.168.56.2 cdh-2-master
    192.168.56.2 cdh-3-slave
    192.168.56.2 cdh-4-slave

    系统安装成功后，先修改网卡 /etc/sysconfig/network-scripts/ifcfg-enp0s3 的 ONBOOT 值为yes，重启 network 服务

    systemctl restart network

  3. 修改主机名及映射主机名（所有节点/重启）

    修改主机名：hostnamectl set-hostname

    设置映射主机名，将以下内容添加到 /etc/hosts 文件：

    192.168.56.2 cdh-2-master
192.168.56.3 cdh-3-slave
192.168.56.4 cdh-4-slave

  4. 关闭防火墙和 selinux（所有节点）

    查看防火墙：systemctl status firewalld.service

    执行关闭命令：systemctl stop firewalld.service

    执行开机禁用防火墙命令：systemctl disable firewalld.service

    关闭selinux：vi /etc/sysconfig/selinux 将 SELINUX=enforcing 改为 SELINUX=disabled

  5. 安装 NTP（所有节点）

    编辑 /etc/ntp.conf 文件添加 NTP 服务器

    server 0.cn.pool.ntp.org
server 1.cn.pool.ntp.org
server 2.cn.pool.ntp.org
server 3.cn.pool.ntp.org

    启动 NTP

    systemctl enable ntpd
systemctl start ntpd

    手工同步网络时间

    ntpdate -u 0.cn.pool.ntp.org

    同步系统时钟

    hwclock --systohc

  6. 配置 ssh 免密登录（所有节点）

    被登录方 操作

    生成 rsa 公私钥对

    ssh-****** -t rsa -P ""

    拷贝公钥到本地 authorized_keys

    cat .ssh/id_rsa.pub >> .ssh/authorized_keys

    修改 authorized_keys 权限

    chmod 600 .ssh/authorized_keys

    验证本地免密登录

    ssh localhost

    登录方 操作

    获取 slave2 的公钥

    ssh {user}@{host} cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

  7. 禁用透明大页面压缩（所有节点）

    打开编辑 /etc/rc.local 文件：

    vi /etc/rc.local

    添加以下内容：

    echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled

  8. 优化交换分区（所有节点）

    RHEL6

    编辑 /etc/sysctl.conf 并添加 vm.swappiness=0

    vi /etc/sysctl.conf

    RHEL7

    tuned 服务会动态调整系统参数，查找 tuned 中配置，直接将配置修改为 vm.swappiness=0

    cd /usr/lib/tuned/
grep "vm.swappiness" * -R

  9. 安装jdk（所有节点）

    yum -y install oracle-j2sdk1.8-1.8.0+update141-1.x86_64.rpm

    /etc/profile 末尾添加以下内容

    #set jdk environment
export JAVA_HOME=/usr/java/jdk1.8.0_141-cloudera
export CLASSPATH=.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib:$CLASSPATH
export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH

  10. 安装 mysql（master）

    wget http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpm
rpm -ivh mysql-community-release-el7-5.noarch.rpm
yum update
yum install mysql-server
systemctl start mysqld

  11. 配置 mysql（master）

    将旧的 InnoDB 日志文件 /var/lib/mysql/ib_logfile0 和 /var/lib/mysql/ib_logfile1 从 /var/lib/mysql/ 中移除

    rm -rf /var/lib/mysql/ib_logfile0 /var/lib/mysql/ib_logfile1

    更新 /etc/my.cnf 配置文件，要满足以下要求：

    • To prevent deadlocks, set the isolation level to READ-COMMITTED.

    • Configure the InnoDB engine. Cloudera Manager will not start if its tables are configured with the MyISAM engine. (Typically, tables revert to MyISAM if the InnoDB engine is misconfigured.) To check which engine your tables are using, run the following command from the MySQL shell:

      mysql> show table status;

    • The default settings in the MySQL installations in most distributions use conservative buffer sizes and memory usage. Cloudera Management Service roles need high write throughput because they might insert many records in the database. Cloudera recommends that you set the innodb_flush_method property to O_DIRECT.

    • Set the max_connections property according to the size of your cluster:

      • Fewer than 50 hosts - You can store more than one database (for example, both the Activity Monitor and Service Monitor) on the same host. If you do this, you should:
        • Put each database on its own storage volume.
        • Allow 100 maximum connections for each database and then add 50 extra connections. For example, for two databases, set the maximum connections to 250. If you store five databases on one host (the databases for Cloudera Manager Server, Activity Monitor, Reports Manager, Cloudera Navigator, and Hive metastore), set the maximum connections to 550.
      • More than 50 hosts - Do not store more than one database on the same host. Use a separate host for each database/host pair. The hosts do not need to be reserved exclusively for databases, but each database should be on a separate host.

    • Binary logging is not a requirement for Cloudera Manager installations. Binary logging provides benefits such as MySQL replication or point-in-time incremental recovery after database restore. Examples of this configuration follow. For more information, see The Binary Log.

    推荐配置如下：

    [mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
transaction-isolation = READ-COMMITTED
# Disabling symbolic-links is recommended to prevent assorted security risks;
# to do so, uncomment this line:
symbolic-links = 0

key_buffer_size = 32M
max_allowed_packet = 32M
thread_stack = 256K
thread_cache_size = 64
query_cache_limit = 8M
query_cache_size = 64M
query_cache_type = 1

max_connections = 550
#expire_logs_days = 10
#max_binlog_size = 100M

#log_bin should be on a disk with enough free space.
#Replace '/var/lib/mysql/mysql_binary_log' with an appropriate path for your
#system and chown the specified folder to the mysql user.
log_bin=/var/lib/mysql/mysql_binary_log

#In later versions of MySQL, if you enable the binary log and do not set
#a server_id, MySQL will not start. The server_id must be unique within
#the replicating group.
server_id=1

binlog_format = mixed

read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M

# InnoDB settings
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit  = 2
innodb_log_buffer_size = 64M
innodb_buffer_pool_size = 4G
innodb_thread_concurrency = 8
innodb_flush_method = O_DIRECT
innodb_log_file_size = 512M

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

sql_mode=STRICT_ALL_TABLES

  12. 启动 mysql（master）

    启动服务并设置开机启动：

    systemctl enable mysqld
systemctl start mysqld

    设置 root 密码和其他安全相关设置：

    /usr/bin/mysql_secure_installation

    [...]
Enter current password for root (enter for none):
OK, successfully used password, moving on...
[...]
Set root password? [Y/n] Y
New password:
Re-enter new password:
Remove anonymous users? [Y/n] Y
[...]
Disallow root login remotely? [Y/n] N
[...]
Remove test database and access to it [Y/n] Y
[...]
Reload privilege tables now? [Y/n] Y
All done!

  13. 安装 mysql JDBC 驱动（master）

    wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gz
tar zxvf mysql-connector-java-5.1.46.tar.gz
mkdir -p /usr/share/java/
cp mysql-connector-java-5.1.46/mysql-connector-java-5.1.46-bin.jar /usr/share/java/mysql-connector-java.jar

  14. 创建 mysql 数据库（master）

    root 用户登录

    mysql -u root -p

    为下表中每个服务创建 utf8 字符集的数据库：

    CREATE DATABASE <database> DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON <database>.* TO '<user>'@'%' IDENTIFIED BY '<password>';
    Service database user
    Cloudera Manager Server scm scm
    Activity Monitor amon amon
    Reports Manager rman rman
    Hue hue hue
    Hive Metastore Server metastore hive
    Sentry Server sentry sentry
    Cloudera Navigator Audit Server nav nav
    Cloudera Navigator Metadata Server navms navms
    Oozie oozie oozie

    代码如下：

    CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY 'scm';
CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON amon.* TO 'amon'@'%' IDENTIFIED BY 'amon';
CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON rman.* TO 'rman'@'%' IDENTIFIED BY 'rman';
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON hue.* TO 'hue'@'%' IDENTIFIED BY 'hue';
CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON metastore.* TO 'hive'@'%' IDENTIFIED BY 'hive';
CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON sentry.* TO 'sentry'@'%' IDENTIFIED BY 'sentry';
CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON nav.* TO 'nav'@'%' IDENTIFIED BY 'nav';
CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON navms.* TO 'navms'@'%' IDENTIFIED BY 'navms';
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON oozie.* TO 'oozie'@'%' IDENTIFIED BY 'oozie';
flush privileges;

  15. 安装 cloudera manager 包

    master 节点：

    yum -y install cloudera-manager-daemons-6.1.0-769885.el7.x86_64.rpm
yum -y install cloudera-manager-agent-6.1.0-769885.el7.x86_64.rpm
yum -y install cloudera-manager-server-6.1.0-769885.el7.x86_64.rpm

    slave 节点：

    yum -y install cloudera-manager-daemons-6.1.0-769885.el7.x86_64.rpm
yum -y install cloudera-manager-agent-6.1.0-769885.el7.x86_64.rpm

  16. 修改 agent 的服务器地址（所有节点）

    打开并编辑 /etc/cloudera-scm-agent/config.ini 文件：

    /etc/cloudera-scm-agent/config.ini

    将 server_host=localhost 改为 master 节点主机名 cdh-2-master

  17. 启动 agent 服务（所有节点）

    systemctl start cloudera-scm-agent

  18. 准备 Cloudera Manager 数据库（master）

    /opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm scm

    若已存在，则执行：

    rm /etc/cloudera-scm-server/db.mgmt.properties

  19. 准备 parcel 包（master）

    将已下载的 CDH-6.1.0-1.cdh6.1.0.p0.770702-el7.parcel 和 manifest.json 拷贝至 /opt/cloudera/parcel-repo/ 目录，并且生成 parcel 包的 sha 值：

    sha1sum /opt/cloudera/parcel-repo/CDH-6.1.0-1.cdh6.1.0.p0.770702-el7.parcel

    将 sha 值填入到 CDH-6.1.0-1.cdh6.1.0.p0.770702-el7.parcel.sha 文件中

    vi /opt/cloudera/parcel-repo/CDH-6.1.0-1.cdh6.1.0.p0.770702-el7.parcel.sha

  20. 启动 Cloudera Manager Server（master）

    启动：

    systemctl start cloudera-scm-server

    查看启动日志：

    tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log

    看到下面的信息，说明启动完成

    INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server.

  21. 启用 Auto-TLS 以自动创建证书（master）

    JAVA_HOME=/usr/java/jdk1.8.0_141-cloudera /opt/cloudera/cm-agent/bin/certmanager --location /opt/cloudera/CMCA setup --configure-services

  22. 登录 web 页面（忽略许可授权，选择免费版）

    账号/密码：admin / admin

    B2-CentOS7安装部署CDH6.1.0（未完成）

  23. 添加集群

    由于环境不满足（cpu、内存），在添加集群时网络经常断开，无法继续配置，内存也满了，有几个注意的要点：

    • 配置存储库时，不要选择自定义，使用默认的 https://archive.cloudera.com/cm6/
    • 安装jdk时，不要勾选，直接下一步，因为已经提前安装好了
    • Install Agents时，由于已经提前安装了agent，所以这一步等进度条走完

    挺遗憾的，没有完全部署完，只能等以后有环境了在补充。

相关文章：

猜你喜欢
相关资源
相似解决方案
热门标签
Java Python linux javascript Mysql C# Docker 算法 前端 SpringBoot Redis Vue spring 设计模式 .net core .net kubernetes c++ 数据库 数据结构 大数据 js 机器学习 微服务 Android Go 程序员 面试 JVM ASP.net core 云原生 人工智能 后端 PHP git CSS golang k8s Nginx Django mybatis 深度学习 多线程 React 架构 devops 爬虫 云计算 Spring Boot LeetCode