openstack ha 部署

一、控制节点架构如下图：

二、初始化环境：

1、配置IP地址：

1、节点1：
ip addr add dev eth0  192.168.142.110/24
echo 'ip addr add dev eth0 192.168.142.110/24' >> /etc/rc.local 
chmod +x /etc/rc.d/rc.local 
2、节点2：
ip addr add dev eth0  192.168.142.111/24
echo 'ip addr add dev eth0 192.168.142.111/24' >> /etc/rc.local 
chmod +x /etc/rc.d/rc.local 
3、节点3：
ip addr add dev eth0  192.168.142.112/24
echo 'ip addr add dev eth0 192.168.142.112/24' >> /etc/rc.local 
chmod +x /etc/rc.d/rc.local 

2、更改主机名：

配置主机名+修改/etc/hosts文件：
1、节点1
hostnamectl --static --transient  set-hostname  controller1
hosts文件：
192.168.142.110 controller1
192.168.142.111 controller2
192.168.142.112 controller3
2、节点2：
hostnamectl --static --transient  set-hostname controller2
hosts文件：
192.168.142.110 controller1
192.168.142.111 controller2
192.168.142.112 controller3
3、节点3：
hostnamectl --static --transient  set-hostname controller3
hosts文件：
192.168.142.110 controller1
192.168.142.111 controller2
192.168.142.112 controller3

3、设置防火墙及selinux：

systemctl disable firewalld
systemctl stop firewalld
sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config 

4、设置时间同步：

yum install ntp -y
ntpdate cn.pool.ntp.org

5、安装基础软件包：

yum install -y centos-release-openstack-ocata
yum upgrade -y 
yum install -y python-openstackclient

 三、安装基础基础服务：

1、安装Pacemaker

（1~4在三个节点都执行）

1、配置免密码登录：
节点1:
ssh-keygen -t rsa
ssh-copy-id root@controller2
ssh-copy-id root@controller3
节点2:
ssh-keygen -t rsa
ssh-copy-id root@controller1
ssh-copy-id root@controller3
节点3:
ssh-keygen -t rsa
ssh-copy-id root@controller1
ssh-copy-id root@controller2

2、安装pacemaker
yum install -y pcs pacemaker corosync fence-agents-all resource-agents

3、启动pcsd服务（开机自启动）
systemctl start pcsd.service
systemctl enable pcsd.service

4、创建集群用户：
echo 'password' |passwd --stdin hacluster  （此用户在安装pcs时候会自动创建）

5、集群各节点之间进行认证：
pcs cluster auth controller1 controller2 controller3 -u hacluster -p password （此处需要输入的用户名必须为pcs自动创建的hacluster，其他用户不能添加成功）

6、创建并启动名为openstack-ha的集群：
pcs cluster setup --start --name openstack-ha controller1 controller2 controller3

6、设置集群自启动： 
pcs cluster enable --all 

7、查看并设置集群属性： 
查看当前集群状态：
pcs cluster status 
检验Corosync的安装及当前corosync状态： 
corosync-cfgtool -s corosync-cmapctl | grep members pcs status corosync 
检查配置是否正确（假若没有输出任何则配置正确）： 
crm_verify -L -V 
禁用STONITH： 
pcs property set stonith-enabled=false 
无法仲裁时候，选择忽略： 
pcs property set no-quorum-policy=ignore

 2、Haproxy安装配置：

（1~3在三个节点都执行）
1、安装haproxy：
yum install -y haproxy lrzsz
2、初始化环境：
echo "net.ipv4.ip_nonlocal_bind=1" > /etc/sysctl.d/haproxy.conf
sysctl -p 
echo 1 > /proc/sys/net/ipv4/ip_nonlocal_bind
cat >/etc/sysctl.d/tcp_keepalive.conf << EOF
net.ipv4.tcp_keepalive_intvl = 1
net.ipv4.tcp_keepalive_probes = 5
net.ipv4.tcp_keepalive_time = 5
EOF

sysctl net.ipv4.tcp_keepalive_intvl=1
sysctl net.ipv4.tcp_keepalive_probes=5
sysctl net.ipv4.tcp_keepalive_time=5
mv /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.bak  &&cd /etc/haproxy/
（上传haproxy.cfg文件）

haproxy的配置文件：

global
    daemon
    group    haproxy                         
    maxconn  4000
    pidfile  /var/run/haproxy.pid           
    user     haproxy  
    stats    socket /var/lib/haproxy/stats   
    log      192.168.142.110 local0
defaults
    mode tcp
    maxconn 10000
    timeout  connect 10s
    timeout  client 1m
    timeout  server 1m
    timeout  check 10s
listen stats                   
    mode          http
    bind          192.168.142.110:8080                       
    stats         enable                     
    stats         hide-version                
    stats uri     /haproxy?openstack          
    stats realm   Haproxy\Statistics           
    stats admin if TRUE 
    stats auth    admin:admin 
    stats refresh 10s
frontend vip-db
    bind 192.168.142.201:3306
    timeout client 90m
    default_backend db-vms-galera

frontend vip-qpid
    bind 192.168.142.215:5672
    timeout client 120s
    default_backend qpid-vms
frontend vip-horizon
    bind 192.168.142.211:80
    timeout client 180s
    cookie SERVERID insert indirect nocache
    default_backend horizon-vms
frontend vip-ceilometer
    bind 192.168.142.214:8777
    timeout client 90s
    default_backend ceilometer-vms
frontend vip-rabbitmq
    option clitcpka
    bind 192.168.142.202:5672
    timeout client 900m
    default_backend rabbitmq-vms
frontend vip-keystone-admin
    bind 192.168.142.203:35357
    default_backend keystone-admin-vms
backend keystone-admin-vms
    balance roundrobin
    server controller1-vm 192.168.142.110:35357 check inter 1s
    server controller2-vm 192.168.142.111:35357 check inter 1s
    server controller3-vm 192.168.142.112:35357 check inter 1s
frontend vip-keystone-public
    bind 192.168.142.203:5000
    default_backend keystone-public-vms
backend keystone-public-vms
    balance roundrobin
    server controller1-vm 192.168.142.110:5000 check inter 1s
    server controller2-vm 192.168.142.111:5000 check inter 1s
    server controller3-vm 192.168.142.112:5000 check inter 1s
frontend vip-glance-api
    bind 192.168.142.205:9191
    default_backend glance-api-vms
backend glance-api-vms
    balance roundrobin
    server controller1-vm 192.168.142.110:9191 check inter 1s
    server controller2-vm 192.168.142.111:9191 check inter 1s
    server controller3-vm 192.168.142.112:9191 check inter 1s
frontend vip-glance-registry
    bind 192.168.142.205:9292
    default_backend glance-registry-vms
backend glance-registry-vms
    balance roundrobin
    server controller1-vm 192.168.142.110:9292 check inter 1s
    server controller2-vm 192.168.142.111:9292 check inter 1s
    server controller3-vm 192.168.142.112:9292 check inter 1s
frontend vip-cinder
    bind 192.168.142.206:8776
    default_backend cinder-vms
backend cinder-vms
    balance roundrobin
    server controller1-vm 192.168.142.110:8776 check inter 1s
    server controller2-vm 192.168.142.111:8776 check inter 1s
    server controller3-vm 192.168.142.112:8776 check inter 1s
frontend vip-swift
    bind 192.168.142.208:8080
    default_backend swift-vms
backend swift-vms
    balance roundrobin
    server controller1-vm 192.168.142.110:8080 check inter 1s
    server controller2-vm 192.168.142.111:8080 check inter 1s
    server controller3-vm 192.168.142.112:8080 check inter 1s
frontend vip-neutron
    bind 192.168.142.209:9696
    default_backend neutron-vms
backend neutron-vms
    balance roundrobin
    server controller1-vm 192.168.142.110:9696 check inter 1s
    server controller2-vm 192.168.142.111:9696 check inter 1s
    server controller3-vm 192.168.142.112:9696 check inter 1s
frontend vip-nova-vnc-novncproxy
    bind 192.168.142.210:6080
    default_backend nova-vnc-novncproxy-vms
backend nova-vnc-novncproxy-vms
    balance roundrobin
    server controller1-vm 192.168.142.110:6080 check inter 1s
    server controller2-vm 192.168.142.111:6080 check inter 1s
    server controller3-vm 192.168.142.112:6080 check inter 1s
frontend vip-nova-vnc-xvpvncproxy
    bind 192.168.142.210:6081
    default_backend nova-vnc-xvpvncproxy-vms
backend nova-vnc-xvpvncproxy-vms
    balance roundrobin
    server controller1-vm 192.168.142.110:6081 check inter 1s
    server controller2-vm 192.168.142.111:6081 check inter 1s
    server controller3-vm 192.168.142.112:6081 check inter 1s
frontend vip-nova-metadata
    bind 192.168.142.210:8775
    default_backend nova-metadata-vms
backend nova-metadata-vms
    balance roundrobin
    server controller1-vm 192.168.142.110:8775 check inter 1s
    server controller2-vm 192.168.142.111:8775 check inter 1s
    server controller3-vm 192.168.142.112:8775 check inter 1s
frontend vip-nova-api
    bind 192.168.142.210:8774
    default_backend nova-api-vms
backend nova-api-vms
    balance roundrobin
    server controller1-vm 192.168.142.110:8774 check inter 1s
    server controller2-vm 192.168.142.111:8774 check inter 1s
    server controller3-vm 192.168.142.112:8774 check inter 1s
backend horizon-vms
    balance roundrobin
    timeout server 108s
    server controller1-vm 192.168.142.110:80 check inter 1s
    server controller2-vm 192.168.142.111:80 check inter 1s
    server controller3-vm 192.168.142.112:80 check inter 1s
frontend vip-heat-cfn
    bind 192.168.142.212:8000
    default_backend heat-cfn-vms
backend heat-cfn-vms
    balance roundrobin
    server controller1-vm 192.168.142.110:8000 check inter 1s
    server controller2-vm 192.168.142.111:8000 check inter 1s
    server controller3-vm 192.168.142.112:8000 check inter 1s
frontend vip-heat-cloudw
    bind 192.168.142.212:8004
    default_backend heat-cloudw-vms
backend heat-cloudw-vms
    balance roundrobin
    server controller1-vm 192.168.142.110:8004 check inter 1s
    server controller2-vm 192.168.142.111:8004 check inter 1s
    server controller3-vm 192.168.142.112:8004 check inter 1s
frontend vip-heat-srv
    bind 192.168.142.212:8004
    default_backend heat-srv-vms
backend heat-srv-vms
    balance roundrobin
    server controller1-vm 192.168.142.110:8004 check inter 1s
    server controller2-vm 192.168.142.111:8004 check inter 1s
    server controller3-vm 192.168.142.112:8004 check inter 1s
backend ceilometer-vms
    balance roundrobin
    server controller1-vm 192.168.142.110:8777 check inter 1s
    server controller2-vm 192.168.142.111:8777 check inter 1s
    server controller3-vm 192.168.142.112:8777 check inter 1s
backend qpid-vms
    stick-table type ip size 2
    stick on dst
    timeout server 120s
    server controller1-vm 192.168.142.110:5672 check inter 1s
    server controller2-vm 192.168.142.111:5672 check inter 1s
    server controller3-vm 192.168.142.112:5672 check inter 1s
backend db-vms-galera
    option httpchk
    option tcpka
    stick-table type ip size 1000
    stick on dst
    timeout server 90m
    server controller1-vm 192.168.142.110:3306 check inter 1s port 9200 backup on-marked-down shutdown-sessions
    server controller2-vm 192.168.142.111:3306 check inter 1s port 9200 backup on-marked-down shutdown-sessions
    server controller3-vm 192.168.142.112:3306 check inter 1s port 9200 backup on-marked-down shutdown-sessions
backend rabbitmq-vms
    option srvtcpka
    balance roundrobin
    timeout server 900m
    server controller1-vm 192.168.142.110:5672 check inter 1s
    server controller2-vm 192.168.142.111:5672 check inter 1s
    server controller3-vm 192.168.142.112:5672 check inter 1s

haproxy.cfg