准备工作
数据库架构
角色 ip地址 主机名 server_id Master 192.168.132.131 master 131 Slave1 192.168.132.132 master-bak 132 Slave2 192.168.132.133 slave2 133
配置三台服务器ssh免秘钥认证
ssh-keygen -t rsa ssh-copy-id 192.168.132.132 ssh-copy-id 192.168.132.133 ssh-copy-id 192.168.132.131
mysql主从搭建
一、修改配置文件 my.cnf
master
[root@master ~]# cat /etc/my.cnf|grep -v "^#\|^$" [mysqld] server_id=131 log_bin = master-bin log_bin_index = master-bin.index enforce_gtid_consistency = 1 log_slave_updates = 1 plugin_load = "rpl_semi_sync_master=semisync_master.so;rpl_semi_sync_slave=semisync_slave.so" loose_rpl_semi_sync_master_enabled = 1 loose_rpl_semi_sync_slave_enabled = 1 loose_rpl_semi_sync_master_timeout = 5000 sql_mode=NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES
master-bak
[root@master-bak ~]# cat /etc/my.cnf|grep -v "^#\|^$" [mysqld] server_id=132 log_bin=slave-bin log_bin_index=slave-bin.index log-slave-updates relay_log_index=slave-relay-bin.index relay_log=slave-relay-bin enforce_gtid_consistency = 1 log_slave_updates = 1 plugin_load = "rpl_semi_sync_master=semisync_master.so;rpl_semi_sync_slave=semisync_slave.so" loose_rpl_semi_sync_master_enabled = 1 loose_rpl_semi_sync_slave_enabled = 1 loose_rpl_semi_sync_master_timeout = 5000 sql_mode=NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES
slave2
[root@slave2 ~]# cat /etc/my.cnf|grep -v "^#\|^$" [mysqld] server_id=133 log-bin=mysql-bin relay_log_index=slave-relay-bin.index relay_log=slave-relay-bin relay_log_purge=0 read_only=1 enforce_gtid_consistency = 1 log_slave_updates = 1 plugin_load = "rpl_semi_sync_master=semisync_master.so;rpl_semi_sync_slave=semisync_slave.so" loose_rpl_semi_sync_master_enabled = 1 loose_rpl_semi_sync_slave_enabled = 1 loose_rpl_semi_sync_master_timeout = 5000 sql_mode=NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES
二、配置主从
授权(在三台服务器上配置复制用户和监控用户，三台服务器都要添加)
mysql5.7会默认加载validate_password 模块，是来控制密码长度和规则的，可以在配置文件里面关闭该模块 加上validate_password = off ，或者在mysql命令行执行set global validate_password_policy=0;来临时取消密码规则。
#添加复制用户 grant replication slave on *.* to 'repl'@'192.168.132.%' identified by '123456'; #添加监控用户 grant all privileges on *.* to 'root'@'192.168.132.%' identified by 'root';
#做完这两步，可测试三个mysql是否都可以远程登录了
master执行(记住file和position)
mysql> show master status; +-------------------+----------+--------------+------------------+-------------------+ | File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set | +-------------------+----------+--------------+------------------+-------------------+ | master-bin.000003 | 1455 | | | | +-------------------+----------+--------------+------------------+-------------------+ 1 row in set (0.00 sec)
master-bak和slave执行
change master to master_host='192.168.132.131',master_user='repl',master_password='repl',master_log_file='mysql-master.000003',master_log_pos=1455;
#设置从服务器只读，不要在配置文件里写 set global read_only=1
三、查看结果
#两个节点执行如下命令
[root@master-bak ~]# mysql -uroot -proot -e "show slave status \G"|grep "Slave_IO_Running\|Slave_SQL_Running" mysql: [Warning] Using a password on the command line interface can be insecure. Slave_IO_Running: Yes Slave_SQL_Running: Yes Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates [root@master-bak ~]#
四、遇到的问题
当遇到io_running为no的时候：
mysql> stop slave； mysql> set GLOBAL SQL_SLAVE_SKIP_COUNTER=1; #前提是配置文件里#gtid_mode = on注释掉
mysql> start slave;
#如果io_running还为no，可尝试从新执行 change master 。。。
##方法二 （重置slave）
mysql> stop slave;
mysql> reset slave;
mysql> start slave;
#方法三 （重设同步日志文件及位置）
mysql> show slave status \G; #查看目前同步到mysql-bin 日志文件的位置
mysql> stop slave;
mysql> change master to master_log_file='上面查到的位置position'，master_log_pos=1;
mysql> start slave;
当遇到 Slave_SQL_Running: No的时候：
- 程序可能在slave上进行了写操作
- 也可能是slave机器重起后，事务回滚造成的.
一般是事务回滚造成的，解决办法：
mysql> stop slave; Query OK, 0 rows affected (0.10 sec) mysql> set GLOBAL SQL_SLAVE_SKIP_COUNTER=1; Query OK, 0 rows affected (0.00 sec) mysql> start slave; Query OK, 0 rows affected (0.00 sec)
#2，从新 stop slave 然后change master
安装mha
安装依赖包
yum install perl-DBD-MySQL perl-Config-Tiny perl-Log-Dispatch perl-Parallel-ForkManager perl-ExtUtils-CBuilder perl-ExtUtils-MakeMaker perl-CPAN
本例使用slave2 作为mha管理机
slave2 安装manager
tar xf mha4mysql-manager-0.57.tar.gz cd mha4mysql-manager-0.57 perl Makefile.PL make && make install
三台mysql安装node
tar xf mha4mysql-node-0.57.tar.gz cd mha4mysql-node-0.57 perl Makefile.PL make && make install
安装过程中出现的问题看最下面
安装完成后会在/usr/local/bin目录下面生成相应的脚本
[root@slave2 bin]# ls /usr/local/bin/masterha_* /usr/local/bin/masterha_check_repl /usr/local/bin/masterha_conf_host /usr/local/bin/masterha_master_switch /usr/local/bin/masterha_check_ssh /usr/local/bin/masterha_manager /usr/local/bin/masterha_secondary_check /usr/local/bin/masterha_check_status /usr/local/bin/masterha_master_monitor /usr/local/bin/masterha_stop [root@slave2 bin]#
复制 mha4mysql-manager-0.53/samples/scripts/下的脚本到/usr/local/bin目录(manager 服务器执行)
master_ip_failover #自动切换时vip管理的脚本，不是必须，如果我们使用keepalived的，我们可以自己编写脚本完成对vip的管理，比如监控mysql，如果mysql异常，我们停止keepalived就行，这样vip就会自动漂移
master_ip_online_change #在线切换时vip的管理，不是必须，同样可以可以自行编写简单的shell完成
power_manager #故障发生后关闭主机的脚本，不是必须
send_report #因故障切换后发送报警的脚本，不是必须，可自行编写简单的shell完成。
修改/usr/local/bin/master_ip_failover 脚本
#!/usr/bin/env perl
use strict;
use warnings FATAL => 'all';
use Getopt::Long;
my (
$command, $ssh_user, $orig_master_host, $orig_master_ip,
$orig_master_port, $new_master_host, $new_master_ip, $new_master_port
);
my $vip = '192.168.132.135/24'; #此处为你要设置的虚拟ip
my $key = '1';
my $ssh_start_vip = "/sbin/ifconfig ens33:$key $vip"; #此处改为你的网卡名称
my $ssh_stop_vip = "/sbin/ifconfig ens33:$key down";
GetOptions(
'command=s' => \$command,
'ssh_user=s' => \$ssh_user,
'orig_master_host=s' => \$orig_master_host,
'orig_master_ip=s' => \$orig_master_ip,
'orig_master_port=i' => \$orig_master_port,
'new_master_host=s' => \$new_master_host,
'new_master_ip=s' => \$new_master_ip,
'new_master_port=i' => \$new_master_port,
);
exit &main();
sub main {
print "\n\nIN SCRIPT TEST====$ssh_stop_vip==$ssh_start_vip===\n\n";
if ( $command eq "stop" || $command eq "stopssh" ) {
my $exit_code = 1;
eval {
print "Disabling the VIP on old master: $orig_master_host \n";
&stop_vip();
$exit_code = 0;
};
if ($@) {
warn "Got Error: $@\n";
exit $exit_code;
}
exit $exit_code;
}
elsif ( $command eq "start" ) {
my $exit_code = 10;
eval {
print "Enabling the VIP - $vip on the new master - $new_master_host \n";
&start_vip();
$exit_code = 0;
};
if ($@) {
warn $@;
exit $exit_code;
}
exit $exit_code;
}
elsif ( $command eq "status" ) {
print "Checking the Status of the script.. OK \n";
exit 0;
}
else {
&usage();
exit 1;
}
}
sub start_vip() {
`ssh $ssh_user\@$new_master_host \" $ssh_start_vip \"`;
}
sub stop_vip() {
return 0 unless ($ssh_user);
`ssh $ssh_user\@$orig_master_host \" $ssh_stop_vip \"`;
}
sub usage {
print
"Usage: master_ip_failover --command=start|stop|stopssh|status --orig_master_host=host --orig_master_ip=ip --orig_master_port=port --new_master_host=host --new_master_ip=ip --new_master_port=port\n";
}
添加修改MHA配置文件
mkdir -p /etc/masterha cp mha4mysql-manager-0.57/samples/conf/app1.cnf /etc/masterha/
cat /etc/masterha/app1.cnf [server default] manager_workdir=/var/log/masterha/app1.log ##manager工作目录 manager_log=/var/log/masterha/app1/manager.log #manager日志 master_binlog_dir=/usr/local/mysql/data/ #master保存binlog的位置，这里的路径要与master里配置的binlog的路径一致，以便mha能找到 #master_ip_failover_script= /usr/local/bin/master_ip_failover #设置自动failover时候的切换脚本，也就是上边的哪个脚本 master_ip_online_change_script= /usr/local/bin/master_ip_online_change #设置手动切换时候的切换脚本 password=root #设置mysql中root用户的密码，这个密码是前文中创建监控用户的那个密码 user=root #设置监控用户root ping_interval=1 #设置监控主库，发送ping包的时间间隔，默认是3秒，尝试三次没有回应的时候自动进行railover remote_workdir=/tmp #设置远端mysql在发生切换时binlog的保存位置 repl_password=repl #设置复制用户的密码 repl_user=repl #设置复制用户的用户 report_script=/usr/local/send_report //设置发生切换后发送的报警的脚本 secondary_check_script= /usr/local/bin/masterha_secondary_check -s 192.168.132.131 -s 192.168.132.132 shutdown_script="" #设置故障发生后关闭故障主机脚本（该脚本的主要作用是关闭主机放在发生脑裂,这里没有使用） ssh_user=root #设置ssh的登录用户名 [server1] hostname=192.168.132.131 port=3306 #candidate_master=1 [server2] hostname=192.168.132.132 port=3306 candidate_master=1 #//设置为候选master，如果设置该参数以后，发生主从切换以后将会将此从库提升为主库，即使这个主库不是集群中事件最新的slave check_repl_delay=0 #默认情况下如果一个slave落后master 100M的relay logs的话，MHA将不会选择该slave作为一个新的master，因为对于这个slave的恢复需要花费很长时间，通过设置check_repl_delay=0,MHA触发切换在选择一个新的master的时候将会忽略复制延时，这个参数对于设置了candidate_master=1的主机非常有用，因为这个候选主在切换的过程中一定是新的master [server3] hostname=192.168.132.133 port=3306
#测试ssh连接情况
[root@slave2 bin]# masterha_check_ssh -conf=/etc/masterha/app1.cnf
#测试mysq集群的连接情况
masterha_check_repl -conf=/etc/masterha/app1.cnf
#检查MHA的状态
[root@slave2 bin]# masterha_check_status --conf=/etc/masterha/app1.cnf app1 is stopped(2:NOT_RUNNING). #没启动
master端添加vip
[root@master ~]# /sbin/ifconfig ens33:1 192.168.132.135/24
启动MHA
nohup masterha_manager --conf=/etc/masterha/app1.cnf --remove_dead_master_conf --ignore_last_failover < /dev/null > /var/log/masterha/app1/manager.log 2>&1 &
查看日志
表示启动成功,退出终端时会有监控进程异常退出的情况，解决办法是将命令写到脚本里，然后后台执行
杀掉主库mysql进程，模拟主库发生故障，进行自动failover操作。
[root@master ~]# service mysql stop
Shutting down MySQL............ SUCCESS!
查看mha 日志（自动切换成功）
Sun Apr 8 16:39:41 2018 - [warning] Got error on MySQL select ping: 2006 (MySQL server has gone away) Sun Apr 8 16:39:41 2018 - [info] Executing SSH check script: save_binary_logs --command=test --start_pos=4 --binlog_dir=/usr/local/mysql/data/ --output_file=/tmp/save_binary_logs_test --manager_version=0.53 --binlog_prefix=master-bin Sun Apr 8 16:39:41 2018 - [info] Executing seconary network check script: /usr/local/bin/masterha_secondary_check -s 192.168.132.132 -s 192.168.132.131 --user=root --master_host=192.168.132.131 --master_ip=192.168.132.131 --master_port=3306 Sun Apr 8 16:39:42 2018 - [info] HealthCheck: SSH to 192.168.132.131 is reachable. Monitoring server 192.168.132.132 is reachable, Master is not reachable from 192.168.132.132. OK. Monitoring server 192.168.132.131 is reachable, Master is not reachable from 192.168.132.131. OK. Sun Apr 8 16:39:42 2018 - [info] Master is not reachable from all other monitoring servers. Failover should start. Sun Apr 8 16:39:42 2018 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '192.168.132.131' (111)) Sun Apr 8 16:39:42 2018 - [warning] Connection failed 1 time(s).. Sun Apr 8 16:39:43 2018 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '192.168.132.131' (111)) Sun Apr 8 16:39:43 2018 - [warning] Connection failed 2 time(s).. Sun Apr 8 16:39:44 2018 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '192.168.132.131' (111)) Sun Apr 8 16:39:44 2018 - [warning] Connection failed 3 time(s).. Sun Apr 8 16:39:44 2018 - [warning] Master is not reachable from health checker! Sun Apr 8 16:39:44 2018 - [warning] Master 192.168.132.131(192.168.132.131:3306) is not reachable! Sun Apr 8 16:39:44 2018 - [warning] SSH is reachable. Sun Apr 8 16:39:44 2018 - [info] Connecting to a master server failed. Reading configuration file /etc/masterha_default.cnf and /etc/masterha/app1.cnf again, and trying to connect to all servers to check server status.. Sun Apr 8 16:39:44 2018 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping. Sun Apr 8 16:39:44 2018 - [info] Reading application default configurations from /etc/masterha/app1.cnf.. Sun Apr 8 16:39:44 2018 - [info] Reading server configurations from /etc/masterha/app1.cnf.. Sun Apr 8 16:39:46 2018 - [info] Dead Servers: Sun Apr 8 16:39:46 2018 - [info] 192.168.132.131(192.168.132.131:3306) Sun Apr 8 16:39:46 2018 - [info] Alive Servers: Sun Apr 8 16:39:46 2018 - [info] 192.168.132.132(192.168.132.132:3306) Sun Apr 8 16:39:46 2018 - [info] 192.168.132.133(192.168.132.133:3306) Sun Apr 8 16:39:46 2018 - [info] Alive Slaves: Sun Apr 8 16:39:46 2018 - [info] 192.168.132.132(192.168.132.132:3306) Version=5.7.17-log (oldest major version between slaves) log-bin:enabled Sun Apr 8 16:39:46 2018 - [info] Replicating from 192.168.132.131(192.168.132.131:3306) Sun Apr 8 16:39:46 2018 - [info] Primary candidate for the new Master (candidate_master is set) Sun Apr 8 16:39:46 2018 - [info] 192.168.132.133(192.168.132.133:3306) Version=5.7.17-log (oldest major version between slaves) log-bin:enabled Sun Apr 8 16:39:46 2018 - [info] Replicating from 192.168.132.131(192.168.132.131:3306) Sun Apr 8 16:39:46 2018 - [info] Checking slave configurations.. Sun Apr 8 16:39:46 2018 - [info] Checking replication filtering settings.. Sun Apr 8 16:39:46 2018 - [info] Replication filtering check ok. Sun Apr 8 16:39:46 2018 - [info] Master is down! Sun Apr 8 16:39:46 2018 - [info] Terminating monitoring script. Sun Apr 8 16:39:46 2018 - [info] Got exit code 20 (Master dead). Sun Apr 8 16:39:46 2018 - [info] MHA::MasterFailover version 0.53. Sun Apr 8 16:39:46 2018 - [info] Starting master failover. Sun Apr 8 16:39:46 2018 - [info] Sun Apr 8 16:39:46 2018 - [info] * Phase 1: Configuration Check Phase.. Sun Apr 8 16:39:46 2018 - [info] Sun Apr 8 16:39:47 2018 - [info] Dead Servers: Sun Apr 8 16:39:47 2018 - [info] 192.168.132.131(192.168.132.131:3306) Sun Apr 8 16:39:47 2018 - [info] Checking master reachability via mysql(double check).. Sun Apr 8 16:39:47 2018 - [info] ok. Sun Apr 8 16:39:47 2018 - [info] Alive Servers: Sun Apr 8 16:39:47 2018 - [info] 192.168.132.132(192.168.132.132:3306) Sun Apr 8 16:39:47 2018 - [info] 192.168.132.133(192.168.132.133:3306) Sun Apr 8 16:39:47 2018 - [info] Alive Slaves: Sun Apr 8 16:39:47 2018 - [info] 192.168.132.132(192.168.132.132:3306) Version=5.7.17-log (oldest major version between slaves) log-bin:enabled Sun Apr 8 16:39:47 2018 - [info] Replicating from 192.168.132.131(192.168.132.131:3306) Sun Apr 8 16:39:47 2018 - [info] Primary candidate for the new Master (candidate_master is set) Sun Apr 8 16:39:47 2018 - [info] 192.168.132.133(192.168.132.133:3306) Version=5.7.17-log (oldest major version between slaves) log-bin:enabled Sun Apr 8 16:39:47 2018 - [info] Replicating from 192.168.132.131(192.168.132.131:3306) Sun Apr 8 16:39:47 2018 - [info] ** Phase 1: Configuration Check Phase completed. Sun Apr 8 16:39:47 2018 - [info] Sun Apr 8 16:39:47 2018 - [info] * Phase 2: Dead Master Shutdown Phase.. Sun Apr 8 16:39:47 2018 - [info] Sun Apr 8 16:39:47 2018 - [info] Forcing shutdown so that applications never connect to the current master.. Sun Apr 8 16:39:47 2018 - [info] Executing master IP deactivatation script: Sun Apr 8 16:39:47 2018 - [info] /usr/local/bin/master_ip_failover --orig_master_host=192.168.132.131 --orig_master_ip=192.168.132.131 --orig_master_port=3306 --command=stopssh --ssh_user=root IN SCRIPT TEST====/sbin/ifconfig ens33:1 down==/sbin/ifconfig ens33:1 192.168.132.135/24=== Disabling the VIP on old master: 192.168.132.131 Sun Apr 8 16:39:47 2018 - [info] done. Sun Apr 8 16:39:47 2018 - [warning] shutdown_script is not set. Skipping explicit shutting down of the dead master. Sun Apr 8 16:39:47 2018 - [info] * Phase 2: Dead Master Shutdown Phase completed. Sun Apr 8 16:39:47 2018 - [info] Sun Apr 8 16:39:47 2018 - [info] * Phase 3: Master Recovery Phase.. Sun Apr 8 16:39:47 2018 - [info] Sun Apr 8 16:39:47 2018 - [info] * Phase 3.1: Getting Latest Slaves Phase.. Sun Apr 8 16:39:47 2018 - [info] Sun Apr 8 16:39:47 2018 - [info] The latest binary log file/position on all slaves is master-bin.000007:154 Sun Apr 8 16:39:47 2018 - [info] Latest slaves (Slaves that received relay log files to the latest): Sun Apr 8 16:39:47 2018 - [info] 192.168.132.132(192.168.132.132:3306) Version=5.7.17-log (oldest major version between slaves) log-bin:enabled Sun Apr 8 16:39:47 2018 - [info] Replicating from 192.168.132.131(192.168.132.131:3306) Sun Apr 8 16:39:47 2018 - [info] Primary candidate for the new Master (candidate_master is set) Sun Apr 8 16:39:47 2018 - [info] 192.168.132.133(192.168.132.133:3306) Version=5.7.17-log (oldest major version between slaves) log-bin:enabled Sun Apr 8 16:39:47 2018 - [info] Replicating from 192.168.132.131(192.168.132.131:3306) Sun Apr 8 16:39:47 2018 - [info] The oldest binary log file/position on all slaves is master-bin.000007:154 Sun Apr 8 16:39:47 2018 - [info] Oldest slaves: Sun Apr 8 16:39:47 2018 - [info] 192.168.132.132(192.168.132.132:3306) Version=5.7.17-log (oldest major version between slaves) log-bin:enabled Sun Apr 8 16:39:47 2018 - [info] Replicating from 192.168.132.131(192.168.132.131:3306) Sun Apr 8 16:39:47 2018 - [info] Primary candidate for the new Master (candidate_master is set) Sun Apr 8 16:39:47 2018 - [info] 192.168.132.133(192.168.132.133:3306) Version=5.7.17-log (oldest major version between slaves) log-bin:enabled Sun Apr 8 16:39:47 2018 - [info] Replicating from 192.168.132.131(192.168.132.131:3306) Sun Apr 8 16:39:47 2018 - [info] Sun Apr 8 16:39:47 2018 - [info] * Phase 3.2: Saving Dead Master's Binlog Phase.. Sun Apr 8 16:39:47 2018 - [info] Sun Apr 8 16:39:47 2018 - [info] Fetching dead master's binary logs.. Sun Apr 8 16:39:47 2018 - [info] Executing command on the dead master 192.168.132.131(192.168.132.131:3306): save_binary_logs --command=save --start_file=master-bin.000007 --start_pos=154 --binlog_dir=/usr/local/mysql/data/ --output_file=/tmp/saved_master_binlog_from_192.168.132.131_3306_20180408163946.binlog --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.53 Creating /tmp if not exists.. ok. Concat binary/relay logs from master-bin.000007 pos 154 to master-bin.000007 EOF into /tmp/saved_master_binlog_from_192.168.132.131_3306_20180408163946.binlog .. Dumping binlog format description event, from position 0 to 123.. ok. No need to dump effective binlog data from /usr/local/mysql/data//master-bin.000007 (pos starts 154, filesize 154). Skipping. /tmp/saved_master_binlog_from_192.168.132.131_3306_20180408163946.binlog has no effective data events. Event not exists. Sun Apr 8 16:39:47 2018 - [info] Additional events were not found from the orig master. No need to save. Sun Apr 8 16:39:47 2018 - [info] Sun Apr 8 16:39:47 2018 - [info] * Phase 3.3: Determining New Master Phase.. Sun Apr 8 16:39:47 2018 - [info] Sun Apr 8 16:39:47 2018 - [info] Finding the latest slave that has all relay logs for recovering other slaves.. Sun Apr 8 16:39:47 2018 - [info] All slaves received relay logs to the same position. No need to resync each other. Sun Apr 8 16:39:47 2018 - [info] Searching new master from slaves.. Sun Apr 8 16:39:47 2018 - [info] Candidate masters from the configuration file: Sun Apr 8 16:39:47 2018 - [info] 192.168.132.132(192.168.132.132:3306) Version=5.7.17-log (oldest major version between slaves) log-bin:enabled Sun Apr 8 16:39:47 2018 - [info] Replicating from 192.168.132.131(192.168.132.131:3306) Sun Apr 8 16:39:47 2018 - [info] Primary candidate for the new Master (candidate_master is set) Sun Apr 8 16:39:47 2018 - [info] Non-candidate masters: Sun Apr 8 16:39:47 2018 - [info] Searching from candidate_master slaves which have received the latest relay log events.. Sun Apr 8 16:39:47 2018 - [info] New master is 192.168.132.132(192.168.132.132:3306) Sun Apr 8 16:39:47 2018 - [info] Starting master failover.. Sun Apr 8 16:39:47 2018 - [info] From: 192.168.132.131 (current master) +--192.168.132.132 +--192.168.132.133 To: 192.168.132.132 (new master) +--192.168.132.133 Sun Apr 8 16:39:47 2018 - [info] Sun Apr 8 16:39:47 2018 - [info] * Phase 3.3: New Master Diff Log Generation Phase.. Sun Apr 8 16:39:47 2018 - [info] Sun Apr 8 16:39:47 2018 - [info] This server has all relay logs. No need to generate diff files from the latest slave. Sun Apr 8 16:39:47 2018 - [info] Sun Apr 8 16:39:47 2018 - [info] * Phase 3.4: Master Log Apply Phase.. Sun Apr 8 16:39:47 2018 - [info] Sun Apr 8 16:39:47 2018 - [info] *NOTICE: If any error happens from this phase, manual recovery is needed. Sun Apr 8 16:39:47 2018 - [info] Starting recovery on 192.168.132.132(192.168.132.132:3306).. Sun Apr 8 16:39:47 2018 - [info] This server has all relay logs. Waiting all logs to be applied.. Sun Apr 8 16:39:47 2018 - [info] done. Sun Apr 8 16:39:47 2018 - [info] All relay logs were successfully applied. Sun Apr 8 16:39:47 2018 - [info] Getting new master's binlog name and position.. Sun Apr 8 16:39:47 2018 - [info] slave-bin.000005:2183 Sun Apr 8 16:39:47 2018 - [info] All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='192.168.132.132', MASTER_PORT=3306, MASTER_LOG_FILE='slave-bin.000005', MASTER_LOG_POS=2183, MASTER_USER='repl', MASTER_PASSWORD='xxx'; Sun Apr 8 16:39:47 2018 - [info] Executing master IP activate script: Sun Apr 8 16:39:47 2018 - [info] /usr/local/bin/master_ip_failover --command=start --ssh_user=root --orig_master_host=192.168.132.131 --orig_master_ip=192.168.132.131 --orig_master_port=3306 --new_master_host=192.168.132.132 --new_master_ip=192.168.132.132 --new_master_port=3306 IN SCRIPT TEST====/sbin/ifconfig ens33:1 down==/sbin/ifconfig ens33:1 192.168.132.135/24=== Enabling the VIP - 192.168.132.135/24 on the new master - 192.168.132.132 Sun Apr 8 16:39:47 2018 - [info] OK. Sun Apr 8 16:39:47 2018 - [info] Setting read_only=0 on 192.168.132.132(192.168.132.132:3306).. Sun Apr 8 16:39:47 2018 - [info] ok. Sun Apr 8 16:39:47 2018 - [info] ** Finished master recovery successfully. Sun Apr 8 16:39:47 2018 - [info] * Phase 3: Master Recovery Phase completed. Sun Apr 8 16:39:47 2018 - [info] Sun Apr 8 16:39:47 2018 - [info] * Phase 4: Slaves Recovery Phase.. Sun Apr 8 16:39:47 2018 - [info] Sun Apr 8 16:39:47 2018 - [info] * Phase 4.1: Starting Parallel Slave Diff Log Generation Phase.. Sun Apr 8 16:39:47 2018 - [info] Sun Apr 8 16:39:47 2018 - [info] -- Slave diff file generation on host 192.168.132.133(192.168.132.133:3306) started, pid: 3586. Check tmp log /var/log/masterha/app1.log/192.168.132.133_3306_20180408163946.log if it takes time.. Sun Apr 8 16:39:48 2018 - [info] Sun Apr 8 16:39:48 2018 - [info] Log messages from 192.168.132.133 ... Sun Apr 8 16:39:48 2018 - [info] Sun Apr 8 16:39:47 2018 - [info] This server has all relay logs. No need to generate diff files from the latest slave. Sun Apr 8 16:39:48 2018 - [info] End of log messages from 192.168.132.133. Sun Apr 8 16:39:48 2018 - [info] -- 192.168.132.133(192.168.132.133:3306) has the latest relay log events. Sun Apr 8 16:39:48 2018 - [info] Generating relay diff files from the latest slave succeeded. Sun Apr 8 16:39:48 2018 - [info] Sun Apr 8 16:39:48 2018 - [info] * Phase 4.2: Starting Parallel Slave Log Apply Phase.. Sun Apr 8 16:39:48 2018 - [info] Sun Apr 8 16:39:48 2018 - [info] -- Slave recovery on host 192.168.132.133(192.168.132.133:3306) started, pid: 3588. Check tmp log /var/log/masterha/app1.log/192.168.132.133_3306_20180408163946.log if it takes time.. Sun Apr 8 16:39:49 2018 - [info] Sun Apr 8 16:39:49 2018 - [info] Log messages from 192.168.132.133 ... Sun Apr 8 16:39:49 2018 - [info] Sun Apr 8 16:39:48 2018 - [info] Starting recovery on 192.168.132.133(192.168.132.133:3306).. Sun Apr 8 16:39:48 2018 - [info] This server has all relay logs. Waiting all logs to be applied.. Sun Apr 8 16:39:48 2018 - [info] done. Sun Apr 8 16:39:48 2018 - [info] All relay logs were successfully applied. Sun Apr 8 16:39:48 2018 - [info] Resetting slave 192.168.132.133(192.168.132.133:3306) and starting replication from the new master 192.168.132.132(192.168.132.132:3306).. Sun Apr 8 16:39:49 2018 - [info] Executed CHANGE MASTER. Sun Apr 8 16:39:49 2018 - [info] Slave started. Sun Apr 8 16:39:49 2018 - [info] End of log messages from 192.168.132.133. Sun Apr 8 16:39:49 2018 - [info] -- Slave recovery on host 192.168.132.133(192.168.132.133:3306) succeeded. Sun Apr 8 16:39:49 2018 - [info] All new slave servers recovered successfully. Sun Apr 8 16:39:49 2018 - [info] Sun Apr 8 16:39:49 2018 - [info] * Phase 5: New master cleanup phease.. Sun Apr 8 16:39:49 2018 - [info] Sun Apr 8 16:39:49 2018 - [info] Resetting slave info on the new master.. Sun Apr 8 16:39:49 2018 - [info] 192.168.132.132: Resetting slave info succeeded. Sun Apr 8 16:39:49 2018 - [info] Master failover to 192.168.132.132(192.168.132.132:3306) completed successfully. Sun Apr 8 16:39:49 2018 - [info] Deleted server1 entry from /etc/masterha/app1.cnf . Sun Apr 8 16:39:49 2018 - [info] ----- Failover Report ----- app1: MySQL Master failover 192.168.132.131 to 192.168.132.132 succeeded Master 192.168.132.131 is down! Check MHA Manager logs at slave2:/var/log/masterha/app1/manager.log for details. Started automated(non-interactive) failover. Invalidated master IP address on 192.168.132.131. The latest slave 192.168.132.132(192.168.132.132:3306) has all relay logs for recovery. Selected 192.168.132.132 as a new master. 192.168.132.132: OK: Applying all logs succeeded. 192.168.132.132: OK: Activated master IP address. 192.168.132.133: This host has the latest relay log events. Generating relay diff files from the latest slave succeeded. 192.168.132.133: OK: Applying all logs succeeded. Slave started, replicating from 192.168.132.132. 192.168.132.132: Resetting slave info succeeded. Master failover to 192.168.132.132(192.168.132.132:3306) completed successfully.