如何将RDS的数据同步到本地自建数据库

如何将RDS的数据同步到本地自建数据库（转载）

  长期以来有很多的用户咨询如何将RDS的数据同步到本地的数据库环境中，本篇文章以在阿里云的ESC服务器为例来说明如何将RDS的数据同步到本地数据库中。RDS对外提供服务是一个DNS地址+端口3306.这样就屏蔽了RDS后端的主从节点，那么该如何将数据同步到本地？那么我们来看一下RDS的主从架构。

应用连接RDS，流量通过SLB指向主从节点的master，所以如果我们连接RDS的账号具有REPLICATION SLAVE,REPLICAITON CLIENTd的权限，则就可以把主库master的产生binlog同步到本地数据库中去，实现数据同步。

     但是这种根据binlog文件和位点来同步RDS的数据到本地的方式非常容易导致同步中断，因为当RDS发生了主备切换（主备切换，重启，跨机迁移），本地数据库所指向RDS的binlog位点则会发生变化。

   位点则会发生变化（RDS主库与备库的binlog位点是不一一致的），这样就会导致本地数据库与RDS的数据复制同步中断。

  在RDS：5.6版本中主备同步使用新复制方式GTID，RDS的主备具有相同的GTID，那么如果主备发生切换，重启或者迁移，主备的GTID是不是发生变化，那么ESC与RDS的同步链路则

 会不发发生中断，所以如果将RDS的数据同步到本地，则需要将RDS升级到5.6版本。

    下面我们将RDS数据同步到本地的一些关键步骤罗列出来：

      1、在ECS服务器上安装MySQL，详细步骤如下：

         1）配置文件如下：

          

 cat /etc/my3308.cnf 
[client]
port            = 3308
socket          = /home/data/mysql3308/run/mysql3308.sock

[mysqld]
port            = 3308
socket          = /home/data/mysql3308/run/mysql3308.sock
datadir         = /home/data/mysql3308/data
#read_only       = on
#skip-grant-tables
#--- GLOBAL ---#
character_set_server    = utf8
lower_case_table_names  = 2
log-output              = FILE
log-error               = /home/data/mysql3308/log/mysql-error.log
#general_log
general_log_file        = /home/data/mysql3308/log/mysql.log
pid-file                = /home/data/mysql3308/log/mysql.pid
slow-query-log          = 1
slow_query_log_file     = /home/data/mysql3308/log/mysql-slow.log
tmpdir                  = /home/data/mysql3308/tmp
long_query_time         = 2
innodb_force_recovery   = 0
#innodb_buffer_pool_dump_at_shutdown = 1
#innodb_buffer_pool_load_at_startup = 1
#--------------#

#thread_concurrency      = 8  
thread_cache_size       = 51  
table_open_cache        = 16384
open_files_limit        = 65535
table_definition_cache  = 16384
sort_buffer_size        = 2M
join_buffer_size        = 2M
read_buffer_size        = 2M
read_rnd_buffer_size    = 8M
key_buffer_size         = 32M
bulk_insert_buffer_size = 16M
myisam_sort_buffer_size = 64M
tmp_table_size          = 32M
max_heap_table_size     = 16M 
query_cache_size       = 32MB

gtid_mode=on
log_slave_updates=1
enforce_gtid_consistency=1

#--- NETWORK ---#
back_log                = 103
max-connections         = 512
max_connect_errors      = 100000
max_allowed_packet      = 32M
interactive_timeout     = 28800
wait_timeout            = 28800
skip-external-locking
#max_user_connections    = 0
external-locking        = FALSE
skip-name-resolve

#--- REPL ---#
server-id               = 2133308
sync_binlog             = 1
log-bin                 = mysql-bin
binlog_format           = row
report_host           =172.17.177.65   #使用内网地址，否则有坑，后期从库连接有问题
report_port             =3308
master-info-repository = TABLE
relay-log-info-repository = TABLE
expire_logs_days        = 10
relay-log               = relay-log
replicate-ignore-db     = test
log_slave_updates       =1
#skip-slave-start
binlog_cache_size       =64M
max_binlog_cache_size   =64M
max_binlog_size         =1024M



#--- INNODB ---#
default_storage_engine          = InnoDB
#innodb_data_file_path           = ibdata1:200M:autoextend
innodb_buffer_pool_size         = 6G
innodb_buffer_pool_instances    = 2
innodb_additional_mem_pool_size = 16M
innodb_log_files_in_group       = 2
innodb_log_file_size            = 256MB
innodb_log_buffer_size          = 16M
innodb_flush_log_at_trx_commit  = 2
innodb_lock_wait_timeout        = 30
innodb_flush_method             = O_DIRECT
innodb_max_dirty_pages_pct      = 75
innodb_io_capacity              = 200
innodb_thread_concurrency       = 32
innodb_open_files               = 65535
innodb_file_per_table           = 1
transaction_isolation           = REPEATABLE-READ
innodb_locks_unsafe_for_binlog  = 0
#innodb_purge_thread             = 4
[mysqldump]
quick
max_allowed_packet = 32M

[mysql]
auto-rehash
# Remove the next comment character if you are not familiar with SQL
#safe-updates
default_character_set=utf8


[mysqlhotcopy]
interactive-timeout

 

 

2、主库

   建立主从复制账号：

grant replication slave,replication client on *.* to \'repl\'@\'%\' identified by \'repl123\';

 

 

3、将RDS中的数据备份

     1）逻辑备份：

 mysqldump -uroot -p  -S /hoem/data/mysql3308/run/mysql3308.sock --skip-opt --create-options --add-drop-table --set-charset  --single-transaction  -q -e     --triggers  -R  -E  --set-gtid-purged=OFF --hex-blob --all-databases >/home/sa/all_database.sql 

     2）物理备份：

      可以使用RDS提供食物物理备份文件恢复到本地MySQL中，可以参考：

       http://help.aliyun.com/knowledge_detail/5973700.html?spm=5176.7114037.1996646101.1.7qe3ot&pos=1

   4、将RDS的全备恢复到本地自建数据库

   5、设置本地数据库与RDS的同步关系

CHANGE MASTER TO
MASTER_HOST=\'172.17.177.65\',
MASTER_PORT=3308,
MASTER_USER=\'repl\',
MASTER_PASSWORD=\'repl123\',
MASTER_AUTO_POSITION = 1;

 

 

出现报错：

mysql> change master to master_host=\'rm-2zeo78t48qq63lithvo.mysql.rds.aliyuncs.com\', master_port=3306, master_user=\'rep\',master_password=\'Repl123#\',master_auto_position = 1;
ERROR 1794 (HY000): Slave is not configured or failed to initialize properly. You must at least set --server-id to enable either a master or a slave. Additional error messages can be found in the MySQL error log.

 

原因是由于RDS的备份文件中包含了RDS得主从复制关系，需要把这些主从复制关系清理掉，清理方法如下：

truncate table  slave_relay_log_info;
truncate table  mysql.slave_master_info;
truncate table  mysql.slave_worker_info;

 

 然后重启MySQL

CHANGE MASTER TO MASTER_HOST=\'172.17.177.65\',MASTER_PORT=3308,MASTER_USER=\'repl\',MASTER_PASSWORD=\'repl123\',MASTER_AUTO_POSITION = 1;

start slave;

 

查看主从复制状态:

 show slave status\G;

结果 start slave 报错：

Last_IO_Error: Got fatal error 1236 from master when reading data from binary log: \'The slave is connecting using CHANGE MASTER TO MASTER_AUTO_POSITION = 1, but the master has purged binary logs containing GTIDs that the slave requires.\'

 

 

 一般两种情况会出现以上现象：

1、在主库上手动执行清除二进制日志文件

2、主库重启，重新同步时

 

解决方法：

 1）在主库上执行以下命令，查询gtid_purged，记录下改值

mysql>show global variables like \'%gtid%\'\G;
*************************** 1. row ***************************
Variable_name: enforce_gtid_consistency
        Value: ON
*************************** 2. row ***************************
Variable_name: gtid_executed
        Value: 19d500c9-6bb6-11e7-968f-7cd30ab8a76e:1-47168560,
1b80a514-1acb-11e7-86e8-7cd30ac20c2c:1-219595,
37021afb-1acb-11e7-86e9-7cd30ac209ea:1-9362,
382a2da5-6bb6-11e7-968f-5cb901891770:1-5231502,
719b5564-20ff-11e7-af5d-70106fac13fa:1-17273221,
7c17f9c2-1b65-11e7-8ad7-246e96063710:1-2561045,
7cc72769-280a-11e8-a29f-7cd30ae45d0e:1-27577930,
7f54f9cc-6b92-11e7-95a7-246e96063710:1-104035,
96a38f0b-20ff-11e7-af5e-70106f4feb64:1-13678911
*************************** 3. row ***************************
Variable_name: gtid_mode
        Value: ON
*************************** 4. row ***************************
Variable_name: gtid_owned
        Value: 
*************************** 5. row ***************************
Variable_name: gtid_purged
        Value: 19d500c9-6bb6-11e7-968f-7cd30ab8a76e:1-47168560,
1b80a514-1acb-11e7-86e8-7cd30ac20c2c:1-219595,
37021afb-1acb-11e7-86e9-7cd30ac209ea:1-9362,
382a2da5-6bb6-11e7-968f-5cb901891770:1-5231502,
719b5564-20ff-11e7-af5d-70106fac13fa:1-17273221,
7c17f9c2-1b65-11e7-8ad7-246e96063710:1-2561045,
7cc72769-280a-11e8-a29f-7cd30ae45d0e:1-27470347,
7f54f9cc-6b92-11e7-95a7-246e96063710:1-104035,
96a38f0b-20ff-11e7-af5e-70106f4feb64:1-13678911

 

 

2）在从库上执行以下命令停止同步线程及重置同步相关信息
    stop slave;
   reset slave;
  reset master;

 

3)从库

注意：打开备份解压文件可以看到文件xtrabackup_slave_info，其中第一行就是我们需要在本地MySQL执行的命令，他表示在备份结束时刻RDS当前GTID值\'

set @@global.gtid_purged=\'19d500c9-6bb6-11e7-968f-7cd30ab8a76e:1-47168560,
1b80a514-1acb-11e7-86e8-7cd30ac20c2c:1-219595,
37021afb-1acb-11e7-86e9-7cd30ac209ea:1-9362,
382a2da5-6bb6-11e7-968f-5cb901891770:1-5231502,
719b5564-20ff-11e7-af5d-70106fac13fa:1-17273221,
7c17f9c2-1b65-11e7-8ad7-246e96063710:1-2561045,
7cc72769-280a-11e8-a29f-7cd30ae45d0e:1-27470347,
7f54f9cc-6b92-11e7-95a7-246e96063710:1-104035,
96a38f0b-20ff-11e7-af5e-70106f4feb64:1-13678911\';

 4）重新开启同步

CHANGE MASTER TO MASTER_HOST=\'172.17.177.65\',MASTER_PORT=3308,MASTER_USER=\'repl\',MASTER_PASSWORD=\'repl123\',MASTER_AUTO_POSITION = 1;
mysql>start slave;