In this blog, I’ll look at how to do MySQL point in time recovery (PITR) correctly.
Sometimes we need to restore from a backup, and then replay the transactions that happened after the backup was taken. This is a common procedure in most disaster recovery plans, when for example you accidentally drop a table/database or run an update/delete without the “where” clause and lose data.
The usual way is to get a copy of your binlogs and use mysqlbinlog to replay those transactions. But this approach has many pitfalls that can make the whole PITR process a nightmare. Some examples:
- You need to make sure to run a single mysqlbinlog command with all related binlogs, and pipe them to mysql at once. Otherwise, if binlog.000001 creates a temporary table, and binlog.000002 requires that temporary table, it will not be present. Each execution of MySQL creates a new connection:
|
1
2
|
# Creates tmp table X
# Uses tmp table X
|
- We can say that it has to be an atomic operation. If it fails halfway through, it will be very difficult to know where it failed and even more difficult to resume from that point forward. There are many reasons for it to fail: InnoDB lock wait timeout / deadlock with some concurrent transaction, server and client have differentmax_allowed_packet and you get a
Lost connection to MySQL server during queryerror, and so on.
So how can we overcome those limitations and have a reliable way to do Point In Time Recovery?
We can restore the backup on the desired server, build a second server with just the minimal MySQL required data and move the all binary logs to this “fake” server datadir. Then we need to configure the server where we want the PITR to happen as a slave of the fake server. From this point forward, it’s going to be pure MySQL replication happening.
To illustrate it, I create a Docker container on the machine. I have Percona Server for MySQL running on the box listening on 3306, and have already restored the backup on it. There is a tarball there with all binlogs required. The saved positions for PITR are as follows:
|
1
2
|
# cat /var/lib/mysql/xtrabackup_binlog_info
1518932
|
I create a folder to store the Docker MySQL datadir:
|
1
2
|
pitr
pitr
|
I start the Docker container. As we can see from xtrabackup_binlog_info, my binlogs are named master-bin and I’ll be setting the same server-id as original master:
|
1
2
3
4
|
mysql
secret
5.7.18
10
|
In case you want to make usage of GTID, append --gtid-mode=ON --enforce_gtid_consistency=ON to the end of the Docker command.
The command above starts a MySQL instance, invokes mysqld –initialize, sets the root password to secret and it’s port 3306 is mapped back to my local 3307 port. Now I’ll stop it, remove the binlogs that it created, uncompress and move all required binlogs to its datadir mapped folder and start it again:
|
1
2
3
4
5
|
ps_pitr
*
pitr
*
ps_pitr
|
If it all worked correctly, at this point we can see the full list of binary logs on the Docker container by connecting on port 3307:
|
1
2
3
4
5
6
7
8
9
10
11
|
.
+
|
+
|
|
|
.
|
+
|
Now, all we need to do is connect to our server, which has the backup restored, and configure it as a slave from 3307:
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
|
p
:
.
6
d7301f8
affiliates
.
its
respective
.
.
;
)
;
)
STATUSG
*
event
127.0.0.1
root
3307
60
000008
449696
000002
28957
000007
Yes
Yes
:
:
:
:
:
:
0
:
0
15217950
11476311
None
:
0
No
:
:
:
:
:
4382
No
0
:
0
:
:
10
0242ac110002
.info
0
NULL
tables
86400
:
:
:
:
:
:
:
0
:
:
:
)
.
STATUSG
*
event
127.0.0.1
root
3307
60
000074
154
000133
381
000074
Yes
Yes
:
:
:
:
:
:
0
:
0
154
819
None
:
0
No
:
:
:
:
:
0
No
0
:
0
:
:
10
0242ac110002
.info
0
NULL
updates
86400
:
:
:
:
:
:
:
0
:
:
:
)
|
If you want to apply logs up to a particular time you can make use of mysqlbinlog to verify what the last position / GTID it should apply, and use START SLAVE UNTIL MASTER_LOG_FILE = 'log_name', MASTER_LOG_POS = log_pos or START SLAVE SQL_THREAD UNTIL SQL_AFTER_GTIDS = 3E11FA47-71CA-11E1-9E33-C80AA9429562:11-56.
Special thanks to Marcos Albe, who originally showed me this MySQL point in time recovery approach