InnoDB：标题页由数据文件中的零字节组成：./ibdata1

Question

我在 openshift (kubernetes) 上部署 MySQL，但出现以下错误

---> 10:01:34     Initializing database ...
---> 10:01:34     Running /opt/rh/rh-mysql57/root/usr/libexec/mysqld --initialize --datadir=/var/lib/mysql/data --ignore-db-dir=lost+found
---> 10:01:35     Starting MySQL server with disabled networking ...
---> 10:01:35     Waiting for MySQL to start ...
2019-10-18T10:01:35.188509Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2019-10-18T10:01:35.188567Z 0 [Warning] 'NO_AUTO_CREATE_USER' sql mode was not set.
2019-10-18T10:01:35.325995Z 0 [Note] /opt/rh/rh-mysql57/root/usr/libexec/mysqld (mysqld 5.7.24) starting as process 40 ...
2019-10-18T10:01:35.430574Z 0 [Note] InnoDB: PUNCH HOLE support available
2019-10-18T10:01:35.430612Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2019-10-18T10:01:35.430621Z 0 [Note] InnoDB: Uses event mutexes
2019-10-18T10:01:35.430627Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2019-10-18T10:01:35.430634Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.7
2019-10-18T10:01:35.430642Z 0 [Note] InnoDB: Using Linux native AIO
2019-10-18T10:01:35.431024Z 0 [Note] InnoDB: Number of pools: 1
2019-10-18T10:01:35.431174Z 0 [Note] InnoDB: Using CPU crc32 instructions
2019-10-18T10:01:35.433296Z 0 [Note] InnoDB: Initializing buffer pool, total size = 2G, instances = 8, chunk size = 128M
2019-10-18T10:01:35.584269Z 0 [Note] InnoDB: Completed initialization of buffer pool
2019-10-18T10:01:35.610249Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2019-10-18T10:01:35.624480Z 0 [ERROR] InnoDB: Header page consists of zero bytes in datafile: ./ibdata1, Space ID:0, Flags: 0. Please refer to http://dev.mysql.com/doc/refman/5.7/en/innodb-troubleshooting-datadict.html  for how to resolve the issue.
2019-10-18T10:01:35.624518Z 0 [ERROR] InnoDB: Corrupted page [page id: space=0, page number=0] of datafile './ibdata1' could not be found in the doublewrite buffer.
2019-10-18T10:01:35.624667Z 0 [ERROR] InnoDB: Plugin initialization aborted with error Data structure corruption
---> 10:01:36     Waiting for MySQL to start ...
2019-10-18T10:01:36.225356Z 0 [ERROR] Plugin 'InnoDB' init function returned error.
2019-10-18T10:01:36.225372Z 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2019-10-18T10:01:36.225377Z 0 [ERROR] Failed to initialize builtin plugins.
2019-10-18T10:01:36.225380Z 0 [ERROR] Aborting
2019-10-18T10:01:36.225393Z 0 [Note] Binlog end
2019-10-18T10:01:36.225454Z 0 [Note] Shutting down plugin 'CSV'
2019-10-18T10:01:36.233760Z 0 [Note] /opt/rh/rh-mysql57/root/usr/libexec/mysqld: Shutdown complete

我在谷歌上搜索，他们建议做以下操作

rm -rf /var/lib/mysql/ib_logfile*

但是正如您在 docker 和 kubernetes 世界中所知道的那样，一旦 pod 无法启动，我们就无法登录到 pod 并删除这些文件。由于这个错误，MySQL pod 没有出现。

我正在使用 glusterfs，我删除了 PVC 并重新创建，但仍然出现同样的错误。

更新1： 在@Crou 推荐后，我做了以下更改

• 我下载了新的“centos/mysql-57-centos7:5.7”镜像，重试还是同样的问题

• 我使用了 mysql:5.7 图像，现在它抛出以下错误

• 我删除了 PVC 并重新创建了不同的大小但仍然相同的错误。

[ERROR] --initialize specified but the data directory has files in it. Aborting. [ERROR] Aborting

Answer 1

在我看来，您的图像已损坏，或者至少您尝试加载的数据已损坏。

即使您确实删除了文件 /var/lib/mysql/ib_logfile*，当重新安排 pod 时，这也会再次中断。

您可以尝试在pod 仍在启动时使用kubectl exec -it <pod_name> bash 进入它并快速进行更改。如果为时已晚，并且使用crashloopbackoff 在几秒钟内失败，您可以尝试在用于创建图像的dockerfile 中修复此问题。如果您无权访问 dockerfile，您可以创建自己的 dockerfile 并基于它构建映像。

这是一个带有Dockerfile reference 的文档，可能会有所帮助。

Dockerfile 包含如下内容：

FROM oraclelinux:7-slim

RUN rm -rf /var/lib/mysql/ib_logfile*

EXPOSE 3306 6606 6446 6447 33060

CMD [""]

请记住，这只是一个简短的示例，可能无法自行处理。

我找到了这个 github mattlord/Docker-InnoDB-Cluster，也许会有所帮助。

Answer 2

经过不同的重试后，我能够解决问题，以下是调查结果

• 我观察到，当我第一次重新创建 PVC 并部署 mysql 时，在这一行花费了太多时间

  Running /opt/rh/rh-mysql57/root/usr/libexec/mysqld --initialize --datadir=/var/lib/mysql/data --ignore-db-dir=lost+found

• 在此期间 LiveProbe 时间耗尽并重新启动容器，导致出现错误。

• 我把livenessProbe下initialDelaySeconds的时间增加到240，这次是4分钟后启动mysql

• 4 分钟后显示Starting MySQL server with disabled networking 并开始进一步操作。

我的应用程序能够连接它，但我不知道这会产生什么影响？

---> 16:18:24     Running /opt/rh/rh-mysql57/root/usr/libexec/mysqld --initialize --datadir=/var/lib/mysql/data --ignore-db-dir=lost+found
---> 16:22:15     Starting MySQL server with disabled networking ...
---> 16:22:15     Waiting for MySQL to start ...
2019-10-18T16:22:15.752499Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2019-10-18T16:22:15.772646Z 0 [Note] /opt/rh/rh-mysql57/root/usr/libexec/mysqld (mysqld 5.7.24) starting as process 69 ...
2019-10-18T16:22:15.849580Z 0 [Note] InnoDB: PUNCH HOLE support available
2019-10-18T16:22:15.849613Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2019-10-18T16:22:15.849617Z 0 [Note] InnoDB: Uses event mutexes
2019-10-18T16:22:15.849621Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2019-10-18T16:22:15.849624Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.7
2019-10-18T16:22:15.849629Z 0 [Note] InnoDB: Using Linux native AIO
2019-10-18T16:22:15.850236Z 0 [Note] InnoDB: Number of pools: 1
2019-10-18T16:22:15.850459Z 0 [Note] InnoDB: Using CPU crc32 instructions
2019-10-18T16:22:15.852211Z 0 [Note] InnoDB: Initializing buffer pool, total size = 2G, instances = 8, chunk size = 128M
2019-10-18T16:22:15.978704Z 0 [Note] InnoDB: Completed initialization of buffer pool
2019-10-18T16:22:16.005183Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2019-10-18T16:22:16.208237Z 0 [Note] InnoDB: Highest supported file format is Barracuda.
2019-10-18T16:22:16.402086Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2019-10-18T16:22:16.414708Z 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
---> 16:22:17     Waiting for MySQL to start ...
---> 16:22:18     Waiting for MySQL to start ...
2019-10-18T16:22:18.601639Z 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2019-10-18T16:22:18.608922Z 0 [Note] InnoDB: 96 redo rollback segment(s) found. 96 redo rollback segment(s) are active.
2019-10-18T16:22:18.608943Z 0 [Note] InnoDB: 32 non-redo rollback segment(s) are active.
2019-10-18T16:22:18.609240Z 0 [Note] InnoDB: Waiting for purge to start
2019-10-18T16:22:18.659387Z 0 [Note] InnoDB: 5.7.24 started; log sequence number 2592555
2019-10-18T16:22:18.659667Z 0 [Note] Plugin 'FEDERATED' is disabled.
2019-10-18T16:22:18.663687Z 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/data/ib_buffer_pool
2019-10-18T16:22:18.808093Z 0 [Note] Found ca.pem, server-cert.pem and server-key.pem in data directory. Trying to enable SSL support using them.
2019-10-18T16:22:18.808117Z 0 [Note] Skipping generation of SSL certificates as certificate files are present in data directory.
2019-10-18T16:22:18.825225Z 0 [Warning] CA certificate ca.pem is self signed.
2019-10-18T16:22:18.828435Z 0 [Note] Skipping generation of RSA key pair as key files are present in data directory.
2019-10-18T16:22:18.885079Z 0 [Note] InnoDB: Buffer pool(s) load completed at 191018 16:22:18
2019-10-18T16:22:18.948358Z 0 [Warning] 'user' entry 'root@localhost' ignored in --skip-name-resolve mode.
2019-10-18T16:22:18.948397Z 0 [Warning] 'user' entry 'mysql.session@localhost' ignored in --skip-name-resolve mode.
2019-10-18T16:22:18.948404Z 0 [Warning] 'user' entry 'mysql.sys@localhost' ignored in --skip-name-resolve mode.
2019-10-18T16:22:18.948428Z 0 [Warning] 'db' entry 'performance_schema mysql.session@localhost' ignored in --skip-name-resolve mode.
2019-10-18T16:22:18.948435Z 0 [Warning] 'db' entry 'sys mysql.sys@localhost' ignored in --skip-name-resolve mode.
2019-10-18T16:22:18.948448Z 0 [Warning] 'proxies_priv' entry '@ root@localhost' ignored in --skip-name-resolve mode.
2019-10-18T16:22:19.034170Z 0 [Warning] 'tables_priv' entry 'user mysql.session@localhost' ignored in --skip-name-resolve mode.
2019-10-18T16:22:19.034191Z 0 [Warning] 'tables_priv' entry 'sys_config mysql.sys@localhost' ignored in --skip-name-resolve mode.
2019-10-18T16:22:19.633088Z 0 [Note] Event Scheduler: Loaded 0 events
2019-10-18T16:22:19.633217Z 1 [Note] Event Scheduler: scheduler thread started with id 1
2019-10-18T16:22:19.633316Z 0 [Note] /opt/rh/rh-mysql57/root/usr/libexec/mysqld: ready for connections.
Version: '5.7.24'  socket: '/tmp/mysql.sock'  port: 0  MySQL Community Server (GPL)
2019-10-18T16:22:19.634316Z 3 [Note] Your password has expired. To log in you must change it using a client that supports expired passwords.
---> 16:22:19     MySQL started successfully
---> 16:22:19     Setting password for MySQL root user ...