使用 100% CPU 的 mySQL

Question

我有一个在 LAMP 堆栈上运行的 PHP 应用程序。此应用程序通过 javascript 向服务器进行 API 回调，以每秒获取更多数据以显示在屏幕上。当有多个用户同时使用它时，比如 80 个，mySQL 将 CPU 猛冲到 100%，直到应用程序完成。

我在用什么：

• mysql 5.7.31
• Ubuntu 18.04

在大小为 m5.xlarge 的 EC2 实例上运行

• 4 个 vCPU
• 16G 内存
• 网络带宽高达 10Gbps

我使用了 percona 关于调整 mySQL 参数的建议，他们说大多数 5.7 都有很好的默认值，期望有几个取决于你的硬件，所以我的 mySQL 配置看起来像这样

mysqld.cnf

[mysqld_safe]
socket          = /var/run/mysqld/mysqld.sock
nice            = 0
default-character-set=utf8

[mysqld]
#
# * Basic Settings
#
user            = mysql
pid-file        = /var/run/mysqld/mysqld.pid
socket          = /var/run/mysqld/mysqld.sock
port            = 3306
basedir         = /usr
datadir         = /var/lib/mysql
tmpdir          = /tmp
lc-messages-dir = /usr/share/mysql
skip-external-locking
character-set-client-handshake = false #force encoding to uft8
character-set-server=utf8
collation-server=utf8_general_ci
sql_mode = 'IGNORE_SPACE,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION'

bind-address            = 0.0.0.0
key_buffer_size         = 16M
max_allowed_packet      = 16M
thread_stack            = 192K
thread_cache_size       = 8
myisam-recover-options  = BACKUP
query_cache_limit       = 1M
query_cache_size        = 256M

log_error = /var/log/mysql/error.log

expire_logs_days        = 10
max_binlog_size   = 100M
#binlog_do_db           = include_database_name
#binlog_ignore_db       = include_database_name
#
# * InnoDB
#
# InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/.
# Read the manual for more InnoDB related options. There are many!
#

innodb_buffer_pool_size = 11G # (adjust value here, 50%-70% of total RAM)
innodb_log_file_size = 256M
innodb_flush_log_at_trx_commit = 1
innodb_flush_method = O_DIRECT

Percona 监控和管理

我还在运行 Percona 监控和管理，这让我可以很好地了解正在发生的事情。

所以当我拥有 100% 的 CPU 时，这就是我所确定的

1. CPU 是 100%，并且在用户空间中 - 这是因为我的 innoDB 缓冲池大小非常大，所有数据都在内存中，所以 HDD 没有被击中，因此没有 IO

2. 未达到最大连接数 - 150 个连接中的 100 个连接在持续时间内被使用

3. 慢查询日志里面什么也没有

4. 顶级计数器似乎是 com_select

5. 还有顶级处理程序 read_next 和 read_rnd_next

6. 查询缓存显示没有缓存

因此，这指向导致此问题的查询。 PMM 有一个很好的查询分析来查看导致问题的查询，这就是它所显示的

所以前 2 个查询是罪魁祸首。网上看了很多大家都指出索引是CPU负载最常见的原因，但是这些表都有索引。所以这里有 2 个查询和表定义以及每个查询的索引，解释语句显示它们也使用索引？

查询 1

SELECT
  `tick`,
  VALUE
FROM
  `stored_path_data`
WHERE
  `stored_path_ID` = ?
  AND `tick` <= ?
  AND `tick` >= ?
ORDER BY
  `tick`

mysql> explain stored_path_data;
+----------------+-------------------------------+------+-----+---------+----------------+
| Field          | Type                          | Null | Key | Default | Extra          |
+----------------+-------------------------------+------+-----+---------+----------------+
| ID             | int(11)                       | NO   | PRI | NULL    | auto_increment |
| stored_path_ID | int(11)                       | NO   | MUL | NULL    |                |
| tick           | int(11)                       | NO   | MUL | NULL    |                |
| value          | decimal(18,7)                 | NO   |     | NULL    |                |
| type           | enum('interpolated','manual') | NO   |     | NULL    |                |
+----------------+-------------------------------+------+-----+---------+----------------+
5 rows in set (0.00 sec)
mysql> show indexes from stored_path_data;
+------------------+------------+----------+--------------+----------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table            | Non_unique | Key_name | Seq_in_index | Column_name    | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+------------------+------------+----------+--------------+----------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| stored_path_data |          0 | PRIMARY  |            1 | ID             | A         |      316875 |     NULL | NULL   |      | BTREE      |         |               |
| stored_path_data |          0 | compound |            1 | stored_path_ID | A         |         997 |     NULL | NULL   |      | BTREE      |         |               |
| stored_path_data |          0 | compound |            2 | tick           | A         |      316875 |     NULL | NULL   |      | BTREE      |         |               |
| stored_path_data |          1 | tick     |            1 | tick           | A         |        1771 |     NULL | NULL   |      | BTREE      |         |               |
+------------------+------------+----------+--------------+----------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
mysql> explain SELECT   tick,value FROM stored_path_data WHERE stored_path_ID = 4887   AND `tick` <= 240   AND `tick` >= 0 ORDER BY   `tick`;
+----+-------------+------------------+------------+-------+---------------+------+---------+------+------+----------+------------------------------------+
| id | select_type | table            | partitions | type  | possible_keys | key  | key_len | ref  | rows | filtered | Extra                              |
+----+-------------+------------------+------------+-------+---------------+------+---------+------+------+----------+------------------------------------+
|  1 | SIMPLE      | stored_path_data | NULL       | range | compound,tick | tick | 4       | NULL |    1 |   100.00 | Using index condition; Using where |
+----+-------------+------------------+------------+-------+---------------+------+---------+------+------+----------+------------------------------------+
1 row in set, 1 warning (0.00 sec)

查询 2

SELECT
  `spd`.`stored_path_ID`,
  `spd`.`value`
FROM
  (
    SELECT
      `stored_path_ID`,
      MAX (`tick`) AS `max_tick`
    FROM
      `stored_path_data`
    WHERE
      `stored_path_ID` IN (...)
      AND `tick` <= ?
    GROUP BY
      `stored_path_ID`
  ) AS `temp`
  INNER JOIN `stored_path_data` AS `spd` ON `temp`.`stored_path_ID` = `spd`.`stored_path_ID`
WHERE
  `spd`.`tick` = `temp`.`max_tick`

mysql> explain SELECT   `spd`.`stored_path_ID`,   `spd`.`value` FROM   (     SELECT       `stored_path_ID`,       MAX (`tick`) AS `max_tick`     FROM       `stored_path_data`     WHERE       `stored_path_ID` IN (4883,4884,4885,4886,4887)       AND `tick` <= 240     GROUP BY       `stored_path_ID`   ) AS `temp`   INNER JOIN `stored_path_data` AS `spd` ON `temp`.`stored_path_ID` = `spd`.`stored_path_ID` WHERE   `spd`.`tick` = `temp`.`max_tick`;
+----+-------------+------------------+------------+-------+---------------+-------------+---------+---------------------------------------------------+------+----------+--------------------------+
| id | select_type | table            | partitions | type  | possible_keys | key         | key_len | ref                                               | rows | filtered | Extra                    |
+----+-------------+------------------+------------+-------+---------------+-------------+---------+---------------------------------------------------+------+----------+--------------------------+
|  1 | PRIMARY     | spd              | NULL       | ALL   | compound,tick | NULL        | NULL    | NULL                                              |    1 |   100.00 | NULL                     |
|  1 | PRIMARY     | <derived2>       | NULL       | ref   | <auto_key0>   | <auto_key0> | 9       | tradingsim.spd.stored_path_ID,tradingsim.spd.tick |    2 |   100.00 | Using index              |
|  2 | DERIVED     | stored_path_data | NULL       | index | compound,tick | compound    | 8       | NULL                                              |    1 |   100.00 | Using where; Using index |
+----+-------------+------------------+------------+-------+---------------+-------------+---------+---------------------------------------------------+------+----------+--------------------------+
3 rows in set, 1 warning (0.00 sec)

同一张表，所以索引是一样的。以上还包括对每个查询的解释。

我注意到这些查询有两件事。

1. 查询 1 使用范围，但在刻度和存储路径 ID 上有复合索引
2. 查询 2 正在使用临时表 - 我已尝试在不使用临时表的情况下改进查询，它有所帮助，但 CPU 仍处于 100% 状态

mySQLTuner

然后我运行了 mysqltuner https://github.com/major/MySQLTuner-perl，这是它给出的建议

...
-------- Recommendations ---------------------------------------------------------------------------
General recommendations:
    Add some space to /snap/amazon-ssm-agent/2012 mountpoint.
    Add some space to /snap/core/10126 mountpoint.
    Add some space to /snap/core/10185 mountpoint.
    Cleanup files from /snap/amazon-ssm-agent/2012 mountpoint or reformat you filesystem.
    Cleanup files from /snap/core/10126 mountpoint or reformat you filesystem.
    Cleanup files from /snap/core/10185 mountpoint or reformat you filesystem.
    setup swappiness lower or equals to 10
    setup Max running number events greater than 1M
    Check all table collations are identical for all tables in tradingsim database.
    Limit charset for column to one charset if possible for tradingsim database.
    Limit collations for column to one collation if possible for tradingsim database.
    ALTER TABLE `tradingsim`.`instances` MODIFY `name` CHAR(0) NOT NULL;
    ALTER TABLE `tradingsim`.`instances` MODIFY `date_display_format` CHAR(0);
    ALTER TABLE `tradingsim`.`instruments` MODIFY `instrument_group_ID` CHAR(0);
    ALTER TABLE `tradingsim`.`news` MODIFY `title` TINYTEXT NOT NULL;
    ALTER TABLE `tradingsim`.`news` MODIFY `body` TEXT NOT NULL;
    ALTER TABLE `tradingsim`.`persons` MODIFY `secondname` VARCHAR(10) NOT NULL;
    ALTER TABLE `tradingsim`.`persons` MODIFY `second_email` CHAR(0);
    ALTER TABLE `tradingsim`.`persons` MODIFY `work_phone` CHAR(0) NOT NULL;
    ALTER TABLE `tradingsim`.`persons` MODIFY `mobile_phone` CHAR(0) NOT NULL;
    ALTER TABLE `tradingsim`.`persons` MODIFY `home_phone` CHAR(0) NOT NULL;
    ALTER TABLE `tradingsim`.`persons` MODIFY `username` VARCHAR(15) NOT NULL;
    ALTER TABLE `tradingsim`.`persons` MODIFY `photo_url` CHAR(0) NOT NULL;
    ALTER TABLE `tradingsim`.`persons` MODIFY `email_type` CHAR(0);
    ALTER TABLE `tradingsim`.`persons` MODIFY `fax_number` CHAR(0) NOT NULL;
    ALTER TABLE `tradingsim`.`persons` MODIFY `mts_priority` CHAR(0);
    ALTER TABLE `tradingsim`.`persons` MODIFY `silent_login_group_ID` CHAR(0);
    ALTER TABLE `tradingsim`.`persons` MODIFY `marketing_feedback` CHAR(0);
    ALTER TABLE `tradingsim`.`persons` MODIFY `person_type` CHAR(0);
    ALTER TABLE `tradingsim`.`persons` MODIFY `left_company` CHAR(0);
    ALTER TABLE `tradingsim`.`persons` MODIFY `immutable_ID` CHAR(0);
    ALTER TABLE `tradingsim`.`persons` MODIFY `media_server_ID` CHAR(0);
    ALTER TABLE `tradingsim`.`persons` MODIFY `jobtitle` CHAR(0);
    ALTER TABLE `tradingsim`.`persons` MODIFY `rdr_training_requirements` CHAR(0);
    ALTER TABLE `tradingsim`.`persons` MODIFY `rdr_qualifications_correct` CHAR(0);
    ALTER TABLE `tradingsim`.`persons` MODIFY `rdr_study_qualifications_correct` CHAR(0);
    ALTER TABLE `tradingsim`.`persons` MODIFY `har` CHAR(0);
    ALTER TABLE `tradingsim`.`persons` MODIFY `personal_email` CHAR(0);
    ALTER TABLE `tradingsim`.`stored_path_data` MODIFY `ID` MEDIUMINT(7) UNSIGNED NOT NULL;
    ALTER TABLE `tradingsim`.`stored_path_data` MODIFY `value` DECIMAL(18, 7) NOT NULL;
    ALTER TABLE `tradingsim`.`trader_responses` MODIFY `instance_ID` CHAR(0);
    Remove unused indexes.
    Restrict Host for 'simulations'@% to simulations@SpecificDNSorIp
    UPDATE mysql.user SET host ='SpecificDNSorIp' WHERE user='simulations' AND host ='%'; FLUSH PRIVILEGES;
    MySQL was started within the last 24 hours - recommendations may be inaccurate
    Reduce your overall MySQL memory footprint for system stability
    Configure your accounts with ip or subnets only, then update your configuration with skip-name-resolve=1
    We will suggest raising the 'join_buffer_size' until JOINs not using indexes are found.
             See https://dev.mysql.com/doc/internals/en/join-buffer-size.html
             (specially the conclusions at the bottom of the page).
    Increase table_open_cache gradually to avoid file descriptor limits
    Read this before increasing table_open_cache over 64: 
    Read this before increasing for MariaDB https://mariadb.com/kb/en/library/optimizing-table_open_cache/
    This is MyISAM only table_cache scalability problem, InnoDB not affected.
    See more details here: https://bugs.mysql.com/bug.php?id=49177
    This bug already fixed in MySQL 5.7.9 and newer MySQL versions.
    Beware that open_files_limit (5000) variable
    should be greater than table_open_cache (2000)
    Before changing innodb_log_file_size and/or innodb_log_files_in_group read this: 
Variables to adjust:
    vm.swappiness <= 10 (echo 10 > /proc/sys/vm/swappiness)
    fs.aio-max-nr > 1M (echo 1048576 > /proc/sys/fs/aio-max-nr)
    query_cache_size (=0)
    query_cache_type (=0)
    query_cache_limit (> 1M, or use smaller result sets)
    join_buffer_size (> 256.0K, or always use indexes with JOINs)
    table_open_cache (> 2000)
    innodb_log_file_size should be (=1G) if possible, so InnoDB total log files size equals to 25% of buffer pool size.
    innodb_buffer_pool_instances(=11)

我尝试了这些调整，但仍然没有成功。

我唯一能想到的最后一件事是以下

1. 使用缓存 - memcached 或 redis
2. 将 mySQL 从服务器上移到 RDS 之类的东西上，在那里我可以升级硬件，但那很昂贵

任何人都可以帮助建议我在这种情况下可以做什么，我完全被难住了！我不认为每秒 100 个连接有什么大不了的。我会遇到表锁定问题吗？虽然这就是统计数据向我展示的内容

我们将不胜感激。

编辑

我发现这篇关于最大连接数和使用 mySQL 进行扩展的非常有趣的文章 - https://mysqlserverteam.com/mysql-connection-handling-and-scaling/

如果你到页面底部查看摘要，我认为与我的情况相关的项目是

经验法则：最大连接数 = 可用 CPU 内核的 4 倍

因此，根据我最大 100 个最大连接的使用量，这意味着我应该瞄准具有 25 个 CPU 内核的服务器或重新构建平台。我认为这可能是它的发展方向。我将对这种大小的服务器进行负载测试，看看效果如何。

编辑 2

mysql> SHOW TABLE STATUS WHERE NAME = 'stored_path_data';
+------------------+--------+---------+------------+------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+-------------------+----------+----------------+---------+
| Name             | Engine | Version | Row_format | Rows | Avg_row_length | Data_length | Max_data_length | Index_length | Data_free | Auto_increment | Create_time         | Update_time | Check_time | Collation         | Checksum | Create_options | Comment |
+------------------+--------+---------+------------+------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+-------------------+----------+----------------+---------+
| stored_path_data | InnoDB |      10 | Dynamic    |    0 |              0 |       16384 |               0 |        32768 |   4194304 |        5084417 | 2020-10-29 06:11:01 | NULL        | NULL       | latin1_swedish_ci |     NULL |                |         |
+------------------+--------+---------+------------+------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+-------------------+----------+----------------+---------+
1 row in set (0.00 sec)

结论

如果人们来这里寻找答案（并且不想通读所有 cmets），只是为了帮助人们，@RickJames 提出了解决这个问题的方法。它确实最终成为了索引，但是我不知道存在一种称为覆盖索引的东西，因此创建索引然后运行 ​​ANALYZE TABLE 解决了我的问题。

CREATE INDEX covering ON stored_path_data(stored_path_ID, tick, value);
ANALYZE TABLE stored_path_data;

我尝试了我上面的建议，即增加 CPU 并在 36 个 CPU EC2 实例上运行 90 个并发用户，这完全是矫枉过正，在索引之前，所有 36 个 CPU 都达到了 100%。我会将我的硬件减少到更适合该应用程序的硬件，但再次感谢 @RickJames 的帮助

Answer 1

“你无法摆脱性能问题”

查询 1可能可以通过向 stored_path_data 添加“覆盖”索引来帮助：INDEX(stored_path_ID, tick, value)

如果EXPLAIN 继续使用其他索引，则删除该其他索引。该索引在很多方面都“覆盖”并优化了查询。

查询 2：

    INNER JOIN  `stored_path_data` AS `spd`
        ON `temp`.`stored_path_ID` = `spd`.`stored_path_ID`
    WHERE  `spd`.`tick` = `temp`.`max_tick`

-->

    INNER JOIN  `stored_path_data` AS `spd`
        ON `temp`.`stored_path_ID` = `spd`.`stored_path_ID`
       AND `spd`.`tick` = `temp`.`max_tick`

我不希望有任何区别，但后者表明了您的意图。同时，我添加了一个与“groupwise max”有关的标签。

CHAR(0) 是怎么回事？？？

ALTER TABLE `tradingsim`.`persons` MODIFY `second_email` CHAR(0);

将long_query_time 拖放到1；也许你会在慢日志中得到一些有用的东西。

query_cache_size = 256M 高，低于 50M 或关闭 QC。它往往会占用生产系统的 CPU。 （图表似乎矛盾——活跃度高，但计数为零？？）

我希望您使用的是 InnoDB，而不是 MyISAM。请提供SHOW CREATE TABLE和SHOW TABLE STATUS。

我讨厌没有单位的图表。 “顶级命令计数器”——它们是“每秒”吗？每秒几百个SELECTs 应该不会像您看到的那么严重。 （也许上面的SHOWs 会有所帮助。）“75.69 负载”是什么意思？ “平均负载？似乎是某事的 %？TOTAL 中缺少什么？237 个查询，但列表中少于 20 个？？

好的，我计算出了负载——它是 query_time * query_count。这是一个很好的指标（数字不是很相关）。 53.47 可能意味着服务器有一半的时间在处理第一个查询。

汇总表——我不符合tick 等的语义，但也许构建和维护一个“汇总表”将是有益的。在某些情况下，它可以将查询速度提高 10 倍。

Answer 2

忘记 mysql 调谐器 - 除非您已经知道自己在做什么，否则它可能弊大于利。

对于 QAN 屏幕截图中的前两个查询，您需要在 stored_path_data(stored_path_ID, tick) 上建立索引。这应该会对性能和 CPU 消耗产生巨大影响。

Answer 3

在您的 EC2 参数组中，考虑以下事项

thread_cache_size=100  # since you have average 93 connections
innodb_buffer_pool_instances=8  # to avoid mutex contention with your 11G data
innodb_lru_scan_depth=100  # from 1024 to conserve 90% CPU cycles used for function
innodb_io_capacity=1900  # from 200 or request through a ticket for your SSD data device

祝你好运，请告诉我们你的结果。