Mysql 进程超过 3500% 的 CPU 使用率和 Qcache_lowmem_prunes 太高

Question

昨晚在线用户急剧增加后，我遇到了mysql CPU使用率的问题。平均服务器负载为 150（1 分钟），顶部的 mysql CPU 使用率为 3500-4000%。演出流程清单；结果是 250 个查询。以下是一些设置和状态：

max_connections=400
query_cache_type=1
join_buffer_size=4096
query_cache_size=5M
query_cache_limit=1048576
query_cache_min_res_unit=4096

这里是执行的

show status like '%qcach%';

30 秒内 3 次

+-------------------------+------------+
| Variable_name           | Value      |
+-------------------------+------------+
| Qcache_free_blocks      | 597        |
| Qcache_free_memory      | 1658184    |
| Qcache_hits             | 1813237610 |
| Qcache_inserts          | 456473072  |
| Qcache_lowmem_prunes    | 433310255  |
| Qcache_not_cached       | 105476947  |
| Qcache_queries_in_cache | 1564       |
| Qcache_total_blocks     | 4027       |
+-------------------------+------------+
+-------------------------+------------+
| Variable_name           | Value      |
+-------------------------+------------+
| Qcache_free_blocks      | 429        |
| Qcache_free_memory      | 1422288    |
| Qcache_hits             | 1813245095 |
| Qcache_inserts          | 456475419  |
| Qcache_lowmem_prunes    | 433312602  |
| Qcache_not_cached       | 105477523  |
| Qcache_queries_in_cache | 1553       |
| Qcache_total_blocks     | 3896       |
+-------------------------+------------+
+-------------------------+------------+
| Variable_name           | Value      |
+-------------------------+------------+
| Qcache_free_blocks      | 481        |
| Qcache_free_memory      | 1546088    |
| Qcache_hits             | 1813254695 |
| Qcache_inserts          | 456477824  |
| Qcache_lowmem_prunes    | 433314721  |
| Qcache_not_cached       | 105478263  |
| Qcache_queries_in_cache | 1699       |
| Qcache_total_blocks     | 4146       |
+-------------------------+------------+

服务器配置：

2 x Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz.
64GB RAM
Mysql running on SSDs.

Answer 1

我已将 query_cache_size 更改为 128M。 mysql procces 使用的 CPU 比以前少得多。

2分钟后刷新状态结果：

| Qcache_free_blocks      | 15777    |
| Qcache_free_memory      | 66772864 |
| Qcache_hits             | 74013    |
| Qcache_inserts          | 5293     |
| Qcache_lowmem_prunes    | 0        |
| Qcache_not_cached       | 5125     |
| Qcache_queries_in_cache | 37880    |
| Qcache_total_blocks     | 92907 

但是，5-10 分钟后 Qcache_lowmem_prunes 再次开始增加（可能比以前慢，但仍以每秒 50-100 个点的速度增加）。

我认为问题不再是 query_cache_size 了。

冲洗后 7 分钟：

| Qcache_free_blocks      | 14825    |
| Qcache_free_memory      | 54665504 |
| Qcache_hits             | 534784   |
| Qcache_inserts          | 44569    |
| Qcache_lowmem_prunes    | 6194     |
| Qcache_not_cached       | 34718    |
| Qcache_queries_in_cache | 49212    |
| Qcache_total_blocks     | 114243   |
+-------------------------+----------+

AVG Query Size = 1616
Frag Indicator = 12.98%
Utilisation = 59.27%
Removal Ratio = 13.9%
Hit Rate = 87.09%

之后，我将 query_cache_min_res_unit 更改为 2048。 冲洗状态后 10 分钟：

| Qcache_free_blocks      | 21987    |
| Qcache_free_memory      | 38858856 |
| Qcache_hits             | 255849   |
| Qcache_inserts          | 20100    |
| Qcache_lowmem_prunes    | 15638    |
| Qcache_not_cached       | 16691    |
| Qcache_queries_in_cache | 64573    |
| Qcache_total_blocks     | 152132   |

AVG Query Size = 1477
Frag Indicator = 14.45%
Utilisation = 71%
Removal Ratio = 79%
Hit Rate = 87%

Answer 2

Wilson Hauck 感谢您的回复。 正如您所写，我仅将 query_cache_min_res_unit 更改为 512。 FLUSH STATUS 后 10 小时：

| Qcache_free_blocks      | 65446    |
| Qcache_free_memory      | 53083696 |
| Qcache_hits             | 39641904 |
| Qcache_inserts          | 2559829  |
| Qcache_lowmem_prunes    | 240861   |
| Qcache_not_cached       | 2564542  |
| Qcache_queries_in_cache | 155462   |
| Qcache_total_blocks     | 385780   |

AVG Query Size = 1385
Frag Indicator = 16.96%
Utilisation = 80%
Removal Ratio = 9.4%
Hit Rate = 88.56%

如您所见，去除率现在低于 10%。我认为这很好。我真的需要更改其他变量吗？

Answer 3

对您的 my.cnf/ini [mysqld] 部分的建议

query_prealloc_size=32K  # from default to avoid RAM acquisition all day
query_alloc_block_size=32K  # from default to manage larger blocks
query_cache_limit=16K  # from 1M to minimize qcache_lowmem_prunes
query_cache_min_res_unit=512  # from ? to minimum for more queries_in_cache

希望在实施后的一天能看到一些反馈和类似的统计数据。

Answer 4

建议您的 my.cnf-ini [mysqld] 部分考虑

innodb_buffer_pool_size=10G  # from 128M to reduce innodb_buffer_pool_reads
innodb_lru_scan_depth=128  # from 1024 per REFMAN when ibp_instances=8
expire_logs_days=5  # from 0 for limited historical error,general,slow logs
slow_query_log=ON  # from OFF to allow awareness of ~ 2000 in 76 days, for improvement of queries
thread_cache_size=100  # from 12 to reduce threads_created 353K
thread_concurrency=30  # from 10 for more concurrency
max_connections=200  # from 400 for max_used_connections 56 x ~ 4
innodb_io_capacity=1000  # from 200 to open the door wider
key_age_threshold=64800  # from 300 second discard to RD again
innodb_change_buffer_max_size=10  # from 25% to reduce set aside for CHGS, etc
max_write_lock_count=16  # from HUGE number to allow RD after nn lcks
join_buffer_size=128k  # from 4096 for join capacity increase
sort_buffer_size=2M  # from 256K to reduce sort_merge_passes ~ 1m
updatable_views_with_limit=NO  # from YES to reduce handler_external_lock cnt

handler_rollback 有 3,412,265 个事件需要处理。需要 1 分钟的 SET GLOBAL innodb_print_all_deadlocks=ON; - 然后关闭以避免填充驱动器。请发布错误日志的适用部分以供分析。