与 MySQL 相比，MariaDB 上的 OUTER JOIN 真的很慢答案

【问题标题】：Really slow OUTER JOIN on MariaDB compared to MySQL与 MySQL 相比，MariaDB 上的 OUTER JOIN 真的很慢
【发布时间】：2021-10-28 07:19:23
【问题描述】：

上下文

考虑这个包含来自特定用户的朋友的表。

CREATE TABLE `ce_mutual_friends`
(
    `user_id` mediumint unsigned NOT NULL,
    `friend_id` mediumint unsigned NOT NULL,
    PRIMARY KEY (`user_id`, `friend_id`),
    INDEX `friend_id_user_id` (`friend_id`, `user_id`)
   
)ENGINE=InnoDB

请注意，该列表是相互的。如果用户 1 与朋友 2 成为朋友，则这自动意味着用户 2 与朋友 1 成为朋友。此表中有 10K 行。

目标

我想要一个用户列表：

直接朋友（relation_degree = 1）
朋友的朋友（relation_degree = 2）
朋友的朋友的朋友（relation_degree = 3）

我通过执行多个 LEFT OUTER JOIN 来做到这一点：

 -- for total network
WITH RECURSIVE cte(level, user_id, friend_id) AS(
  SELECT 1, user_id, friend_id
    FROM ce_mutual_friends
  UNION
  SELECT cte.level+1, cte.user_id, t2.friend_id
    FROM cte
      INNER JOIN ce_mutual_friends t2
        ON t2.user_id = cte.friend_id
    WHERE cte.level+1 <= 3 
)
SELECT cte.user_id, cte.friend_id, min(cte.level)
  FROM cte
  GROUP BY cte.user_id, cte.friend_id;

此查询每天运行数次以执行一些后台任务。

问题

我正在考虑从 MySQL 迁移到 MariaDB。在我的开发环境中，我使用这个 docker-compose.yml 来启动 2 个容器。

version: '3'
services:
  mysql:
    container_name: mysql_db
    restart: always
    image: mysql:latest
    ports: 
        - "3306:3306" 
    environment:
       - MYSQL_ROOT_PASSWORD=secret
    volumes:
      - ./mysql-data:/var/lib/mysql
  mariadb:
    container_name: mariadb
    restart: always
    image: mariadb:latest
    ports: 
        - "3305:3306" 
    environment:
       - MYSQL_ROOT_PASSWORD=secret
    volumes:
      - ./mariadb-data:/var/lib/mysql

我将我的数据库从 MySQL 导出和导入到 MariaDB。我希望 MariaDB 上的查询与 MySQL 上的查询相同或稍快，但令我惊讶的是，MariaDB 的查询要差得多。

MySQL 1.25 秒
MariaDB 26 秒

如果我将查询修改为仅获取一个特定用户（此查询实际上在我们的应用程序中经常运行），那么持续时间是：

MySQL 0.002 秒
MariaDB 0.015 秒

这里是具体的用户调整查询：

 -- for user 1
WITH RECURSIVE cte(level, user_id, friend_id) AS(
  SELECT 1, user_id, friend_id
    FROM ce_mutual_friends
    WHERE user_id = 1
  UNION
  SELECT cte.level+1, cte.user_id, t2.friend_id
    FROM cte
      INNER JOIN ce_mutual_friends t2
        ON t2.user_id = cte.friend_id
    WHERE cte.level+1 <= 3 
)
SELECT cte.user_id, cte.friend_id, min(cte.level)
  FROM cte
  GROUP BY cte.user_id, cte.friend_id;

在 MySQL 上解释选择


+------+----------------+-------------------+------------+-------+---------------+---------+---------+---------------+--------+----------+--------------------------+-------------------+
|  id  |  select_type   |       table       | partitions | type  | possible_keys |   key   | key_len |      ref      |  rows  | filtered |          Extra           |                   |
+------+----------------+-------------------+------------+-------+---------------+---------+---------+---------------+--------+----------+--------------------------+-------------------+
| 1    | PRIMARY        | <derived2>        | NULL       | ALL   | NULL          | NULL    | NULL    | NULL          | 131532 | 100.00   | "Using temporary"        |                   |
| 2    | DERIVED        | ce_mutual_friends | NULL       | index | NULL          | PRIMARY | 6       | NULL          | 10506  | 100.00   | "Using index"            |                   |
| 3    | UNION          | cte               | NULL       | ALL   | NULL          | NULL    | NULL    | NULL          | 10506  | 100.00   | "Recursive; Using where" |                   |
| 3    | UNION          | t2                | NULL       | ref   | PRIMARY       | PRIMARY | 3       | cte.friend_id | 11     | 100.00   | "Using index"            |                   |
| NULL | "UNION RESULT" | "<union2          | 3>"        | NULL  | ALL           | NULL    | NULL    | NULL          | NULL   | NULL     | NULL                     | "Using temporary" |
+------+----------------+-------------------+------------+-------+---------------+---------+---------+---------------+--------+----------+--------------------------+-------------------+


+-------------------------------------+----------------+
|            Variable_name            |     Value      |
+-------------------------------------+----------------+
| bulk_insert_buffer_size             | 8388608        |
| innodb_buffer_pool_chunk_size       | 134217728      |
| innodb_buffer_pool_dump_at_shutdown | ON             |
| innodb_buffer_pool_dump_now         | OFF            |
| innodb_buffer_pool_dump_pct         | 25             |
| innodb_buffer_pool_filename         | ib_buffer_pool |
| innodb_buffer_pool_in_core_file     | ON             |
| innodb_buffer_pool_instances        | 1              |
| innodb_buffer_pool_load_abort       | OFF            |
| innodb_buffer_pool_load_at_startup  | ON             |
| innodb_buffer_pool_load_now         | OFF            |
| innodb_buffer_pool_size             | 134217728      |
| innodb_change_buffer_max_size       | 25             |
| innodb_change_buffering             | all            |
| innodb_log_buffer_size              | 16777216       |
| innodb_sort_buffer_size             | 1048576        |
| join_buffer_size                    | 262144         |
| key_buffer_size                     | 8388608        |
| myisam_sort_buffer_size             | 8388608        |
| net_buffer_length                   | 16384          |
| preload_buffer_size                 | 32768          |
| read_buffer_size                    | 131072         |
| read_rnd_buffer_size                | 262144         |
| select_into_buffer_size             | 131072         |
| sort_buffer_size                    | 262144         |
| sql_buffer_result                   | OFF            |
+-------------------------------------+----------------+

EXPLAIN SELECT & Buffer varsON MariaBD


+------+-------------------+-------------------+-------+---------------+---------+---------+-------------------------+-------+-----------------------------------+--+
|  id  |    select_type    |       table       | type  | possible_keys |   key   | key_len |           ref           | rows  |               Extra               |  |
+------+-------------------+-------------------+-------+---------------+---------+---------+-------------------------+-------+-----------------------------------+--+
| 1    | PRIMARY           | <derived2>        | ALL   | NULL          | NULL    | NULL    | NULL                    | 10506 | "Using temporary; Using filesort" |  |
| 2    | DERIVED           | ce_mutual_friends | index | NULL          | PRIMARY | 6       | NULL                    | 10506 | "Using index"                     |  |
| 3    | "RECURSIVE UNION" | t2                | index | PRIMARY       | PRIMARY | 6       | NULL                    | 10506 | "Using index"                     |  |
| 3    | "RECURSIVE UNION" | <derived2>        | ref   | key0          | key0    | 4       | mysite_prod_.t2.user_id | 10    | "Using where"                     |  |
| NULL | "UNION RESULT"    | "<union2          | 3>"   | ALL           | NULL    | NULL    | NULL                    | NULL  | NULL                              |  |
+------+-------------------+-------------------+-------+---------------+---------+---------+-------------------------+-------+-----------------------------------+--+


+-------------------------------------+----------------+
|            Variable_name            |     Value      |
+-------------------------------------+----------------+
| aria_pagecache_buffer_size          | 134217728      |
| aria_sort_buffer_size               | 268434432      |
| bulk_insert_buffer_size             | 8388608        |
| innodb_buffer_pool_chunk_size       | 134217728      |
| innodb_buffer_pool_dump_at_shutdown | ON             |
| innodb_buffer_pool_dump_now         | OFF            |
| innodb_buffer_pool_dump_pct         | 25             |
| innodb_buffer_pool_filename         | ib_buffer_pool |
| innodb_buffer_pool_load_abort       | OFF            |
| innodb_buffer_pool_load_at_startup  | ON             |
| innodb_buffer_pool_load_now         | OFF            |
| innodb_buffer_pool_size             | 2147483648     |
| innodb_change_buffer_max_size       | 25             |
| innodb_change_buffering             | all            |
| innodb_log_buffer_size              | 16777216       |
| innodb_sort_buffer_size             | 1048576        |
| join_buffer_size                    | 262144         |
| join_buffer_space_limit             | 2097152        |
| key_buffer_size                     | 134217728      |
| mrr_buffer_size                     | 262144         |
| myisam_sort_buffer_size             | 134216704      |
| net_buffer_length                   | 16384          |
| preload_buffer_size                 | 32768          |
| read_buffer_size                    | 131072         |
| read_rnd_buffer_size                | 262144         |
| sort_buffer_size                    | 2097152        |
| sql_buffer_result                   | OFF            |
+-------------------------------------+----------------+

我已经在 MySQL Workbench 上测试了这些查询。据我了解，这不应该有任何区别。但如果我错了，请纠正我。
为什么 MariaDB 比 MySQL 慢这么多？使用 docker stats 和 htop 我发现在查询期间没有 RAM 短缺和 1 个 cpu 核心峰值。

【问题讨论】：

我很好奇您是否会看到使用INNER JOIN 而不是OUTER JOIN 的性能差异。无论如何，连接实际上是内部连接，因此这不会影响结果。
两个数据库都应该运行得很好。尝试分析每个数据库上的查询。 dev.mysql.com/doc/refman/8.0/en/analyze-table.html
在我看来，您的 MariaDB 容器内存不足或出现其他故障。它挂起 30 秒并断开您的连接。那不是“慢”，那是非常不正常的。如果它只是很慢，它不会断开连接。
遵循规则 here 为多：多表建立索引。
@GordonLinoff - INNER 和 OUTER 被忽略。

标签： mysql mariadb outer-join

【解决方案1】：

在 MySQL >= 8.0.14 中，Recursive Common Table Expression 的性能应该更高：

WITH RECURSIVE cte(level, uid) AS(
  SELECT 1, friend_id
    FROM ce_mutual_friends
    WHERE user_id = 1
  UNION
  SELECT cte.level+1, friend_id
    FROM cte
      INNER JOIN ce_mutual_friends
        ON user_id = cte.uid
    WHERE cte.level+1 <= 3 
)
SELECT cte.uid, min(cte.level)
  FROM cte
  GROUP BY cte.uid;

【讨论】：

这同样适用于 MariaDB-10.2+。
对于用户特定的查询，这大约需要 0.015 秒，我在 MariaDB 上是 0.045 秒！我今天会尝试更多地研究它，并为整个维护查询制作一个。
实际...仔细查找后发现结果不一样。我必须将 user_id = 1 放在 CTE 之外的 select 语句中。这实际上更慢（> 1s）。对于整个网络来说，它仍然更快。但在 Maria 上仍然表现不佳。