如何识别 MySQL 查询的非线性时间增加？答案

【问题标题】：How do I identify non-linear increase in time for MySQL query?如何识别 MySQL 查询的非线性时间增加？
【发布时间】：2021-09-15 20:26:07
【问题描述】：

以下是我正在运行的两个查询。一个需要 75-80 秒，一个需要 1.0-1.5 秒。这两个结果都显示了预期的 50 行 channel_administrators.channel_partner_ids。较快和较慢查询之间的区别在于SELECT 从登录表中选择唯一登录。登录表有 460833 行，我知道这应该会减慢查询速度。我发现这出乎意料的原因是，当在一个channel_administrators.channel_partner_id 上单独运行此代码时，对于最大的channel_administrators.channel_partner_id，结果会在大约 0.2 到 0.7 秒内返回，而对于 50 个结果，我预计它不会超过 50-秒。

我预计时间增加在最坏的情况下是线性的，但时间增加似乎不止于此。这种非线性增加让我觉得我做错了什么（非常？），但我不知道如何找出我的查询出了什么问题。谁能告诉我为什么这个查询会出现非线性时间增加？

我在帖子底部包含了我运行的一些测试查询及其最新时间。

编辑： 我认为这种现象的最佳示例是查看测试 2 和测试 3。这些示例已尽可能地精简，它表明运行逻辑一次运行很快，但 50 次运行非常缓慢。

编辑 2：我添加了更多数据，在 6.93 秒而不是 75 秒内获得了相同的结果。对于我的系统，我认为这是一个可以接受的结果。我现在就写下这个问题的答案。

80 秒查询：

SELECT 
    info.managed_id,
    info.channel_name,
    info.registered_users, 
    info.new_users, 
    info.active_users, 
    info.coupon_opens
    
FROM channel_administrators

LEFT JOIN (    
    SELECT 
        channel_partners.id AS managed_id,
        channel_partners.name as channel_name,
        (
            SELECT COUNT(users.id) 
            FROM users
            WHERE users.channel_partner_id = channel_partners.id
        ) AS registered_users,
        (
            SELECT COUNT(DISTINCT users.id)
            FROM users
            WHERE users.channel_partner_id = channel_partners.id
            AND users.created_at BETWEEN '2021-06-03' AND '2021-07-03' -- 30 days
        ) AS new_users,
        (
            SELECT COUNT(DISTINCT logins.user_id)
            FROM logins
            WHERE logins.channel_partner_id = channel_partners.id
            AND logins.created_at BETWEEN '2021-06-03' AND '2021-07-03' -- 30 days
        ) AS active_users,
        (
            SELECT COUNT(coupon_trackings.id) AS coupon_view_count
            FROM coupon_trackings
            WHERE coupon_trackings.channel_partner_id = channel_partners.id
            AND coupon_trackings.created_at BETWEEN '2021-06-03' AND '2021-07-03' -- 30 days
        ) AS coupon_opens

    FROM channel_partners
) AS info
ON managed_id = channel_administrators.channel_partner_id

WHERE channel_administrators.user_id = 54184

ORDER BY info.channel_name

1.5 秒查询（注释掉差异）：

SELECT 
    info.managed_id,
    info.channel_name,
    info.registered_users, 
    info.new_users, 
--     info.active_users, 
    info.coupon_opens
    
FROM channel_administrators

LEFT JOIN (    
    SELECT 
        channel_partners.id AS managed_id,
        channel_partners.name as channel_name,
        (
            SELECT COUNT(users.id) 
            FROM users
            WHERE users.channel_partner_id = channel_partners.id
        ) AS registered_users,
        (
            SELECT COUNT(DISTINCT users.id)
            FROM users
            WHERE users.channel_partner_id = channel_partners.id
            AND users.created_at BETWEEN '2021-06-03' AND '2021-07-03' -- 30 days
        ) AS new_users,
--         (
--             SELECT COUNT(DISTINCT logins.user_id)
--             FROM logins
--             WHERE logins.channel_partner_id = channel_partners.id
--             AND logins.created_at BETWEEN '2021-06-03' AND '2021-07-03' -- 30 days
--         ) AS active_users,
        (
            SELECT COUNT(coupon_trackings.id) AS coupon_view_count
            FROM coupon_trackings
            WHERE coupon_trackings.channel_partner_id = channel_partners.id
            AND coupon_trackings.created_at BETWEEN '2021-06-03' AND '2021-07-03' -- 30 days
        ) AS coupon_opens

    FROM channel_partners
) AS info
ON managed_id = channel_administrators.channel_partner_id

WHERE channel_administrators.user_id = 54184

ORDER BY info.channel_name

以下是我用来在结果最大的频道上测试各个时间的查询。

测试 1：0.441s - 对于单个最大通道：

SELECT 
    channel_partners.id AS managed_id,
    channel_partners.name as channel_name,
    (
        SELECT COUNT(users.id) 
        FROM users
        WHERE users.channel_partner_id = channel_partners.id
    ) AS registered_users,
    (
        SELECT COUNT(DISTINCT users.id)
        FROM users
        WHERE users.channel_partner_id = channel_partners.id
        AND users.created_at BETWEEN '2021-06-03' AND '2021-07-03' -- 30 days
    ) AS new_users,
    (
        SELECT COUNT(DISTINCT logins.user_id)
        FROM logins
        WHERE logins.channel_partner_id = channel_partners.id
        AND logins.created_at BETWEEN '2021-06-03' AND '2021-07-03' -- 30 days
    ) AS active_users,
    (
        SELECT COUNT(coupon_trackings.id) AS coupon_view_count
        FROM coupon_trackings
        WHERE coupon_trackings.channel_partner_id = channel_partners.id
        AND coupon_trackings.created_at BETWEEN '2021-06-03' AND '2021-07-03' -- 30 days
    ) AS coupon_opens

FROM channel_partners

WHERE channel_partners.id = 3255770

测试 2：0.368 秒 - 最大渠道的活跃用户：

SELECT COUNT(DISTINCT logins.user_id)
FROM logins
WHERE logins.channel_partner_id = 3255770
AND logins.created_at BETWEEN '2021-06-03' AND '2021-07-03' -- 30 days

测试3：75.2s 只需登录信息

SELECT 
    info.managed_id,
    info.channel_name,
    info.active_users    
FROM channel_administrators

LEFT JOIN (    
    SELECT 
        channel_partners.id AS managed_id,
        channel_partners.name as channel_name,
        (
            SELECT COUNT(DISTINCT logins.user_id)
            FROM logins
            WHERE logins.channel_partner_id = channel_partners.id
            AND logins.created_at BETWEEN '2021-06-03' AND '2021-07-03' -- 30 days
        ) AS active_users

    FROM channel_partners
) AS info
ON info.managed_id = channel_administrators.channel_partner_id

WHERE channel_administrators.user_id = 54184

测试 4：6.93 秒 - 通过重写取得进展

SELECT 
    channel_partners.id AS managed_id,
    channel_partners.name as channel_name,
    (
        SELECT COUNT(users.id) 
        FROM users
        WHERE users.channel_partner_id = channel_partners.id
    ) AS registered_users,
    (
        SELECT COUNT(DISTINCT users.id)
        FROM users
        WHERE users.channel_partner_id = channel_partners.id
        AND users.created_at BETWEEN '2021-06-03' AND '2021-07-03' -- 30 days
    ) AS new_users,
    (
        SELECT COUNT(DISTINCT logins.user_id)
        FROM logins
        WHERE logins.channel_partner_id = channel_partners.id
        AND logins.created_at BETWEEN '2021-06-03' AND '2021-07-03' -- 30 days
    ) AS active_users,
    (
        SELECT COUNT(coupon_trackings.id) AS coupon_view_count
        FROM coupon_trackings
        WHERE coupon_trackings.channel_partner_id = channel_partners.id
        AND coupon_trackings.created_at BETWEEN '2021-06-03' AND '2021-07-03' -- 30 days
    ) AS coupon_opens

FROM channel_partners

WHERE (channel_partners.id IN (SELECT 
        channel_administrators.channel_partner_id
    FROM channel_administrators
    WHERE channel_administrators.user_id = 54184
    )
)

编辑：从查询中添加结果（删除名称）

80年代查询：

|managed_id|registered_users|new_users|active_users|coupon_opens|
|----------|----------------|---------|------------|------------|
|14        |1146            |46       |282         |893         |
|27        |2159            |48       |206         |635         |
|15        |2039            |68       |490         |2560        |
|16        |15              |0        |1           |0           |
|20        |1391            |53       |413         |1614        |
|21        |3               |0        |0           |0           |
|43        |1051            |36       |255         |1234        |
|44        |706             |19       |85          |276         |
|46        |16              |0        |4           |8           |
|47        |68              |1        |5           |30          |
|48        |169             |6        |40          |308         |
|49        |408             |13       |118         |434         |
|52        |52              |1        |11          |54          |
|53        |378             |11       |111         |391         |
|54        |34              |1        |5           |57          |
|75        |576             |7        |59          |145         |
|3255347   |773             |12       |99          |167         |
|685131    |142             |0        |9           |91          |
|76        |22              |0        |9           |25          |
|55        |276             |5        |68          |251         |
|56        |2232            |79       |534         |1644        |
|57        |78              |0        |10          |47          |
|58        |708             |10       |109         |364         |
|59        |1274            |42       |465         |1929        |
|60        |133             |0        |37          |97          |
|3         |0               |0        |127         |257         |
|2144749   |0               |0        |4           |40          |
|61        |629             |9        |119         |363         |
|63        |857             |36       |267         |892         |
|64        |49              |1        |13          |21          |
|65        |723             |15       |281         |1152        |
|66        |77              |0        |17          |48          |
|67        |123             |10       |59          |190         |
|68        |693             |8        |191         |387         |
|70        |80              |0        |31          |58          |
|71        |214             |1        |41          |102         |
|72        |104             |2        |23          |49          |
|3255770   |3149            |86       |542         |2280        |
|3255771   |3012            |39       |526         |2056        |
|77        |180             |9        |89          |239         |
|477       |677             |5        |286         |583         |
|478       |335             |191      |235         |2226        |
|479       |162             |12       |51          |159         |
|480       |57              |0        |8           |12          |
|302       |51              |3        |17          |32          |
|303       |213             |37       |116         |598         |
|373109    |9               |3        |6           |4           |
|373110    |10              |2        |5           |0           |
|373111    |29              |9        |16          |29          |
|3255810   |0               |0        |0           |0           |

2s查询：

|managed_id|registered_users|new_users|coupon_opens|
|----------|----------------|---------|------------|
|14        |1146            |46       |893         |
|27        |2159            |48       |635         |
|15        |2039            |68       |2560        |
|16        |15              |0        |0           |
|20        |1391            |53       |1614        |
|21        |3               |0        |0           |
|43        |1051            |36       |1234        |
|44        |706             |19       |276         |
|46        |16              |0        |8           |
|47        |68              |1        |30          |
|48        |169             |6        |308         |
|49        |408             |13       |434         |
|52        |52              |1        |54          |
|53        |378             |11       |391         |
|54        |34              |1        |57          |
|75        |576             |7        |145         |
|3255347   |773             |12       |167         |
|685131    |142             |0        |91          |
|76        |22              |0        |25          |
|55        |276             |5        |251         |
|56        |2232            |79       |1644        |
|57        |78              |0        |47          |
|58        |708             |10       |364         |
|59        |1274            |42       |1929        |
|60        |133             |0        |97          |
|3         |0               |0        |257         |
|2144749   |0               |0        |40          |
|61        |629             |9        |363         |
|63        |857             |36       |892         |
|64        |49              |1        |21          |
|65        |723             |15       |1152        |
|66        |77              |0        |48          |
|67        |123             |10       |190         |
|68        |693             |8        |387         |
|70        |80              |0        |58          |
|71        |214             |1        |102         |
|72        |104             |2        |49          |
|3255770   |3149            |86       |2280        |
|3255771   |3012            |39       |2056        |
|77        |180             |9        |239         |
|477       |677             |5        |583         |
|478       |335             |191      |2226        |
|479       |162             |12       |159         |
|480       |57              |0        |12          |
|302       |51              |3        |32          |
|303       |213             |37       |598         |
|373109    |9               |3        |4           |
|373110    |10              |2        |0           |
|373111    |29              |9        |29          |
|3255810   |0               |0        |0           |

【问题讨论】：

请包含两个查询的解释结果，否则很难看出哪里出了问题！
我已经添加了降价表。
我在帖子中间添加了测试。测试 2 和测试 3 真正展示了剥离问题。
我添加了“测试 4”，将登录结果降至 6.93 秒。我认为这可能会解决我的问题，但需要测试。

标签： mysql sql performance query-optimization

【解决方案1】：

每个表都需要此复合索引，其中的列按给定顺序排列：

INDEX(channel_partner_id, created_at)

如果有的话，只需在 channel_partner_id 上删除相应的索引。

将此“covering”索引添加到channel_administrators

INDEX(user_id, channel_partner_id)

如果存在INDEX(user_id)，则删除它。

通过没有嵌套的 Select 来简化查询：

    SELECT  cp.id AS managed_id,
            cp.name as channel_name,
            (   SELECT  COUNT(users.id)
                    FROM  users
                    WHERE  users.channel_partner_id = cp.id 
            ) AS registered_users,
            ((etc))
        FROM  channel_partners AS cp
        JOIN  channel_administrators AS ca
                ON cp.managed_id = ca.channel_partner_id
        WHERE  ca.user_id = 54184
        ORDER  BY  channel_name

提示：考虑改变

created_at BETWEEN '2021-06-03' AND '2021-07-03' -- 30 days

到

    created_at >= '2021-06-03'
AND created_at  < '2021-06-03' + INTERVAL 30 DAY

如果更合适，也可以使用+ INTERVAL 1 MONTH。

如果您仍有性能问题，让我们看看查询是什么样的并提供SHOW CREATE TABLE。

【讨论】：

您好，抱歉。我昨天生病了，没有检查这个。我知道日期选择是这样做的不好的方法。我实际上打算从网页上的表单中获取日期字符串，所以目前完成日期的方式是 WIP。谢谢你的建议。我现在将尝试实施您的解决方案。再次感谢您。

【解决方案2】：

我仍然不知道为什么它运行缓慢，但我找到了一个提高速度的解决方案，并将显示我如何到达那里的步骤。

首先，在原始问题的测试 2 和测试 3 中，您可以看到对 logins 表的单个查询非常快，但是当查询多个 channel_partner_id 时，整个查询变得非常慢。我怀疑这与我检查 channel_partner_id 与登录表的方式有关。

我重写了查询以从列表而不是从选择中获取channel_partner_id。最终结果如下所示：

SELECT 
    channel_partners.id AS managed_id,
    channel_partners.name as channel_name,
    (
        SELECT COUNT(users.id) 
        FROM users
        WHERE users.channel_partner_id = channel_partners.id
    ) AS registered_users,
    (
        SELECT COUNT(DISTINCT users.id)
        FROM users
        WHERE users.channel_partner_id = channel_partners.id
        AND users.created_at BETWEEN '2021-06-03' AND '2021-07-03' -- 30 days
    ) AS new_users,
    (
        SELECT COUNT(DISTINCT logins.user_id)
        FROM logins
        WHERE logins.channel_partner_id = channel_partners.id
        AND logins.created_at BETWEEN '2021-06-03' AND '2021-07-03' -- 30 days
    ) AS active_users,
    (
        SELECT COUNT(coupon_trackings.id) AS coupon_view_count
        FROM coupon_trackings
        WHERE coupon_trackings.channel_partner_id = channel_partners.id
        AND coupon_trackings.created_at BETWEEN '2021-06-03' AND '2021-07-03' -- 30 days
    ) AS coupon_opens

FROM channel_partners

WHERE (channel_partners.id IN (SELECT 
        channel_administrators.channel_partner_id
    FROM channel_administrators
    WHERE channel_administrators.user_id = 54184
    )
)

这个查询花费了 6.93 秒，这仍然很慢，但它更接近我预期的查询所花费的时间。

我无法解释为什么一种方式更快而另一种方式更慢。

【讨论】：

我还是个堆栈溢出的新手，我是否将自己的答案标记为已解决？