MySQL第一个/最后一个值和使用前一个值的计算答案

【问题标题】：MySQL first / last value and calculations with previous valueMySQL第一个/最后一个值和使用前一个值的计算
【发布时间】：2020-03-26 08:28:14
【问题描述】：

我有一个巨大的表格，每分钟都有数据行。某些列的计算差异为 [间隔的最后一个值] - [之前的最后一个间隔的值]。但是有没有办法加快查询速度？已经有主索引和日期索引。不应创建任何其他索引。目前，查询大约 500.000 行运行大约 11 秒。

问题出在 DATE_SUB() 函数的计算附近。

MySQL：10.1.44-MariaDB-0ubuntu0.18.04.1

PHP：7.2.24-0ubuntu0.18.04.3

*id* || *select_type*      || *table*   || *type* || *possible_keys* || *key* || *key_len* || *ref* || *rows* || *filtered* || *Extra*
1    || PRIMARY            || a10       || ALL    || date            ||       ||           ||       || 513754 || 21.86      || Using where; Using temporary; Using filesort
4    || DEPENDENT SUBQUERY || table_min || index  || date            || date  || 5         ||       || 1      || 100.00     || Using where
3    || DEPENDENT SUBQUERY || table_min || index  || date            || date  || 5         ||       || 1      || 100.00     || Using where
2    || DEPENDENT SUBQUERY || table_min || index  || date            || date  || 5         ||       || 1      || 100.00     || Using where

查询：

EXPLAIN EXTENDED 
SELECT  SQL_CALC_FOUND_ROWS tm.date,
CAST(SUBSTRING_INDEX(GROUP_CONCAT(tm.DB223_DBD10_float ORDER BY  tm.date DESC),
                ",", 1) AS double
    ) AS `10-44-2`,
                CAST(SUBSTRING_INDEX(GROUP_CONCAT(tm.DB223_DBD14_float ORDER BY  tm.date DESC),
                ",", 1) AS double) AS `10-45-3`, MIN(tm.DB221_DBD218_float
                    ) AS `10-2-4`,
                MAX(tm.DB221_DBD218_float) AS `10-2-5`, MIN(tm.DB221_DBD222_float) AS `10-3-6`,
MAX(tm.DB221_DBD222_float) AS `10-3-7`, MIN(tm.DB221_DBD278_float) AS `10-4-8`,
MAX(tm.DB221_DBD278_float) AS `10-4-9`, (CAST(SUBSTRING_INDEX(GROUP_CONCAT(tm.DB222_DBD10_integer
                                    ORDER BY  tm.date DESC), ",", 1) AS double) - 
            (
                SELECT  DB222_DBD10_integer
                    FROM  table_min
                    WHERE  date <= DATE_SUB(tm.date, INTERVAL 1 DAY)
                    ORDER BY  date DESC
                LIMIT  1)
   ) AS `10-18-10`,
                    (CAST(SUBSTRING_INDEX(GROUP_CONCAT(tm.DB222_DBD46_integer
       ORDER BY  tm.date DESC),
                        ",", 1) AS double) - (
                SELECT  DB222_DBD46_integer
                    FROM  table_min
                    WHERE  date <= DATE_SUB(tm.date, INTERVAL 1 DAY)
                    ORDER BY  date DESC
                LIMIT  1)
                    ) AS `10-36-11`,
                    (CAST(SUBSTRING_INDEX(GROUP_CONCAT(tm.DB222_DBD50_integer
       ORDER BY  tm.date DESC),
                        ",", 1) AS double
                          ) - (
                SELECT  DB222_DBD50_integer
                    FROM  table_min
                    WHERE  date <= DATE_SUB(tm.date, INTERVAL 1 DAY)
                    ORDER BY  date DESC
                LIMIT  1)
                    ) AS `10-37-12`,
                CAST(SUBSTRING_INDEX(GROUP_CONCAT(tm.DB223_DBD124_float
       ORDER BY  tm.date DESC),
                ",", 1) AS double
                    ) AS `10-47-13`,
                CAST(SUBSTRING_INDEX(GROUP_CONCAT(tm.DB223_DBD120_float
       ORDER BY  tm.date DESC),
                ",", 1) AS double
                    ) AS `10-46-14`,
                CAST(SUBSTRING_INDEX(GROUP_CONCAT(tm.DB223_DBD128_float
       ORDER BY  tm.date DESC),
                ",", 1) AS double
                    ) AS `10-48-15`,
                CAST(SUBSTRING_INDEX(GROUP_CONCAT(tm.DB223_DBD132_float
       ORDER BY  tm.date DESC),
                ",", 1) AS double
                    ) AS `10-49-17`,
                CAST(SUBSTRING_INDEX(GROUP_CONCAT(tm.DB223_DBD136_float
       ORDER BY  tm.date DESC),
                ",", 1) AS double
                    ) AS `10-50-18`,
                CAST(SUBSTRING_INDEX(GROUP_CONCAT(tm.DB223_DBD140_float
       ORDER BY  tm.date DESC),
                ",", 1) AS double
                    ) AS `10-51-19`,
                CAST(SUBSTRING_INDEX(GROUP_CONCAT(tm.DB223_DBD144_float
       ORDER BY  tm.date DESC),
                ",", 1) AS double
                    ) AS `10-52-21`,
                CAST(SUBSTRING_INDEX(GROUP_CONCAT(tm.DB223_DBD148_float
       ORDER BY  tm.date DESC),
                ",", 1) AS double
                    ) AS `10-53-22`,
                CAST(SUBSTRING_INDEX(GROUP_CONCAT(tm.DB223_DBD310_float
       ORDER BY  tm.date DESC),
                ",", 1) AS double
                    ) AS `10-54-24`,
                CAST(SUBSTRING_INDEX(GROUP_CONCAT(tm.DB223_DBD314_float
       ORDER BY  tm.date DESC),
                ",", 1) AS double
                    ) AS `10-55-25`,
                    (CAST(SUBSTRING_INDEX(GROUP_CONCAT(tm.DB223_DBD310_float
       ORDER BY  tm.date DESC),
                        ",",
                                                ",", 1) AS double),NULL)) AS `10-0-26`,
                        1) AS double)/NULLIF(CAST(SUBSTRING_INDEX(GROUP_CONCAT(tm.DB223_DBD314_float
       ORDER BY  tm.date DESC),
CAST(SUBSTRING_INDEX(GROUP_CONCAT(tm.DB221_DBD538_float
       ORDER BY  tm.date DESC),
                ",", 1) AS double) AS `10-31-27`,
                MIN(tm.DB221_DBD326_float) AS `10-9-28`,
                MAX(tm.DB221_DBD326_float) AS `10-9-29`,
                MIN(tm.DB221_DBD450_float) AS `10-29-30`,
                MAX(tm.DB221_DBD450_float) AS `10-29-31`,
                MIN(tm.DB221_DBD406_float) AS `10-27-32`,
                MAX(tm.DB221_DBD406_float) AS `10-27-33`,
                MIN(tm.DB221_DBD562_float) AS `10-41-34`,
                MAX(tm.DB221_DBD562_float) AS `10-41-35`
    FROM  table_min AS tm WHERE  tm.date>="2020-01-01 00:00"
      AND  tm.date<="2020-01-31 23:59:59"
    GROUP BY  YEAR(tm.date), MONTH(tm.date), DAY(tm.date)
    ORDER BY  tm.date
    LIMIT  0,20

从今年开始的 SQL 数据转储： https://www.tds-net.de/table_min.zip

【问题讨论】：

指定 MySQL 版本。
抱歉，已添加到问题文本中。
请向您的问题添加数据，并从提供的数据中获得想要的结果。如果您有兴趣自己解决，请参阅mariadb.com/kb/en/window-functions-overview
SQL 转储已添加为 ZIP 文件（~120.000 行）。

标签： mysql indexing subquery calculation

【解决方案1】：

2 个唯一键。或许你可以摆脱id，将date提升为PK？
SQL_CALC_FOUND_ROWS 强制遍历所有选定的行。你能摆脱它吗？
LIMIT 0,20 -- 这是“分页”的前奏吗？如果是这样，remember where you left off 效率更高。
GROUP BY YEAR(tm.date), MONTH(tm.date), DAY(tm.date) ORDER BY tm.date 可能导致两种情况。（请参阅EXPLAIN FORMAT=JSON SELECT ... 进行验证。我认为，这将避免第二种排序，更简单，但可能不会更快：GROUP BY DATE(tm.date) ORDER BY DATE(tm.date)。请注意，他们现在说的是同样的话，从而避免了额外的排序。
SELECT tm.date ... GROUP BY ... 不合适；也换一个。
先获取数据，然后进行数据透视。（这将涉及另一层嵌套的SELECTs。）

可能最好从语句中获取 3 个值，例如单个查询。但我不知道最好的办法把它折回去。

            SELECT  DB222_DBD10_integer
                FROM  table_min
                WHERE  date <= DATE_SUB(tm.date, INTERVAL 1 DAY)
                ORDER BY  date DESC
            LIMIT  1)

【讨论】：

- 无法摆脱 ID，因为日期列是通过 cronjob 填充的，该 cronjob 聚合日期并写回主表（复杂性，避免重复） - 分页需要 SQL_CALC_FOUND_ROWS 和 LIMIT，新数据是每分钟填写一次，因此一对 ID 和日期列可以更改 - 用户可以更改间隔（分钟、qhour、小时、日、周、月），因此需要 GROUP 子句 - 为什么SELECT tm.date 不正确？我需要日期和附加字段。 - ... then pivot.这意味着建立逻辑，例如PHP 代替 MySQL？
时间问题是选择WHERE date <= DATE_SUB(tm.date, INTERVAL 1 DAY)。如果该查询被删除，查询运行速度很快。
@TDS - 如果没有总行数，当没有更多行数时您会停止。 “left off”在分页中替换 OFFSET。阅读“only_full_group_by”。
如果我对分页的理解正确：添加INDEX (id, date)，找到符合LIMIT 1 的第一个日期，然后使用date > first date 和LIMIT X 而不是LIMIT X, Y 查询所有其他数据？
@TDS - LIMIT X 说“传递前 X 行”； LIMIT X,Y 说“跳过 X 行，然后传送 Y 行”。