SQL聚合查询，非线性成本

Question

我正在对 postgres 13 + postgis 3 + timescaleDB 2 数据库中的一些 timeseris GPS 数据进行复杂聚合。我正在查看的表每天有几百万个条目，我想在几个月内进行聚合（每天一行，每个 gps_id，每个组间隙 ID）。

假设我创建了一个函数来执行聚合：

--pseudo code, won't actually work...
CREATE FUNCTION my_agg_func(starttime, endtime)
AS

WITH gps_window AS 
(SELECT gps.id,
        gps.geom,
        gps.time,
        -- find where there are 1 hour gaps in data 
        lag(ais.time) OVER (PARTITION BY gps.id ORDER BY gps.time) <= (gps.time - '01:00:00'::interval) AS time_step,
       -- find where there are 0.1 deg gaps in position
        st_distance(gps.geom, lag(gps.geom) OVER (PARTITION BY gps.id ORDER BY gps.time)) >= 0.1 AS dist_step
 FROM gps
 WHERE gps.time BETWEEN starttime AND endtime
), groups AS (
SELECT gps_window.id,
       gps_window.geom,
       gps_window.time,
       count(*) FILTER (WHERE gps_window.time_step) OVER (PARTITION BY gps_window.id ORDER BY gps_window.time) AS time_grp,
       count(*) FILTER (WHERE gps_window.dist_step) OVER (PARTITION BY gps_window.id ORDER BY gps_window.time) AS dist_grp
       FROM gps_window
      --get rid of duplicate points
      WHERE gps_window.dist > 0
      ) 
SELECT 
  gps_id,
  date(gps.time),
  time_grp,
  dist_grp
  st_setsrid(st_makeline(gps_window."position" ORDER BY gps_window.event_time), 4326) AS geom,
FROM groups
WHERE gps_time BETWEEN starttime AND endtime
GROUP BY gps.id, date(gps.time), time_grp, dist_grp

gap_id 函数正在检查来自相同 gps_id 的连续 gps 点，这些点彼此相距太远，移动速度过快或消息之间的时间过长。聚合基本上是从 gps 点创建一条线。的最终结果是一堆线，其中线中的所有点都是“合理的”。

要运行 1 天的聚合函数（开始时间 = '2020-01-01'，结束时间 = '2020-01-02'），大约需要 12 秒才能完成。如果我选择一周的数据，则需要 10 分钟。如果我选择一个月的数据，则需要 15 小时以上才能完成。

我期望线性性能，因为无论如何数据都会每天分组，但事实并非如此。绕过这个性能瓶颈的明显方法是在 for 循环中运行它：

for date in date_range(starttime, endtime):
    my_agg_func(date, date+1) 

我可以在 Python 中做到这一点，但有什么想法可以让 for 循环在 postgres 中运行或将聚合查询更改为线性？

Answer 1

时间间隔的聚合（在 SQL 文献中称为 COLLAPSE 运算符）导致编写复杂的查询，其执行成本可能是指数或多项式，具体取决于所使用的方法。 SNODGRASS 或 Chris Date 的旧经典 SQL 公式是指数级的。最近，Microsoft SQL Server MVP Itzik Ben Gan 编写了一个多项式形式，它提供了出色的响应时间，但使用了 CROSS APPLY，由 Microsoft 发明并被 Oracle 接管......查询如下：

WITH 
C1 AS (SELECT ITV_ITEM, ITV_DEBUT AS ts, +1 AS genre, NULL AS e,
              ROW_NUMBER() OVER(PARTITION BY ITV_ITEM ORDER BY ITV_DEBUT) AS s
       FROM   T_INTERVAL_ITV
       UNION  ALL
       SELECT ITV_ITEM, ITV_FIN AS ts, -1 AS genre, 
              ROW_NUMBER() OVER(PARTITION BY ITV_ITEM ORDER BY ITV_FIN) AS e,
              NULL AS s
       FROM T_INTERVAL_ITV),
C2 AS (SELECT C1.*, ROW_NUMBER() OVER(PARTITION BY ITV_ITEM ORDER BY ts, genre DESC) 
                 AS se
       FROM   C1),
C3 AS (SELECT ITV_ITEM, ts, 
              FLOOR((ROW_NUMBER() OVER(PARTITION BY ITV_ITEM ORDER BY ts) - 1) / 2 + 1) 
                 AS grpnum
       FROM   C2
       WHERE  COALESCE(s - (se - s) - 1, (se - e) - e) = 0),
C4 AS (SELECT ITV_ITEM, MIN(ts) AS ITV_DEBUT, max(ts) AS ITV_FIN
       FROM C3
       GROUP BY ITV_ITEM, grpnum)
SELECT A.ITV_ITEM, A.ITV_DEBUT, A.ITV_FIN
FROM   (SELECT DISTINCT ITV_ITEM 
        FROM   T_INTERVAL_ITV) AS U
       CROSS APPLY (SELECT *
                    FROM   C4
                    WHERE  ITV_ITEM = U.ITV_ITEM) AS A
ORDER BY ITV_ITEM, ITV_DEBUT, ITV_FIN;

您能否使用 LATERAL 连接转换此 MS SQL Server 特定查询？这将帮助您获得更好的执行时间。