【问题标题】:Aggregating the most recent joined records per week每周汇总最近加入的记录
【发布时间】:2016-07-01 13:14:37
【问题描述】:

我在 Postgres 中有一个 updates 表,它是 9.4.5,如下所示:

goal_id    | created_at | status
1          | 2016-01-01 | green
1          | 2016-01-02 | red
2          | 2016-01-02 | amber

还有一个像这样的goals 表:

id | company_id
1  | 1
2  | 2

我想为每家公司创建一个图表,显示他们每周所有目标的状态。

我认为这需要生成过去 8 周的一系列数据,找到该周之前每个目标的最新更新,然后计算找到的更新的不同状态。

到目前为止我所拥有的:

SELECT EXTRACT(year from generate_series) AS year, 
       EXTRACT(week from generate_series) AS week,
       u.company_id,
       COUNT(*) FILTER (WHERE u.status = 'green') AS green_count,
       COUNT(*) FILTER (WHERE u.status = 'amber') AS amber_count,
       COUNT(*) FILTER (WHERE u.status = 'red') AS red_count
FROM generate_series(NOW() - INTERVAL '2 MONTHS', NOW(), '1 week')
LEFT OUTER JOIN (
  SELECT DISTINCT ON(year, week)
         goals.company_id,
         updates.status, 
         EXTRACT(week from updates.created_at) week,
         EXTRACT(year from updates.created_at) AS year,
         updates.created_at 
  FROM updates
  JOIN goals ON goals.id = updates.goal_id
  ORDER BY year, week, updates.created_at DESC
) u ON u.week = week AND u.year = year
GROUP BY 1,2,3

但这有两个问题。 u 上的连接似乎没有像我想象的那样工作。它似乎加入了从内部查询返回的每一行(?),并且这只选择了那一周发生的最新更新。如果需要,它应该获取该周之前的最新更新。

这是一些相当复杂的 SQL,我喜欢一些关于如何完成它的意见。

表结构和信息

目标表有大约 1000 个目标 ATM,并且每周增长大约 100 个:

                                           Table "goals"
     Column      |            Type             |                         Modifiers
-----------------+-----------------------------+-----------------------------------------------------------
 id              | integer                     | not null default nextval('goals_id_seq'::regclass)
 company_id      | integer                     | not null
 name            | text                        | not null
 created_at      | timestamp without time zone | not null default timezone('utc'::text, now())
 updated_at      | timestamp without time zone | not null default timezone('utc'::text, now())
Indexes:
    "goals_pkey" PRIMARY KEY, btree (id)
    "entity_goals_company_id_fkey" btree (company_id)
Foreign-key constraints:
    "goals_company_id_fkey" FOREIGN KEY (company_id) REFERENCES companies(id) ON DELETE RESTRICT

updates 表有大约 1000 个,并且每周增长大约 100 个:

                                         Table "updates"
   Column   |            Type             |                            Modifiers
------------+-----------------------------+------------------------------------------------------------------
 id         | integer                     | not null default nextval('updates_id_seq'::regclass)
 status     | entity.goalstatus           | not null
 goal_id    | integer                     | not null
 created_at | timestamp without time zone | not null default timezone('utc'::text, now())
 updated_at | timestamp without time zone | not null default timezone('utc'::text, now())
Indexes:
    "goal_updates_pkey" PRIMARY KEY, btree (id)
    "entity_goal_updates_goal_id_fkey" btree (goal_id)
Foreign-key constraints:
    "updates_goal_id_fkey" FOREIGN KEY (goal_id) REFERENCES goals(id) ON DELETE CASCADE

 Schema |       Name        | Internal name | Size | Elements | Access privileges | Description
--------+-------------------+---------------+------+----------+-------------------+-------------
 entity | entity.goalstatus | goalstatus    | 4    | green   +|                   |
        |                   |               |      | amber   +|                   |
        |                   |               |      | red      |                   |

【问题讨论】:

  • 我怀疑你想要一个window function - 你可以按你的时间片分区
  • @Codeman 嗯,看起来你是对的。我从来没有使用过窗口函数。你碰巧知道有什么好的资源可以看吗?谢谢!
  • 可能是我联系你的那个:)
  • 如果您将示例数据扩展到几十行并根据该示例数据添加预期结果,将会有所帮助。这将有助于理解所需的逻辑并验证解决方案的正确性。如果您的真实数据集很重要(100K+ 行),告诉我们每个表有多少行不会有什么坏处。解决方案的效率取决于数据分布是很常见的。
  • 您应该提供显示数据类型和约束的实际表定义。并且始终是您的 Postgres 版本。

标签: sql postgresql greatest-n-per-group


【解决方案1】:

您每周需要一个数据项和目标(在汇总每个公司的计数之前)。这是generate_series()goals 之间的普通CROSS JOIN。 (可能)昂贵的部分是从updates 获取当前的state。与@Paul already suggested 一样,LATERAL 加入似乎是最好的工具。不过,仅对updates 执行此操作,并对LIMIT 1 使用更快的技术。

并使用date_trunc() 简化日期处理。

SELECT w_start
     , g.company_id
     , count(*) FILTER (WHERE u.status = 'green') AS green_count
     , count(*) FILTER (WHERE u.status = 'amber') AS amber_count
     , count(*) FILTER (WHERE u.status = 'red')   AS red_count
FROM   generate_series(date_trunc('week', NOW() - interval '2 months')
                     , date_trunc('week', NOW())
                     , interval '1 week') w_start
CROSS  JOIN goals g
LEFT   JOIN LATERAL (
   SELECT status
   FROM   updates
   WHERE  goal_id = g.id
   AND    created_at < w_start
   ORDER  BY created_at DESC
   LIMIT  1
   ) u ON true
GROUP  BY w_start, g.company_id
ORDER  BY w_start, g.company_id;

要使这个快速,您需要一个多列索引

CREATE INDEX updates_special_idx ON updates (goal_id, created_at DESC, status);

created_at 的降序是最好的,但不是绝对必要的。 Postgres 几乎可以以同样快的速度向后扫描索引。 (Not applicable for inverted sort order of multiple columns, though.)

顺序索引列。为什么?

第三列status 仅附加以允许updates 上的快速index-only scans。相关案例:

9 周的 1k 目标(您的 2 个月间隔与至少 9 周重叠)只需要对只有 1k 行的第二个表进行 9k 索引查找。对于像这样的小表,性能应该不是什么大问题。但是,一旦每个表中多了几千个,顺序扫描的性能就会下降。

w_start 表示每周的开始。因此,计数是针对本周开始的。如果您坚持,您可以仍然提取年份和周(或任何其他详细信息代表您的周):

   EXTRACT(isoyear from w_start) AS year
 , EXTRACT(week    from w_start) AS week

最好使用ISOYEAR,就像@Paul 解释的那样。

SQL Fiddle.

相关:

【讨论】:

  • 可爱! :-) 我考虑过使用交叉连接,但出于某种原因决定反对它。 @Eric 应该接受你的回答。显然 perf 是关于测试的,但我对你的版本更有信心。 :-)
  • @Paul:这个查询是从我最近的一个类似项目中衍生出来的——我在其中进行了很多测试,以找到这种查询和索引的组合以达到最佳性能。尽管如此,布丁的证据在于吃。
【解决方案2】:

这对于LATERAL 连接似乎很有用:

SELECT  EXTRACT(ISOYEAR FROM s) AS year,
        EXTRACT(WEEK FROM s) AS week,
        u.company_id,
        COUNT(u.goal_id) FILTER (WHERE u.status = 'green') AS green_count,
        COUNT(u.goal_id) FILTER (WHERE u.status = 'amber') AS amber_count,
        COUNT(u.goal_id) FILTER (WHERE u.status = 'red') AS red_count
FROM    generate_series(NOW() - INTERVAL '2 months', NOW(), '1 week') s(w)
LEFT OUTER JOIN LATERAL (
  SELECT  DISTINCT ON (g.company_id, u2.goal_id) g.company_id, u2.goal_id, u2.status
  FROM    updates u2
  INNER JOIN goals g
  ON      g.id = u2.goal_id
  WHERE   u2.created_at <= s.w
  ORDER BY g.company_id, u2.goal_id, u2.created_at DESC
) u 
ON true
WHERE   u.company_id IS NOT NULL
GROUP BY year, week, u.company_id
ORDER BY u.company_id, year, week
;

顺便说一句,我提取的是 ISOYEAR 而不是 YEAR 以确保我在 1 月初左右获得合理的结果。例如EXTRACT(YEAR FROM '2016-01-01 08:49:56.734556-08')2016EXTRACT(WEEK FROM '2016-01-01 08:49:56.734556-08')53

编辑:你应该测试你的真实数据,但我觉得这应该更快:

SELECT  year,
        week,
        company_id,
        COUNT(goal_id) FILTER (WHERE last_status = 'green') AS green_count,
        COUNT(goal_id) FILTER (WHERE last_status = 'amber') AS amber_count,
        COUNT(goal_id) FILTER (WHERE last_status = 'red') AS red_count
FROM    (
  SELECT  EXTRACT(ISOYEAR FROM s) AS year,
          EXTRACT(WEEK FROM s) AS week,
          u.company_id,
          u.goal_id,
          (array_agg(u.status ORDER BY u.created_at DESC))[1] AS last_status
  FROM    generate_series(NOW() - INTERVAL '2 months', NOW(), '1 week') s(t)
  LEFT OUTER JOIN ( 
    SELECT  g.company_id, u2.goal_id, u2.created_at, u2.status
    FROM    updates u2
    INNER JOIN goals g 
    ON      g.id = u2.goal_id
  ) u 
  ON      s.t >= u.created_at
  WHERE   u.company_id IS NOT NULL
  GROUP BY year, week, u.company_id, u.goal_id
) x
GROUP BY year, week, company_id
ORDER BY company_id, year, week
;

但仍然没有窗口功能。 :-) 您还可以通过将(array_agg(...))[1] 替换为真正的first 函数来加快速度。您必须自己定义,但 Postgres wiki 上的实现很容易通过 Google 搜索。

【讨论】:

  • 哦,哇,我以前听说过 LATERAL 加入但从未使用过它们。太棒了,谢谢!
  • 他们非常好!我通常从LATERALjoins 获得了出色的性能,但我有点担心这个。我同意@Codeman 的观点,这种 感觉 就像你也可以使用 windows 函数一样。 . .但如果是这样,我不知道该怎么做!
【解决方案3】:

我使用 PostgreSQL 9.3。我对你的问题很感兴趣。我检查了你的数据结构。比我创建以下表格。

我插入以下记录;

公司

目标

更新

之后我写了以下查询,以供更正

SELECT c.id company_id, c.name company_name, u.status goal_status, 
         EXTRACT(week from u.created_at) goal_status_week,
         EXTRACT(year from u.created_at) AS goal_status_year 
FROM company c
INNER JOIN goals g ON g.company_id = c.id 
INNER JOIN updates u ON u.goal_id = g.id
ORDER BY goal_status_year DESC, goal_status_week DESC;

我得到以下结果;

最后我将此查询与周系列合并

SELECT
             gs.company_id,
             gs.company_name,
             gs.goal_status,
             EXTRACT(year from w) AS year, 
       EXTRACT(week from w) AS week,
             COUNT(gs.*) cnt
FROM generate_series(NOW() - INTERVAL '3 MONTHS', NOW(), '1 week') w
LEFT JOIN(
SELECT c.id company_id, c.name company_name, u.status goal_status, 
             EXTRACT(week from u.created_at) goal_status_week,
       EXTRACT(year from u.created_at) AS goal_status_year 
FROM company c
INNER JOIN goals g ON g.company_id = c.id 
INNER JOIN updates u ON u.goal_id = g.id ) gs 
ON gs.goal_status_week = EXTRACT(week from w) AND gs.goal_status_year = EXTRACT(year from w)
GROUP BY company_id, company_name, goal_status, year, week
ORDER BY  year DESC, week DESC;

我得到了这个结果

祝你有美好的一天。

【讨论】:

    猜你喜欢
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 2018-04-23
    • 2022-01-01
    • 2018-06-02
    • 1970-01-01
    • 2018-01-13
    相关资源
    最近更新 更多