绕过 BigQuery 子查询并应用限制答案

【问题标题】：Getting around BigQuery subquery & apply limitations绕过 BigQuery 子查询并应用限制
【发布时间】：2018-11-13 07:43:09
【问题描述】：

我有一个 SQL Server 查询，我正在尝试将其转换为在 BigQuery 中运行。涉及三个表：

日历月

     FirstDayOfMonth        |    FirstDayOfNextMonth
----------------------------+----------------------------
2017-02-01 00:00:00.000 UTC | 2017-03-01 00:00:00.000 UTC
2017-03-01 00:00:00.000 UTC | 2017-04-01 00:00:00.000 UTC

客户

clientid |     name       | etc.
---------+----------------+------
1        |  Bob's Shop    |
2        | Anne's Cookies |

客户端日志

 id | clientid |   timestamp    | price_current | price_old | license_count_current | license_count_old |
----+----------+----------------+---------------+-----------+-----------------------+---------------
1   |     1    | 2017-02-01 UTC |      1200     |     0     |          10           |          0        |
2   |     1    | 2018-02-03 UTC |      2400     |    1200   |          20           |         10        |
3   |     2    | 2016-07-13 UTC |      1200     |     0     |          10           |          0        |
4   |     2    | 2018-03-30 UTC |       0       |    1200   |           0           |         10        |

T-SQL 查询如下所示：

SELECT 
    FirstDayOfMonth, FirstDayOfNextMonth,
    (SELECT SUM(sizeatdatelog.price_current) 
     FROM clients c
     CROSS APPLY (SELECT TOP 1 * 
                  FROM clientlogs 
                  WHERE clientid = c.clientid 
                    AND [timestamp] < cm.FirstDayOfMonth 
                  ORDER BY [timestamp] DESC) sizeatdatelog
     WHERE sizeatdatelog.license_count_current > 0) as StartingRevenue,
    (another subquery for starting client count) as StartingClientCount,
    (another subquery for churned revenue) as ChurnedRevenue,
    (there are about 6 other subqueries)
FROM 
    CalendarMonths cm
ORDER BY 
    cm.FirstDayOfMonth

最终输出如下：

     FirstDayOfMonth        |    FirstDayOfNextMonth      | StartingRevenue | StartingClientCount | etc
-------------------------------------------------------------------------------------------------------
2017-02-01 00:00:00.000 UTC | 2017-03-01 00:00:00.000 UTC |   68382995.43   |        79430        |
2017-03-01 00:00:00.000 UTC | 2017-04-01 00:00:00.000 UTC |   69843625.12   |        80430        |

在 BigQuery 中，我在 select 子句中添加了一个简单的子查询，效果很好：

SELECT FirstDayOfMonth, FirstDayOfNextMonth, (SELECT clientId FROM clientlogs LIMIT 1 ) as cl 
FROM CalendarMonths cm
ORDER BY cm.FirstDayOfMonth

但是，一旦我向子查询添加 where 子句，我就会收到以下错误消息：

错误：不支持引用其他表的相关子查询，除非它们可以去相关，例如通过将它们转换为有效的 JOIN。

我应该如何从这一点着手？如果我无法在一个查询中获得我正在寻找的结果，也许我应该考虑创建多个创建临时表的计划作业，然后将它们连接在一起的最终计划作业。或者，也许我可以考虑通过 GCP 在代码中执行此操作，或者在应用脚本中使用 BigQuery API。数据量不大，查询也不经常运行。我在寻找可维护性而不是效率，所以理想情况下有一种方法可以将这些数据放入一个查询中。

【问题讨论】：

BigQuery - Complex Correlated query 的可能重复项查看此处显示的答案并检查那里的答案是否适合您。

标签： google-bigquery

【解决方案1】：

以下是 BigQuery 标准 SQL

#standardSQL
SELECT FirstDayOfMonth, FirstDayOfNextMonth, 
  SUM(price_current) StartingRevenue, COUNT(1) StartingClientCount 
FROM (
  SELECT FirstDayOfMonth, FirstDayOfNextMonth, 
    clientid, price_current
  FROM (
    SELECT FirstDayOfMonth, FirstDayOfNextMonth, clientid,
      FIRST_VALUE(price_current) OVER(latest_values) price_current,
      FIRST_VALUE(license_count_current) OVER(latest_values) license_count_current
    FROM `project.dataset.CalendarMonths` cm
    JOIN `project.dataset.ClientLogs` cl
    ON `timestamp` < FirstDayOfMonth 
    WINDOW latest_values AS (PARTITION BY clientid ORDER BY `timestamp` DESC ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING)
  )
  WHERE license_count_current > 0
  GROUP BY FirstDayOfMonth, FirstDayOfNextMonth, clientid, price_current
)
GROUP BY FirstDayOfMonth, FirstDayOfNextMonth
ORDER BY FirstDayOfMonth

上面很可能可以扩展到您的其余子查询

【讨论】：

这周我会看看这个查询。谢谢！
@JonathanMcIntire - 运气好吗？
感谢您的帮助，我取得了进步。这需要一段时间，因为我在晚上到处都在做这个，但我相信我已经接近了。
很高兴来到这里。保持你的时间:o)
谢谢你的回答，我能够想出一个解决方案。非常感谢！

【解决方案2】：

相关子查询如

选择前 1 个 * 来自客户日志 WHERE clientid = c.clientid AND [时间戳]

在BigQuery中通常需要通过聚合来重写

SELECT ARRAY_AGG(foo ORDER BY [timestamp] DESC LIMIT 1)[offset(0)] FROM ... 作为 foo WHERE相关条件

BigQuery 更有可能以

的形式处理简单的相关子查询

选择 {可选聚合} 从表 WHERE {相关条件}

【讨论】：

【解决方案3】：

为了社区，我发布了我最终使用的查询。非常感谢 Mikhail Berlyant 在这方面的帮助。

我最终将查询分解为 CTE，这样我就可以使用相关的子查询来获取我需要的特定数据。

WITH previousMonths AS (
    SELECT *
    FROM (
        SELECT FirstDayOfMonth, FirstDayOfNextMonth, account_c,
          FIRST_VALUE(acl.timestamp_c ) OVER (start_values) timestamp_c,
          FIRST_VALUE(acl.acv_current_c ) OVER (start_values) acv_current_c,
          FIRST_VALUE(acl.license_count_current_c) OVER(start_values) license_count_current_c,
          FIRST_VALUE(acl.price_current_c) OVER (start_values) price_current_c
        FROM warehouse.project.calendar_months cm
        JOIN warehouse.project.account_change_logs acl ON timestamp_c < FirstDayOfMonth
        WINDOW start_values AS (PARTITION BY account_c, FirstDayOfMonth ORDER BY timestamp_c DESC)
    )
    GROUP BY FirstDayOfMonth, FirstDayOfNextMonth, account_c, 
      timestamp_c, acv_current_c, license_count_current_c, price_current_c
  ), 
  currentMonth AS (
    SELECT *
    FROM (
        SELECT FirstDayOfMonth, FirstDayOfNextMonth, account_c,
          FIRST_VALUE(acl.timestamp_c ) OVER (change_values) timestamp_c,
          FIRST_VALUE(acl.acv_current_c ) OVER (change_values) acv_current_c,
          FIRST_VALUE(acl.license_count_current_c) OVER(change_values) license_count_current_c,
          FIRST_VALUE(acl.acv_old_c) OVER(PARTITION BY account_c, FirstDayOfMonth ORDER BY timestamp_c) acv_old_at_start_of_month_c,
          FIRST_VALUE(acl.license_count_old_c) OVER(PARTITION BY account_c, FirstDayOfMonth ORDER BY timestamp_c) license_count_old_at_start_of_month_c,
          FIRST_VALUE(acl.price_current_c) OVER (change_values) price_current_c
        FROM warehouse.project.calendar_months  cm
        JOIN warehouse.project.account_change_logs acl
        ON timestamp_c >= FirstDayOfMonth AND timestamp_c < FirstDayOfNextMonth 
        WINDOW change_values AS (PARTITION BY account_c, FirstDayOfMonth ORDER BY timestamp_c DESC)
    )

    GROUP BY FirstDayOfMonth, FirstDayOfNextMonth, account_c, 
      timestamp_c, acv_current_c, acv_old_at_start_of_month_c, license_count_current_c, 
      license_count_old_at_start_of_month_c, price_current_c
)

SELECT FirstDayOfMonth, FirstDayOfNextMonth, 
  (SELECT COUNT(acv_current_c) FROM previousMonths pm WHERE pm.FirstDayOfMonth = cal.FirstDayOfMonth
    AND license_count_current_c > 0) as StartingAccounts,

  (SELECT COUNT(acv_current_c) FROM currentMonth cm WHERE cm.FirstDayOfMonth = cal.FirstDayOfMonth
    AND license_count_old_at_start_of_month_c = 0 AND license_count_current_c > 0) as NewAccounts,

  (SELECT COUNT(acv_current_c) FROM currentMonth cm WHERE cm.FirstDayOfMonth = cal.FirstDayOfMonth
    AND license_count_current_c = 0) as ChurnAccounts,

  (SELECT SUM(license_count_current_c) FROM previousMonths pm WHERE pm.FirstDayOfMonth = cal.FirstDayOfMonth
    AND license_count_current_c > 0) as StartingUsers,

  (SELECT SUM(license_count_current_c) FROM currentMonth cm WHERE cm.FirstDayOfMonth = cal.FirstDayOfMonth
    AND license_count_old_at_start_of_month_c = 0 AND license_count_current_c > 0) as NewUsers,

  (SELECT SUM(license_count_current_c - license_count_old_at_start_of_month_c) FROM currentMonth cm WHERE cm.FirstDayOfMonth = cal.FirstDayOfMonth
    AND license_count_old_at_start_of_month_c < license_count_current_c
    AND license_count_old_at_start_of_month_c <> 0) as ExpansionUsers,

  (SELECT SUM(license_count_old_at_start_of_month_c - license_count_current_c) FROM currentMonth cm WHERE cm.FirstDayOfMonth = cal.FirstDayOfMonth
    AND license_count_old_at_start_of_month_c > license_count_current_c
    AND license_count_current_c <> 0) as ContractionUsers,

  (SELECT SUM(license_count_old_at_start_of_month_c - license_count_current_c) FROM currentMonth cm WHERE cm.FirstDayOfMonth = cal.FirstDayOfMonth
    AND license_count_old_at_start_of_month_c > license_count_current_c
    AND license_count_current_c = 0) as ChurnUsers,

  (SELECT SUM(acv_current_c) FROM previousMonths pm WHERE pm.FirstDayOfMonth = cal.FirstDayOfMonth
    AND license_count_current_c > 0) as StartingARR 

    --etc, etc,


FROM warehouse.project.calendar_months cal
ORDER BY FirstDayOfMonth

【讨论】：