在 BigQuery 中滚动 90 天活跃用户，提高性能 (DAU/MAU/WAU)

Question

我正在尝试获取特定日期的唯一事件数，回滚 90/30/7 天。我已经使用下面的查询在有限数量的行上进行了这项工作，但是对于大型数据集，我会从变得庞大的聚合字符串中得到内存错误。

我正在寻找一种更有效的方法来实现相同的结果。

表格看起来像这样：

+---+------------+-------------+
|   |     date   |     userid  |
+---+------------+-------------+
| 1 | 2013-05-14 | xxxxx       |
| 2 | 2017-03-14 | xxxxx       |
| 3 | 2018-01-24 | xxxxx       |
| 4 | 2013-03-21 | xxxxx       |
| 5 | 2014-03-19 | xxxxx       |
| 6 | 2015-09-03 | xxxxx       |
| 7 | 2014-02-06 | xxxxx       |
| 8 | 2014-10-30 | xxxxx       |
| ..| ...        | ...         |
+---+------------+-------------+

所需结果的格式：

+---+------------+---------------------------------------------+
|   |     date   | active_users_7_days | active_users_90_days  |
+---+------------+---------------------------------------------+
| 1 | 2013-05-14 | 1240                | 34339                 |
| 2 | 2017-03-14 | 4334                | 54343                 |
| 3 | 2018-01-24 | .....               | .....                 |
| 4 | 2013-03-21 | .....               | .....                 |
| 5 | 2014-03-19 | .....               | .....                 |
| 6 | 2015-09-03 | .....               | .....                 |
| 7 | 2014-02-06 | .....               | .....                 |
| 8 | 2014-10-30 | .....               | .....                 |
| ..| ...        | .....               | .....                 |
+---+------------+---------------------------------------------+

我的查询如下所示：

#standardSQL
    WITH
      T1 AS(
      SELECT
        date,
        STRING_AGG(DISTINCT userid) AS IDs
      FROM
        `consumer.events`
      GROUP BY
        date ),
      T2 AS(
      SELECT
        date,
        STRING_AGG(IDs) OVER(ORDER BY UNIX_DATE(date) RANGE BETWEEN 90 PRECEDING
          AND CURRENT ROW) AS IDs
      FROM
        T1 )
    SELECT
      date,
      (
      SELECT
        COUNT(DISTINCT (userid))
      FROM
        UNNEST(SPLIT(IDs)) AS userid) AS NinetyDays
    FROM
      T2

Answer 1

计算唯一身份用户需要大量资源，如果您希望通过滚动窗口获得结果，则需要更多资源。对于可扩展的解决方案，请查看 HLL++ 等近似算法：

• https://medium.freecodecamp.org/counting-uniques-faster-in-bigquery-with-hyperloglog-5d3764493a5a

对于精确计数，这会起作用（但随着窗口变大会变慢）：

#standardSQL
SELECT DATE_SUB(date, INTERVAL i DAY) date_grp
 , COUNT(DISTINCT owner_user_id) unique_90_day_users
 , COUNT(DISTINCT IF(i<31,owner_user_id,null)) unique_30_day_users
 , COUNT(DISTINCT IF(i<8,owner_user_id,null)) unique_7_day_users
FROM (
  SELECT DATE(creation_date) date, owner_user_id
  FROM `bigquery-public-data.stackoverflow.posts_questions` 
  WHERE EXTRACT(YEAR FROM creation_date)=2017
  GROUP BY 1, 2
), UNNEST(GENERATE_ARRAY(1, 90)) i
GROUP BY 1
ORDER BY date_grp

近似解决方案产生结果的速度更快（14s vs 366s，但结果是近似的）：

#standardSQL
SELECT DATE_SUB(date, INTERVAL i DAY) date_grp
 , HLL_COUNT.MERGE(sketch) unique_90_day_users
 , HLL_COUNT.MERGE(DISTINCT IF(i<31,sketch,null)) unique_30_day_users
 , HLL_COUNT.MERGE(DISTINCT IF(i<8,sketch,null)) unique_7_day_users
FROM (
  SELECT DATE(creation_date) date, HLL_COUNT.INIT(owner_user_id) sketch
  FROM `bigquery-public-data.stackoverflow.posts_questions` 
  WHERE EXTRACT(YEAR FROM creation_date)=2017
  GROUP BY 1
), UNNEST(GENERATE_ARRAY(1, 90)) i
GROUP BY 1
ORDER BY date_grp

---

更新的查询提供了正确的结果 - 删除少于 90 天的行（在没有日期缺失的情况下有效）：

#standardSQL
SELECT DATE_SUB(date, INTERVAL i DAY) date_grp
 , HLL_COUNT.MERGE(sketch) unique_90_day_users
 , HLL_COUNT.MERGE(DISTINCT IF(i<31,sketch,null)) unique_30_day_users
 , HLL_COUNT.MERGE(DISTINCT IF(i<8,sketch,null)) unique_7_day_users
 , COUNT(*) window_days
FROM (
  SELECT DATE(creation_date) date, HLL_COUNT.INIT(owner_user_id) sketch
  FROM `bigquery-public-data.stackoverflow.posts_questions` 
  WHERE EXTRACT(YEAR FROM creation_date)=2017
  GROUP BY 1
), UNNEST(GENERATE_ARRAY(1, 90)) i
GROUP BY 1
HAVING window_days=90
ORDER BY date_grp

Answer 2

您可以汇总日期并求和。什么是聚合？取最近的日期：

select count(*) as num_users,
       sum(case when date > datediff(current_date, interval -30 day) then 1 else 0 end) as num_users_30days,
       sum(case when date > datediff(current_date, interval -60 day) then 1 else 0 end) as num_users_60days,
       sum(case when date > datediff(current_date, interval -90 day) then 1 else 0 end) as num_users_90days
from (select user_id, max(date) as max(date)
      from `consumer.events` e
      group by user_id
     ) e;

如果用户最近的日期在该时间段内，则应计算该用户。

您可以通过在子查询中使用where 子句来获取特定日期的“当前日期”。