BigQuery 合并 JSON 文档

Question

我想要一个带有 JSON 列的表。此 JSON 列可以包含任意文档。我想根据它们在另一列中可用的时间戳合并这些文档。有没有办法通过时间戳合并这些 JSON 文档？

这是一个例子：

at t3 time {a:"1", b:"2"}
at t2 time {b:"4"}
at t1 time {a:"4", c:"5"}

我想创建{a:"1", b:"2", c:"5"} 作为输出。在 BigQuery 中可以做到这一点吗？

Answer 1

以下是 BigQuery 标准 SQL

#standardSQL
SELECT STRING_AGG(y, ', ' ORDER BY y) json
FROM (
  SELECT STRING_AGG(TRIM(x) ORDER BY t DESC LIMIT 1) y
  FROM `project.dataset.table`, 
    UNNEST(SPLIT(REGEXP_REPLACE(json, r'{|}', ''))) x
  GROUP BY TRIM(SPLIT(x, ':')[OFFSET(0)]) 
)   

注意：上述解决方案是通用的，不需要事先知道属性名称（如a、b 等），而是解析并提取它会找到的任何内容。显然，它依赖于您的示例中的简单 json 假设

您可以使用您问题中的示例数据进行测试，使用上面的示例，如下例所示

#standardSQL
WITH `project.dataset.table` AS (
  SELECT '{"a":"1", "b":"2"}' json, 3 t UNION ALL
  SELECT '{"b":"4"}', 2 UNION ALL 
  SELECT '{"a":"4", "c":"5"}', 1  
)
SELECT STRING_AGG(y, ', ' ORDER BY y) json
FROM (
  SELECT STRING_AGG(TRIM(x) ORDER BY t DESC LIMIT 1) y
  FROM `project.dataset.table`, 
    UNNEST(SPLIT(REGEXP_REPLACE(json, r'{|}', ''))) x
  GROUP BY TRIM(SPLIT(x, ':')[OFFSET(0)]) 
)

结果

Row json     
1   "a":"1", "b":"2", "c":"5"     

因为（正如我所提到的）它足够通用 - 您可以添加具有更多属性的行，而无需更改代码，如下所示

#standardSQL
WITH `project.dataset.table` AS (
  SELECT '{"a":"1", "b":"2"}' json, 3 t UNION ALL
  SELECT '{"b":"4"}', 2 UNION ALL 
  SELECT '{"a":"4", "c":"5"}', 1 UNION ALL
  SELECT '{"abc":"1", "xyz":"2"}', 3 UNION ALL
  SELECT '{"abc":"3", "vwy":"4"}', 3  
)
SELECT STRING_AGG(y, ', ' ORDER BY y) json
FROM (
  SELECT STRING_AGG(TRIM(x) ORDER BY t DESC LIMIT 1) y
  FROM `project.dataset.table`, 
    UNNEST(SPLIT(REGEXP_REPLACE(json, r'{|}', ''))) x
  GROUP BY TRIM(SPLIT(x, ':')[OFFSET(0)]) 
)

结果

Row json     
1   "a":"1", "abc":"1", "b":"2", "c":"5", "vwy":"4", "xyz":"2"   

Answer 2

以下是使用 BigQuery 标准 SQL 函数和您的数据的可能解决方案：

#standardSQL
WITH test AS (
  SELECT '{"a":"1", "b":"2"}' AS json, 3 AS t UNION ALL
  SELECT '{"b":"4"}' AS json, 2 AS t UNION ALL 
  SELECT '{"a":"4", "c":"5"}' AS json, 1 AS t 
)
SELECT data_row, TO_JSON_STRING(data_row) AS json_row
FROM (
  SELECT 
    ARRAY_TO_STRING(ARRAY_AGG(a IGNORE NULLS ORDER BY t DESC LIMIT 1),'') AS a,
    ARRAY_TO_STRING(ARRAY_AGG(b IGNORE NULLS ORDER BY t DESC LIMIT 1),'') AS b,
    ARRAY_TO_STRING(ARRAY_AGG(c IGNORE NULLS ORDER BY t DESC LIMIT 1),'') AS c
  FROM(
    SELECT JSON_EXTRACT_SCALAR(json,'$.a') AS a,
    JSON_EXTRACT_SCALAR(json,'$.b') AS b,
    JSON_EXTRACT_SCALAR(json,'$.c') AS c, 
    t
    FROM test
  )
) AS data_row

请注意，ARRAY_AGG 仅用于为每个文档查找最新的非 NULL 值，因此将其转换为 STRING 和 ARRAY_TO_STRING。这个查询的结果是，应该是期望的：

Row data_row.a  data_row.b  data_row.c  json_row     
1   1           2           5           {"a":"1","b":"2","c":"5"}   

此查询的问题是您必须指定所有文档（在本例中为a,b,c）。

Answer 3

可能有更好的方法，而我的第一个想法是：

1. 使用带有唯一分隔符的 STRING_AGG 聚合多个 json 字符串（如下例所示：'||||||'）
2. 使用 JavaScript UDF 解析 JSON/合并/输出为字符串。

#standardSQL
CREATE TEMPORARY FUNCTION merge_json(json_string STRING)
RETURNS STRING
LANGUAGE js
AS 
"""
  // TODO 1: split json string with '||||||' to get multiple parts
  // .    2: parse each json parts into object
  //      3: merge objects in your own way

  // fake output, just to demonstrate the idea
  var obj = JSON.parse('{"a":"1", "b":"2", "c":"5"}')
  return JSON.stringify(obj);
""";
WITH
  sample_data AS (
  SELECT '{a:"1", b:"2"}' AS json, 1000 AS timestamp
  UNION ALL
  SELECT '{b:"4"}' AS json, 2000 AS timestamp
  UNION ALL 
  SELECT '{a:"4", c:"5"}' AS json, 1000 AS timestamp )

SELECT timestamp, merge_json(STRING_AGG(json, '||||||')) as joined_json
FROM sample_data
GROUP BY timestamp

输出：