UNNEST 数组存储在对象内？

Question

我正在尝试取消嵌套存储在 BigQuery 中的对象内的数组。对于 JSON，我相对缺乏经验，但我一直在寻找解决方案，但没有运气。

我还有其他类似的情况，我在 BigQuery 中创建了 UDF 来定义 JSON 数组和 JSON 对象的结构，它们是分开的，但这是我遇到的第一个情况，其中一个在另一个内部。

示例结构（JSON 名称为“路由”）；

{"legs": 

        [{"eta": "2020-04-25T00:00:00.000Z", "etd": "2020-03-31T00:00:00.000Z", "change": "new"}], 

"arrival_time": "2020-04-25T00:00:00.000+00:00", "cut_off_date": "2020-03-29", 
"departure_time": "2020-03-31T00:00:00.000+00:00", "last_pickup_on": "2020-03-29"}

我尝试了很多不同的东西，很难列出，所以我想知道是否有人对此有一个我可能忽略的简单解决方案？我已通读 BQ 文档并尝试过...

UNNEST(routes) as routes_unnest

这不起作用，以及存储的 UDF，其结构与下面类似......

CREATE OR REPLACE FUNCTION `project`.udfs.extract_routes(input STRING)
  RETURNS STRUCT<
    arrival_time TIMESTAMP,
    cut_off_date STRING,
    departure_time TIMESTAMP,
    last_pickup_on STRING,
    legs ARRAY<
    STRUCT<
      eta TIMESTAMP,
      etd TIMESTAMP,
      change STRING,
  LANGUAGE js AS "return JSON.parse(input);";

我相信我的失败在于我的查询，如果是这种情况，那么我会被如何查询数组部分所困扰。在这里非常感谢您的帮助！感谢您的宝贵时间。

Answer 1

以下是 BigQuery 标准 SQL

#standardSQL
CREATE TEMP FUNCTION json2array(input STRING)
RETURNS ARRAY<STRING>
LANGUAGE js AS '''
  return JSON.parse(input).map(x=>JSON.stringify(x));
'''; 
SELECT 
  JSON_EXTRACT_SCALAR(routes, '$.arrival_time') AS arrival_time,
  JSON_EXTRACT_SCALAR(routes, '$.cut_off_date') AS cut_off_date,
  JSON_EXTRACT_SCALAR(routes, '$.departure_time') AS departure_time,
  JSON_EXTRACT_SCALAR(routes, '$.last_pickup_on') AS last_pickup_on,
  JSON_EXTRACT_SCALAR(leg, '$.eta') eta,
  JSON_EXTRACT_SCALAR(leg, '$.etd') etd,
  JSON_EXTRACT_SCALAR(leg, '$.change') change
FROM `project.dataset.table`,
UNNEST(json2array(JSON_EXTRACT(routes, '$.legs'))) leg

你可以像下面的例子一样使用虚拟数据测试，玩上面的例子

#standardSQL
CREATE TEMP FUNCTION json2array(input STRING)
RETURNS ARRAY<STRING>
LANGUAGE js AS '''
  return JSON.parse(input).map(x=>JSON.stringify(x));
'''; 
WITH `project.dataset.table` AS (
SELECT '''
  {"legs": 
  [
    {"eta": "2020-04-25T00:00:00.000Z", "etd": "2020-03-31T00:00:00.000Z", "change": "new"},
    {"eta": "2020-04-25T00:00:00.000Z", "etd": "2020-03-31T00:00:00.000Z", "change": "new1"},
    {"eta": "2020-04-25T00:00:00.000Z", "etd": "2020-03-31T00:00:00.000Z", "change": "new2"}
  ], 
  "arrival_time": "2020-04-25T00:00:00.000+00:00", 
  "cut_off_date": "2020-03-29", 
  "departure_time": "2020-03-31T00:00:00.000+00:00", 
  "last_pickup_on": "2020-03-29"}
''' routes
)
SELECT 
  JSON_EXTRACT_SCALAR(routes, '$.arrival_time') AS arrival_time,
  JSON_EXTRACT_SCALAR(routes, '$.cut_off_date') AS cut_off_date,
  JSON_EXTRACT_SCALAR(routes, '$.departure_time') AS departure_time,
  JSON_EXTRACT_SCALAR(routes, '$.last_pickup_on') AS last_pickup_on,
  JSON_EXTRACT_SCALAR(leg, '$.eta') eta,
  JSON_EXTRACT_SCALAR(leg, '$.etd') etd,
  JSON_EXTRACT_SCALAR(leg, '$.change') change
FROM `project.dataset.table`,
UNNEST(json2array(JSON_EXTRACT(routes, '$.legs'))) leg

下面的输出

Row arrival_time                cut_off_date    departure_time          last_pickup_on  eta etd change   
1   2020-04-25T00:00:00.000+00:00   2020-03-29  2020-03-31T00:00:00.000+00:00   2020-03-29  2020-04-25T00:00:00.000Z    2020-03-31T00:00:00.000Z    new  
2   2020-04-25T00:00:00.000+00:00   2020-03-29  2020-03-31T00:00:00.000+00:00   2020-03-29  2020-04-25T00:00:00.000Z    2020-03-31T00:00:00.000Z    new1     
3   2020-04-25T00:00:00.000+00:00   2020-03-29  2020-03-31T00:00:00.000+00:00   2020-03-29  2020-04-25T00:00:00.000Z    2020-03-31T00:00:00.000Z    new2     

或者，如果您仍需要以 [未嵌套] JSON 形式输出，则可以使用 TO_JSON_STRING 结束查询

#standardSQL
CREATE TEMP FUNCTION json2array(input STRING)
RETURNS ARRAY<STRING>
LANGUAGE js AS '''
  return JSON.parse(input).map(x=>JSON.stringify(x));
'''; 
SELECT TO_JSON_STRING(t) AS route
FROM (
  SELECT 
    JSON_EXTRACT_SCALAR(routes, '$.arrival_time') AS arrival_time,
    JSON_EXTRACT_SCALAR(routes, '$.cut_off_date') AS cut_off_date,
    JSON_EXTRACT_SCALAR(routes, '$.departure_time') AS departure_time,
    JSON_EXTRACT_SCALAR(routes, '$.last_pickup_on') AS last_pickup_on,
    JSON_EXTRACT_SCALAR(leg, '$.eta') eta,
    JSON_EXTRACT_SCALAR(leg, '$.etd') etd,
    JSON_EXTRACT_SCALAR(leg, '$.change') change
  FROM `project.dataset.table`,
  UNNEST(json2array(JSON_EXTRACT(routes, '$.legs'))) leg
) t   

在这种情况下，输出如下所示

Row route    
1   {"arrival_time":"2020-04-25T00:00:00.000+00:00","cut_off_date":"2020-03-29","departure_time":"2020-03-31T00:00:00.000+00:00","last_pickup_on":"2020-03-29","eta":"2020-04-25T00:00:00.000Z","etd":"2020-03-31T00:00:00.000Z","change":"new"}     
2   {"arrival_time":"2020-04-25T00:00:00.000+00:00","cut_off_date":"2020-03-29","departure_time":"2020-03-31T00:00:00.000+00:00","last_pickup_on":"2020-03-29","eta":"2020-04-25T00:00:00.000Z","etd":"2020-03-31T00:00:00.000Z","change":"new1"}    
3   {"arrival_time":"2020-04-25T00:00:00.000+00:00","cut_off_date":"2020-03-29","departure_time":"2020-03-31T00:00:00.000+00:00","last_pickup_on":"2020-03-29","eta":"2020-04-25T00:00:00.000Z","etd":"2020-03-31T00:00:00.000Z","change":"new2"}    

其中每一行都是一个 JSON，可以使用 JSON_EXTRACT_SCALAR 函数轻松地进一步查询

{
"arrival_time": "2020-04-25T00:00:00.000+00:00",
"cut_off_date": "2020-03-29",
"departure_time": "2020-03-31T00:00:00.000+00:00",
"last_pickup_on": "2020-03-29",
"eta": "2020-04-25T00:00:00.000Z",
"etd": "2020-03-31T00:00:00.000Z",
"change": "new"
}