将 JSON 加载到 BigQuery：字段有时是数组，有时是字符串答案

【问题标题】：Loading JSON into BigQuery: Field is sometimes an array and sometimes a string将 JSON 加载到 BigQuery：字段有时是数组，有时是字符串
【发布时间】：2020-12-30 16:05:49
【问题描述】：

我正在尝试将 JSON 数据加载到 BigQuery。我的数据导致问题的摘录如下所示：

 [{"Value":"123","Code":"A"},{"Value":"000","Code":"B"}]
 {"Value":"456","Code":"A"}
 [{"Value":"123","Code":"A"},{"Value":"789","Code":"C"},{"Value":"000","Code":"B"}]
 {"Value":"Z","Code":"A"}

我已将此字段的架构定义为：

  {
    "fields": [
      {
        "mode": "NULLABLE",
        "name": "Code",
        "type": "STRING"
      },
      {
        "mode": "NULLABLE",
        "name": "Value",
        "type": "STRING"
      }
    ],
    "mode": "REPEATED",
    "name": "Properties",
    "type": "RECORD"
  }

但我无法成功地将字符串和数组值提取到一个重复字段中。此 SQL 将成功提取字符串值：

JSON_EXTRACT_SCALAR(json_string,'$.Properties.Code') as Code,
JSON_EXTRACT_SCALAR(json_string,'$.Properties.Value') as Value

并且这条 SQL 将成功提取数组值：

  ARRAY(
    SELECT
      STRUCT(
        JSON_EXTRACT_SCALAR(Properties_Array,'$.Code') AS Code,
        JSON_EXTRACT_SCALAR(Properties_Array,'$.Value') AS Value
      )
    FROM UNNEST(JSON_EXTRACT_ARRAY(json_string,'$.Properties')) Properties_Array)
  AS Properties

我正在尝试找到一种方法让 BigQuery 将此字符串作为单元素数组读取，而不是对数据进行预处理。这在#StandardSQL 中可行吗？

【问题讨论】：

标签： arrays google-cloud-platform struct google-bigquery

【解决方案1】：

以下示例适用于 BigQuery 标准 SQL

#standardSQL
WITH `project.dataset.table` as (
  SELECT '{"Properties":[{"Value":"123","Code":"A"},{"Value":"000","Code":"B"}]}' json_string UNION ALL
  SELECT '{"Properties":{"Value":"456","Code":"A"}}' UNION ALL
  SELECT '{"Properties":[{"Value":"123","Code":"A"},{"Value":"789","Code":"C"},{"Value":"000","Code":"B"}]}' UNION ALL
  SELECT '{"Properties": {"Value":"Z","Code":"A"}}'  
)
SELECT json_string, 
  ARRAY(
    SELECT STRUCT(
        JSON_EXTRACT_SCALAR(Properties,'$.Code') AS Code,
        JSON_EXTRACT_SCALAR(Properties,'$.Value') AS Value
      )
    FROM UNNEST(IFNULL(
      JSON_EXTRACT_ARRAY(json_string,'$.Properties'), 
      [JSON_EXTRACT(json_string,'$.Properties')])) Properties
  ) AS Properties  
FROM `project.dataset.table`

有输出

【讨论】：

非常感谢@mikhail-berlyant！像魅力一样工作。我尝试了几种 UDF、CASE 语句和 CTE 的组合，并预感到解决方案将涉及检查空数组。