【问题标题】:Snowflake - Infer Schema from JSON data in Variant Column DynamicallySnowflake - 从变体列中的 JSON 数据动态推断模式
【发布时间】:2020-10-17 14:39:28
【问题描述】:

专家,

我们有一个场景来从表中加载的 JSON 数据推断架构。它必须动态完成,并且表中的 JSON 数据也将具有不同的模式。

例子:

row 1-> address <array>[ id string ,name string ]
row 2-> address<array> [addr<object> {id:"1",name:"abc"}]
row 3-> address<array> [addr<object> {id:"2",name:"dfg",Zips<array>[zip1:6009,zip2:789]}]

我知道我们可以使用 LATERAL FLATTEN & recursive 来推断模式。但是,当我们需要切碎 Zips 时,我们需要将上述数据进行扁平化查询,如下所示。

LATERAL FLATTEN (jsondata:address ,recursive =>true) a
LATERAL FLATTEN (a.value:addr,recursive =>true) b -> this is causing issue
LATERAL FLATTEN (c.value:Zips,recursive =>true) c 

当我们展平对象数据类型时,它会展平到元素级别,有没有办法检查并动态避免对象展平。

问候, 戈皮

【问题讨论】:

  • 我对要求并不完全清楚。我的一位同事编写了一个存储过程来读取变量列并在其之上创建一个视图。我重构并概括了代码。这符合您的需要吗?如果是这样,我可以将您定向到源 SP 的博客条目和我的重构代码。
  • @GregPavlik - 感谢您的回复。是的。然而,这有点棘手,其中 VARIANT 列将包含我的示例中提到的多个数据组合(ARRAY、OBJECT、STRING)。我需要为每个数组和对象创建一个单独的视图。我确实提到了其中一篇博客 - support.snowflake.net/s/article/…。如果您引用了不同的内容,请分享。
  • 你找到了我引用的博客。该博客中的 SP 和我重构的代码都不会提取对象并推断其中的数据类型。它可以作为做类似事情的起点。

标签: snowflake-cloud-data-platform flatten


【解决方案1】:

Snowflake 的semi-structured data query 功能提供了data-type inspection functions,可用于有条件地处理单个表列中的此类不同输入。

特别是,在分解outer arrays into whole rows之后,你可以使用IS_ARRAYIS_OBJECT: operator(带有NULL结果检查)函数来分离记录产生逻辑,然后组合使用UNION ALL 将行放入单个输出中。

问题lacks a clear/usable sample or schema of data (and an expected output) 所以我在下面对数据的外观做了四个假设,并从根中为每种类型添加了一个过滤器。每个的大体思路都是一样的(检查类型,划分数据集,处理每种类型),你应该能够推断和调整。

WITH tbl AS (
-- Sample table data

  select parse_json('{"address": [["sa_id1", "sa_name1"], ["sa_id2", "sa_name2"]], "other_outer_field": 1}') jsondata
  union all
  select parse_json('{"address": [{"id": "sr_id1", "name": "sr_name1"}, {"id": "sr_id2", "name": "sr_name2"}], "other_outer_field": 2}') jsondata
  union all
  select parse_json('{"address": [{"id": "zr_id1", "name": "zr_name1", "zips": ["10001", "10002", "10003"]}, {"id": "zr_id2", "name": "zr_name2", "zips": ["20001", "20002"]}], "other_outer_field": 3}') jsondata
  union all
  select parse_json('{"address": {"id": "zr_id1", "name": "zr_name1", "zips": ["10001", "10002", "10003"]}, "other_outer_field": 4}') jsondata

), all_address_array_formats AS (
-- Table's actual row: { …, "address": [ … ], … } when the address field is an array

  SELECT
    jsondata:other_outer_field AS o_f,
    each_address.value AS address_container
  FROM tbl, LATERAL FLATTEN(jsondata:address) each_address
  WHERE IS_ARRAY(jsondata:address)
  
), all_address_object_formats AS (
-- Table's actual row: { …, "address": { … }, … } when the address field is an object (we do not need to flatten here)

  SELECT
    jsondata:other_outer_field AS o_f,
    jsondata:address AS address_container
  FROM tbl
  WHERE IS_OBJECT(jsondata:address)

), just_array_members AS (
-- For address array with nested arrays: [ [id1, name1], [id2, name2], … ]

  SELECT
    o_f,
    address_container[0]::varchar AS id,
    address_container[1]::varchar AS name,
    NULL AS zipcode
  FROM all_address_array_formats
  WHERE
    IS_ARRAY(address_container)

), simple_record_members AS (
-- For address array with objects, but no zipcode fields: [ { id: id1, name: name1 }, { id: id2, name: name2 }, … ]

  SELECT
    o_f,
    address_container:id::varchar AS id,
    address_container:name::varchar AS name,
    NULL AS zipcode
  FROM all_address_array_formats
  WHERE
    IS_OBJECT(address_container)
    AND address_container:zips IS NULL

), zipcode_record_members AS (
-- For address array with objects, each with multiple zipcodes: [ { id: id1, name: name1, zips: [ zip1_1, zip1_2, … ] }, { id: id2, name: name2, zips: [zip2_1, zip2_2, …] }, … ]

  SELECT
    o_f,
    address_container:id::varchar AS id,
    address_container:name::varchar AS name,
    per_zip.value::varchar AS zipcode
  FROM all_address_array_formats, LATERAL FLATTEN(address_container:zips) per_zip
  WHERE
    IS_OBJECT(address_container)
    AND address_container:zips IS NOT NULL

), zipcodes_within_object AS (
-- For address of object type, a single one with multiple zipcodes: { id: id1, name: name1, zips: [ zip1_1, zip1_2, … ] }

  SELECT
    o_f,
    address_container:id::varchar AS id,
    address_container:name::varchar AS name,
    per_zip.value::varchar AS zipcode
  FROM all_address_object_formats, LATERAL FLATTEN(address_container:zips) per_zip
  WHERE
    IS_OBJECT(address_container)
    AND address_container:zips IS NOT NULL
  
)
SELECT o_f, id, name, zipcode FROM just_array_members UNION ALL
SELECT o_f, id, name, zipcode FROM simple_record_members UNION ALL
SELECT o_f, id, name, zipcode FROM zipcode_record_members UNION ALL
SELECT o_f, id, name, zipcode FROM zipcodes_within_object;

注意:该示例还展示了如何从表中的原始对象(列:o_f)中继续携带除 address 之外的任何其他字段,这些字段未通过展平功能。

对于输入:

+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+                                                   
| JSONDATA                                                                                                                                                                           |
|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| {"address": [["sa_id1", "sa_name1"], ["sa_id2", "sa_name2"]], "other_outer_field": 1}                                                                                              |
| {"address": [{"id": "sr_id1", "name": "sr_name1"}, {"id": "sr_id2", "name": "sr_name2"}], "other_outer_field": 2}                                                                  |
| {"address": [{"id": "zr_id1", "name": "zr_name1", "zips": ["10001", "10002", "10003"]}, {"id": "zr_id2", "name": "zr_name2", "zips": ["20001", "20002"]}], "other_outer_field": 3} |
| {"address": {"id": "zr_id1", "name": "zr_name1", "zips": ["10001", "10002", "10003"]}, "other_outer_field": 4}                                                                     |
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

这会产生:

+-----+--------+----------+---------+                                           
| O_F | ID     | NAME     | ZIPCODE |
|-----+--------+----------+---------|
| 1   | sa_id1 | sa_name1 | NULL    |
| 1   | sa_id2 | sa_name2 | NULL    |
| 2   | sr_id1 | sr_name1 | NULL    |
| 2   | sr_id2 | sr_name2 | NULL    |
| 3   | zr_id1 | zr_name1 | 10001   |
| 3   | zr_id1 | zr_name1 | 10002   |
| 3   | zr_id1 | zr_name1 | 10003   |
| 3   | zr_id2 | zr_name2 | 20001   |
| 3   | zr_id2 | zr_name2 | 20002   |
| 4   | zr_id1 | zr_name1 | 10001   |
| 4   | zr_id1 | zr_name1 | 10002   |
| 4   | zr_id1 | zr_name1 | 10003   |
+-----+--------+----------+---------+

【讨论】:

  • @ Kirby - 感谢您提供详细的答案,这与我正在寻找的内容相似。然而,我们所拥有的场景是 - 动态地将具有多模式的 JSON 数据框架/切碎到多视图中。在下一条评论中继续
  • 在您的示例中只是一个小添加 - Address (Array) 然后我们有 ADDR (Object) 然后只有我们有 Zips(Array) 。在这个场景中从 Zips 中获取值 - 我应该有两个 flatten - FROM,LATERAL FLATTEN (jsondata:address) a,LATERAL FLATTEN (a.value:ADDR.Zips) -> 当我从 caluse 动态地构建这个时怎么办避免展平 ADDR 并直接使用 Zips(我们这样做是通过 Thro 循环),但是我想了解是否有其他方法。如果我的理解不正确,请纠正我。
猜你喜欢
  • 2023-04-03
  • 2023-01-27
  • 2021-06-16
  • 2016-11-03
  • 2020-10-03
  • 2021-10-28
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
相关资源
最近更新 更多