【问题标题】:Reading Multiple levels of repeated JSON structure data in Google BigQuery在 Google BigQuery 中读取多级重复 JSON 结构数据
【发布时间】:2020-03-30 16:45:58
【问题描述】:

我正在尝试阅读以下 JSON 结构。能够将数据提高到 1 级并正确查看其读数,但是当我深入了解其他部分时,它无法正确读取数据。下面是输入数据和我正在使用的代码和输出供您阅读。请求您的帮助。

  • 输入数据:

    { "ProjectId": "P.2000002", "OperationId": "O.2000002.01", "ActivityId": "A.2000002.01.01", “描述”: ””, “组合”:[ { "ComboId": "9146", “需要”: { “DownHoleTools”:{ “主要工具”:[ { "工具代码": "19139", "工具描述": "VSIT-C/D", “IsEdoApplicable”:是的, “来源”:“A”, “需求持续时间信息”:{ "开始日期": "2019-09-09T17:42:10", “结束日期”:“2019-09-19T23:00:00” }, “哈希数”:1, "分类名称": "VSIT-C/D", “分类类型”:0, “组信息”:{ “代码”:“1519”, “描述”:“VSIT” }, “类别信息”:{ “代码”:“1519”, “描述”:“VSIT” }, “注释”: ””, “已删除”:假, “零件号”: ””, “描述”: ””, "创建日期": "0001-01-01T00:00:00", “由...制作”: ””, "LastModifiedDate": "0001-01-01T00:00:00", "LastModifiedBy": "", “身份证”:“1” }, { "工具代码": "7030", "工具描述": "VSIB-P", “IsEdoApplicable”:假, “来源”:“A”, “需求持续时间信息”:{ "开始日期": "2019-09-09T17:42:12", “结束日期”:“2019-09-19T23:00:00” }, “哈希数”:1, "分类名称": "VSIB-P", “分类类型”:0, “组信息”:{ “代码”:“1519”, “描述”:“VSIT” }, “类别信息”:{ “代码”:“1519”, “描述”:“VSIT” }, “注释”: ””, “已删除”:假, “零件号”: ””, “描述”: ””, "创建日期": "0001-01-01T00:00:00", “由...制作”: ””, "LastModifiedDate": "0001-01-01T00:00:00", "LastModifiedBy": "", “身份证”:“1” }, { "工具代码": "3707", "工具描述": "HILT-TLD-H", “IsEdoApplicable”:假, “来源”:“A”, “需求持续时间信息”:{ "开始日期": "2020-02-12T15:18:32", “结束日期”:“2020-02-13T15:18:32” }, “哈希数”:1, "分类名称": "HILT-TLD-H", “分类类型”:0, “组信息”:{ “代码”:“842”, “描述”:“HILT” }, “类别信息”:{ “代码”:“842”, “描述”:“HILT” }, “注释”: ””, “已删除”:假, “零件号”: ””, “描述”:“HILT-TLD-H”, "创建日期": "0001-01-01T00:00:00", “由...制作”: ””, "LastModifiedDate": "0001-01-01T00:00:00", "LastModifiedBy": "", “身份证”:“1” }, { "工具代码": "3707", "工具描述": "HILT-TLD-H", “IsEdoApplicable”:假, “来源”:“A”, “需求持续时间信息”:{ "开始日期": "2020-02-12T15:18:32", “结束日期”:“2020-02-13T15:18:32” }, “哈希数”:2, "分类名称": "HILT-TLD-H", “分类类型”:0, “组信息”:{ “代码”:“842”, “描述”:“HILT” }, “类别信息”:{ “代码”:“842”, “描述”:“HILT” }, “注释”: ””, “已删除”:假, “零件号”: ””, “描述”:“HILT-TLD-H”, "创建日期": "0001-01-01T00:00:00", “由...制作”: ””, "LastModifiedDate": "0001-01-01T00:00:00", "LastModifiedBy": "", “身份证”:“1” } ], “备份工具”:[

        ]
      },
      "SurfaceTools": {
        "PrimaryTools": [
          {
            "ToolCode": "19153",
            "ToolDescription": "MDT_Surface Eqpt",
            "IsEdoApplicable": false,
            "Source": "A",
            "DemandDurationInfo": {
              "StartDate": "2020-02-12T15:18:32",
              "EndDate": "2020-02-13T15:18:32"
            },
            "HashNumber": 1,
            "ClassificationName": "MDT_Surface Eqpt",
            "ClassificationType": 1,
            "GroupInfo": {
              "Code": "965",
              "Description": "MDT Accessories"
            },
            "CategoryInfo": {
              "Code": "965",
              "Description": "MDT Accessories"
            },
            "Comments": "",
            "IsDeleted": false,
            "PartNumber": "",
            "Description": "MDT_Surface Eqpt",
            "CreatedDate": "0001-01-01T00:00:00",
            "CreatedBy": "",
            "LastModifiedDate": "0001-01-01T00:00:00",
            "LastModifiedBy": "",
            "Id": "1"
          },
          {
            "ToolCode": "19153",
            "ToolDescription": "MDT_Surface Eqpt",
            "IsEdoApplicable": false,
            "Source": "A",
            "DemandDurationInfo": {
              "StartDate": "2020-02-12T15:18:32",
              "EndDate": "2020-02-13T15:18:32"
            },
            "HashNumber": 2,
            "ClassificationName": "MDT_Surface Eqpt",
            "ClassificationType": 1,
            "GroupInfo": {
              "Code": "965",
              "Description": "MDT Accessories"
            },
            "CategoryInfo": {
              "Code": "965",
              "Description": "MDT Accessories"
            },
            "Comments": "",
            "IsDeleted": false,
            "PartNumber": "",
            "Description": "MDT_Surface Eqpt",
            "CreatedDate": "0001-01-01T00:00:00",
            "CreatedBy": "",
            "LastModifiedDate": "0001-01-01T00:00:00",
            "LastModifiedBy": "",
            "Id": "1"
          }
        ],
        "BackupTools": [
    
        ]
      },
      "Techniques": {
        "PrimaryTools": [
    
        ],
        "BackupTools": [
    
        ]
      },
      "Services": [
    
      ],
      "Tools": ""
    },
    "ComboType": 2,
    "HashCode": "",
    "SequenceNumber": "",
    "ConveyanceInfo": "",
    "CreatedDate": "0001-01-01T00:00:00",
    "CreatedBy": "",
    "LastModifiedDate": "0001-01-01T00:00:00",
    "LastModifiedBy": "",
    "Id": "98e9418f-e50a-417b-affb-5fc4c1f71f39"
    

    }, { "ComboId": "5970", “需要”: { “DownHoleTools”:{ “PrimaryTools”:[

        ],
        "BackupTools": [
    
        ]
      },
      "SurfaceTools": {
        "PrimaryTools": [
    
        ],
        "BackupTools": [
    
        ]
      },
      "Techniques": {
        "PrimaryTools": [
    
        ],
        "BackupTools": [
    
        ]
      },
      "Services": [
    
      ],
      "Tools": ""
    },
    "ComboType": 1,
    "HashCode": "",
    "SequenceNumber": "",
    "ConveyanceInfo": "",
    "CreatedDate": "0001-01-01T00:00:00",
    "CreatedBy": "",
    "LastModifiedDate": "0001-01-01T00:00:00",
    "LastModifiedBy": "",
    "Id": "944cf025-2a8c-4372-9f87-6c80c844ac68"
    

    }, { "ComboId": "5971", “需要”: { “DownHoleTools”:{ “PrimaryTools”:[

        ],
        "BackupTools": [
    
        ]
      },
      "SurfaceTools": {
        "PrimaryTools": [
    
        ],
        "BackupTools": [
    
        ]
      },
      "Techniques": {
        "PrimaryTools": [
    
        ],
        "BackupTools": [
    
        ]
      },
      "Services": [
    
      ],
      "Tools": ""
    },
    "ComboType": 0,
    "HashCode": "",
    "SequenceNumber": "",
    "ConveyanceInfo": "",
    "CreatedDate": "0001-01-01T00:00:00",
    "CreatedBy": "",
    "LastModifiedDate": "0001-01-01T00:00:00",
    "LastModifiedBy": "",
    "Id": "0a9338b2-aa95-4d5a-8e57-1305e78fec0c"
    

    }, { "ComboId": "26793", “需要”: { “DownHoleTools”:{ “PrimaryTools”:[

        ],
        "BackupTools": [
    
        ]
      },
      "SurfaceTools": {
        "PrimaryTools": [
    
        ],
        "BackupTools": [
    
        ]
      },
      "Techniques": {
        "PrimaryTools": [
    
        ],
        "BackupTools": [
    
        ]
      },
      "Services": [
        {
          "Code": "GIWS",
          "Name": "Grease Injection WHE Service",
          "Description": "",
          "GroupInfos": "",
          "ClassificationType": 4,
          "LegacySystemMapping": [
            {
              "LegacyId": "EE61B186-CE57-46A8-B280-FE913CC8FF33",
              "LegacySystemMappedProperty": "Grease Injection WHE Service",
              "LegacySystemName": "ODM"
            },
            {
              "LegacyId": "2e4b1aea-3d91-43b6-9c32-a165a546ed39",
              "LegacySystemMappedProperty": "Grease Injection WHE Service",
              "LegacySystemName": "OSCompliance"
            }
          ],
          "Source": "A",
          "Comments": "",
          "Id": "9c405cc1-5231-4baf-864f-7974bb4fbe07"
        },
        {
          "Code": "SCNNGWS",
          "Name": "Slick Cable Non-Grease Injection WHE Service",
          "Description": "",
          "GroupInfos": "",
          "ClassificationType": 4,
          "LegacySystemMapping": [
            {
              "LegacyId": "7B41D3DD-A6A7-47AF-81CD-5D8C248183B5",
              "LegacySystemMappedProperty": "Slick Cable Non-Grease Injection WHE Service",
              "LegacySystemName": "ODM"
            },
            {
              "LegacyId": "399e61ed-353a-404f-aad3-1e84c46cb273",
              "LegacySystemMappedProperty": "Slick Cable Non-Grease Injection WHE Service",
              "LegacySystemName": "OSCompliance"
            }
          ],
          "Source": "A",
          "Comments": "",
          "Id": "0a4848d3-7d93-4721-a556-2ecf1b0a7f43"
        },
        {
          "Code": "TPWS",
          "Name": "Third Party WHE Service",
          "Description": "",
          "GroupInfos": "",
          "ClassificationType": 4,
          "LegacySystemMapping": [
            {
              "LegacyId": "CAF5754A-3F90-40C6-82AC-0A4F484A4E74",
              "LegacySystemMappedProperty": "Third Party WHE Service",
              "LegacySystemName": "ODM"
            },
            {
              "LegacyId": "50ff1668-0b3b-489c-a7ad-794e327028e5",
              "LegacySystemMappedProperty": "Third Party WHE Service",
              "LegacySystemName": "OSCompliance"
            }
          ],
          "Source": "A",
          "Comments": "",
          "Id": "ee923f9e-210f-4c36-8fc4-f1e9521d0cbe"
        },
        {
          "Code": "WLPPS",
          "Name": "Wireline Low Pressure Packoff Service",
          "Description": "",
          "GroupInfos": "",
          "ClassificationType": 4,
          "LegacySystemMapping": [
            {
              "LegacyId": "CB79CDE1-2271-418D-9E67-D4B07E94AC61",
              "LegacySystemMappedProperty": "Wireline Low Pressure Packoff Service",
              "LegacySystemName": "ODM"
            },
            {
              "LegacyId": "514650dd-5b71-4889-b991-0d77dd355666",
              "LegacySystemMappedProperty": "Wireline Low Pressure Packoff Service",
              "LegacySystemName": "OSCompliance"
            }
          ],
          "Source": "A",
          "Comments": "",
          "Id": "2216c1c0-41ba-40aa-90c2-1096e8191d2c"
        }
      ],
      "Tools": ""
    },
    "ComboType": 2,
    "HashCode": "",
    "SequenceNumber": "",
    "ConveyanceInfo": "",
    "CreatedDate": "0001-01-01T00:00:00",
    "CreatedBy": "",
    "LastModifiedDate": "0001-01-01T00:00:00",
    "LastModifiedBy": "",
    "Id": "446d43e1-6476-4408-b850-e5f233933ba9"
    

    } ], "创建日期": "2019-09-09T13:12:14.94", “创建者”:“VHiremath”, "LastModifiedDate": "2020-02-27T07:40:08.071", "LastModifiedBy": "VPanath", “ID”:“5d764fae3d6a351088a1c9d3” }

代码:

CREATE TEMPORARY FUNCTION CUSTOM_JSON_EXTRACT(json STRING, json_path STRING)
RETURNS ARRAY<STRING>
LANGUAGE js AS """
  var result = jsonPath(JSON.parse(json), json_path);
  if(result){return result;} 
  else {return [];}
"""
OPTIONS (
    library="gs://temp-dev-workspace/json_temp/jsonpath-0.8.0.js"
);

SELECT job_id,
combo_id,
tool_code
from temp.dbm_eqp_data,
UNNEST(CUSTOM_JSON_EXTRACT(conv_column, '$.Combos[*].ComboId')) combo_id,
UNNEST(CUSTOM_JSON_EXTRACT(conv_column, '$.Combos[*].Demands.DownHoleTools.PrimaryTools[*].ToolCode')) tool_code
;

预期输出:

【问题讨论】:

  • @Mikhail Berlyant - 在这里请求您的专业帮助,我已经尝试了之前建议的所有可能的解决方案,但我在这里遗漏了一些东西。请求您的帮助
  • 1.您只能联系已经参与该帖子的 SO 成员 - 在这种情况下,他们会在收件箱中收到您的“消息”。否则将被忽略。 2. 你需要将你的问题缩小到易于使用/使用的东西。尝试简化(缩短)您的示例并解释预期结果是什么
  • @MikhailBerlyant - 非常感谢,你永远是一个救世主......我已经给出了示例输入数据和预期的输出以及我目前需要处理的代码。请求您的帮助。
  • 我会尽快检查:o)
  • @MikhailBerlyant - 再次非常感谢

标签: json google-bigquery unnest json-extract


【解决方案1】:

以下是 BigQuery 标准 SQL

#standardSQL
CREATE TEMPORARY FUNCTION CUSTOM_JSON_EXTRACT(json STRING, json_path STRING)
RETURNS ARRAY<STRING>
LANGUAGE js AS '''
  var result = jsonPath(JSON.parse(json), json_path);
  if(result){return result;} 
  else {return [];}
'''
OPTIONS (
    library="gs://temp-dev-workspace/json_temp/jsonpath-0.8.0.js"
);
SELECT 
  job_id,
  combo_id,
  tool_code
FROM `temp.dbm_eqp_data`,
UNNEST(CUSTOM_JSON_EXTRACT(conv_column, '$.Combos[*].ComboId')) combo_id
LEFT JOIN UNNEST(CUSTOM_JSON_EXTRACT(conv_column, '$.Combos[?(@.ComboId=="' || combo_id || '")].Demands.DownHoleTools.PrimaryTools[*].ToolCode')) tool_code

如果应用于您问题的样本数据 - 输出是

Row job_id  combo_id    tool_code    
1   1       9146        19139    
2   1       9146        7030     
3   1       9146        3707     
4   1       9146        3707     
5   1       5970        null     
6   1       5971        null     
7   1       26793       null     

【讨论】:

  • 非常感谢,我已经进一步扩展它以读取每个工具代码的其他属性,它工作得非常好。
  • 我试图进一步扩展它以读取所有属性,它只是执行并且从不返回任何结果。我在这里做错了。代码: UNNEST(CUSTOM_JSON_EXTRACT(conv_column, '$.Equipments.Combos[*].ComboId')) combo_id LEFT JOIN UNNEST(CUSTOM_JSON_EXTRACT(conv_column, '$.Combos[?(@.ComboId=="' || combo_id || '")].Demands.DownHoleTools.PrimaryTools[*].ToolCode')) tool_code LEFT JOIN UNNEST(CUSTOM_JSON_EXTRACT(conv_column, '$.Combos[?(@.ComboId=="' || combo_id || '")] .Demands.DownHoleTools.PrimaryTools[?(@.ToolCode=="' || tool_code || '")].ToolDescription')) tool_description
  • cmets 不是获取丰富信息的好媒体 - 无论如何 - 您当前的问题已完全回答 - 如果您有新问题 - 请发布新问题,我(我们)将很乐意提供帮助。这就是它在 SO 的工作方式:o)
  • 当然,我会发布一个新问题。再次非常感谢
  • 等一下 - 我觉得你只是拼写错误 - 给我等一下
猜你喜欢
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 2016-04-25
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 2018-07-17
相关资源
最近更新 更多