如果键不同，如何解析 JSON 数据？

Question

我正在尝试创建一个将 json 解析为 csv 格式的脚本。我在 python 中创建了一个脚本，该脚本只有在对象中存在所有键时才能正常工作。如果某些对象中的某些键不可用，则代码将被破坏。如果我添加 try catch 而不是脚本跳过 feild 值而不是为相同的键创建空值（如果不存在）。

下面的脚本不适用于下面的 json，因为 json first_name 和 last_name .. etc 包含不存在于其他列表中或仅存在于第一个索引处的 dict。 json在下面的路径上如下

https://github.com/david8981/sample-data/blob/main/data1.json

下面是python代码

import json
import csv


def parse_from_list(json_list, output_path):
    # Grab field names
    field_names = []
    for key in json_list[0].keys():
        current_key_list = set()
        found_list = False
        for obj in json_list:
            if isinstance(obj[key], dict):
                found_list = True
                for subkey in obj[key]:
                    current_key_list.add(subkey)
        if found_list:
            field_names.extend(['%s|||---|||%s' % (key, x) for x in list(current_key_list)])
        else:
            field_names.append(key)

    # Loop through objects and grab out data
    json_data = []
    for obj in json_list:
        new_data = {}
        for field in field_names:
            new_value = ''
            if '|||---|||' in field:
                if field.split('|||---|||')[0] in obj and field.split('|||---|||')[1] in obj[field.split('|||---|||')[0]]:
                    new_value = obj[field.split('|||---|||')[0]][field.split('|||---|||')[1]]
            else:
                if field in obj:
                    new_value = obj[field]
            new_data[field.replace('|||---|||', '_')] = new_value
        json_data.append(new_data)

    keys = json_data[0].keys()

    with open(output_path, 'w', newline='') as output_file:
        dict_writer = csv.DictWriter(output_file, keys)
        dict_writer.writeheader()
        dict_writer.writerows(json_data)
    pass


parse_from_list(json.load(open('data.json', 'rb'))['data']['items'], 'output.csv')

代码在以下 json 中运行良好

https://github.com/david8981/sample-data/blob/main/data2.json

Answer 1

调试您的代码非常困难，因为您的测试数据非常庞大，而且您没有提供任何关于其工作原理的解释。也就是说，问题在于它使用第一个对象中的键来确定要在列表中的所有其余对象中查找哪些键，这当然会在一个不存在时导致KeyError。

解决此问题的一种简单方法是通过检查json_list 中的对象all 中的键来“获取”字段名称。这样做将使它适用于您的两个 JSON 文件。

import json
import csv


def parse_from_list(json_list, output_path):
    # Determine field names from keys in all items in json_list.
    field_names = []
    for obj in json_list:
        for key in obj.keys():
            current_keys = set()
            found_dict = False
            if isinstance(obj[key], dict):
                found_dict = True
                for subkey in obj[key]:
                    current_keys.add(subkey)
            if found_dict:
                field_names.extend([f'{key}|||---|||{x}' for x in current_keys])
            else:
                field_names.append(key)

    # Loop through objects and grab out data
    json_data = []
    for obj in json_list:
        new_data = {}
        for field in field_names:
            new_value = ''
            if '|||---|||' in field:
                if(field.split('|||---|||')[0] in obj and
                   field.split('|||---|||')[1] in obj[field.split('|||---|||')[0]]):
                    new_value = obj[field.split('|||---|||')[0]][field.split('|||---|||')[1]]
            else:
                if field in obj:
                    new_value = obj[field]
            new_data[field.replace('|||---|||', '_')] = new_value
        json_data.append(new_data)

    keys = json_data[0].keys()

    with open(output_path, 'w', newline='') as output_file:
        dict_writer = csv.DictWriter(output_file, keys)
        dict_writer.writeheader()
        dict_writer.writerows(json_data)


with open('data1.json', 'rb') as json_file:
    json_data = json.load(json_file)
    parse_from_list(json_data['data']['items'], 'output.csv')

Answer 2

如果您的结果文件中有一组已知的字段，那么您可以手动指定它们。

data0 = [
    {"name": "a", "age": 10},
    {"name": "b", "age": 12},
]

data1 = [
    {"name": "c", "gender": "F"},
    {"name": "d", "gender": "M"},
]

import csv

expected_fields = [
    "name",
    "age",
    "gender"
]

with open("./test.csv", "w", newline="") as file_out:
    writer = csv.DictWriter(file_out, fieldnames=expected_fields)
    writer.writeheader()
    writer.writerows(data0)
    writer.writerows(data1)

如果您需要根据两个输入中的字段集动态指定字段，您可以这样做：

expected_fields = set(data0[0].keys()).union(data1[0].keys())

甚至

expected_fields = set(key for row in (data0 + data1) for key in row.keys())

鉴于您在以下方面的具体数据：

https://github.com/david8981/sample-data/blob/main/data1.json

这段代码：

import csv
import json

def flatten_dictionary(d):
    out = {}
    for key, val in d.items():
        if isinstance(val, dict):
            deeper = flatten_dictionary(val).items()
            out.update({f"{key}_{key2}": val2 for key2, val2 in deeper})
        elif isinstance(val, list):
            for subindex, subdict in enumerate(val):
                ## -----------------------
                ## dive into each item if needed
                ## -----------------------
                ## deeper = flatten_dictionary(subdict).items()
                ## out.update({f"{key}_{str(subindex)}_{key2}": val2 for key2, val2 in deeper})
                ## -----------------------
                out.update({f"{key}_{str(subindex)}": subdict})
        else:
            out[key] = val
    return out

with open("test.json", "r", encoding="utf-8") as file_in:
    users = [flatten_dictionary(row) for row in json.load(file_in)["data"]["items"]]

expected_fields = sorted(set(key for row in users for key in row.keys()))

with open("./test.csv", "w", newline="") as file_out:
    writer = csv.DictWriter(file_out, fieldnames=expected_fields)
    writer.writeheader()
    writer.writerows(users)

将为您提供包含以下列的扁平化 CSV：

albums_query
avatar_favicon
avatar_inline
avatar_message
avatar_print
avatar_profile
avatar_type
biography
date_pattern
deleted
email
email_address_privacy
first_name
followers_query
following_query
friendly_date_enabled
friendly_date_max_age
href
id
images_query
kudos_given_query
kudos_received_query
language
last_name
last_visit_time
location
login
mailbox_type
messages_query
metrics_type
online_status
online_status_privacy
personal_info_privacy
public_images_query
rank_bold
rank_icon_left
rank_id
rank_name
rank_position
rank_rank_status
rank_type
registration_data_registration_time
registration_data_type
reviews_query
roles_query
settings_query
solutions_authored_query
threads_participated_query
topics_query
type
user_badges_items_0
user_badges_items_1
user_badges_items_2
user_badges_list_item_type
user_badges_size
videos_query
view_href
web_page_url