事先不知道字段时使用 DictWriter 编写 CSV

Question

我正在将一大段文本解析成字典，最终目标是创建一个 CSV 文件，其中键作为列标题。

csv.DictWriter(csvfile, fieldnames, restval='', extrasaction='raise', dialect='excel', *args, **kwds)

问题出现是因为任何第 n 行的 dict 都可以包含一个新的、以前从未使用过的键。然后，我希望 CSV 也包含此新密钥的列。简而言之，我的所有字段都是事先不知道的，所以我无法在一开始就编译出完整的fieldnames。

有没有推荐的方法让csv.DictWriter 不忽略缺少的字段，而是将它们添加到fieldnames？此时仅更改 fieldnames 会使前面的行的字段数错误地减少。

Answer 1

我尝试使用 csv 的 writerow 方法，而不是使用 DictWriter，这在您的情况下可能会造成混淆，因为字典没有排序。 这是我所做的：

"""
a) First took all the keys of dictionary and sorted it, which is not necessary.
b) Created a result list which appends value related the headers which is key of our input dict and if key is not available then .get() will return None. 
   So result list will contain lists for rows data.
c) Wrote header and each row from result list in csv file
"""

data_dict = [{ "Header_1":"data_1", "Header_2":"data_2", "Header_3":"data_3"},
             { "Header_1":"data_4", "Header_2":"data_5", "Header_3":"data_6"},
             { "Header_1":"data_7", "Header_2":"data_8", "Header_3":"data_9", "Header_4":"data_10"},
             { "Header_1":"data_11", "Header_3":"data_12"},
             { "Header_1":"data_13", "Header_2":"data_14", "Header_3":"data_15"}]

"""
   In the third dict we have extra key, value.
   In forth we dont have have header_2 were we aspect blank value in our csv file.
"""
process_data = [ [k,v] for _dict in data_dict for k,v in _dict.iteritems() ]           

headers = [ i[0] for i in process_data ]
headers = sorted(list(set(headers)))

result = []
for _dict in data_dict:
    row = []
    for header in headers:
        row.append(_dict.get(header, None))
    result.append(row)


import csv
with open('demo.csv', 'wb') as csvfile:
    spamwriter = csv.writer(csvfile, delimiter=';', dialect='excel', 
                            quotechar='|', quoting=csv.QUOTE_MINIMAL)
    spamwriter.writerow(headers)    
    for r in result:
        spamwriter.writerow(r)

Answer 2

这是我在 Python 3 中的解决方案。它可以改进，但我发现如果您不需要排序的输出，使用 DictWriter 会更直接。

# input file:
#   {"foo": 1, "bar": 2}
#   {"foo": 3, "baz": 4}
#   {"bar": 5, "zag": 6}
#   {"foo": 7, "baz": 8, "zag": 9}
#   {"whammy": 10}

import json
import csv

FILENAME_IN = 'json_lines_in.log'
FILENAME_OUT = 'log_data.csv'

def json_to_csv_export(filename_in, filename_out):
    
    # load all records - list of dicts
    records = []
    with open(filename_in, 'r') as json_file:
        for line in json_file:
            records.append(json.loads(line))

    # 'set' ensures unique entries
    fieldnames_set = set()

    # discover field names
    for record in records:
        for field in record:
            fieldnames_set.add(field)

    # write csv
    with open(filename_out, 'w', newline='') as csv_file:
        writer = csv.DictWriter(csv_file, fieldnames=fieldnames_set, extrasaction='ignore')
        writer.writeheader()
        writer.writerows(records)

if __name__ == '__main__':
    json_to_csv_export(FILENAME_IN, FILENAME_OUT)

# output:
#  whammy,zag,foo,baz,bar
#   ,,1,,2
#   ,,3,4,
#   ,6,,,5
#   ,9,7,8,
#   10,,,,

CSV output image

Answer 3

我做了以下事情：收集标题的所有唯一值并创建这些值的列表。使用该列表，您可以使用默认值 (restval='') 来忽略不在行中的值。