将以下 json 转换为 csv答案

【问题标题】：convert the following json to csv将以下 json 转换为 csv
【发布时间】：2019-05-23 06:03:29
【问题描述】：

我想使用 Python 将 3 GB 的 JSON 数据转换为 CSV 格式。我编写的这段代码将数据转换为 CSV，但将其存储在单个单元格中。我不想要“相关”字段（使用 re 删除）。

JSON 格式

{'asin': '0001048791', 'salesRank': {'Books': 6334800}, 'imUrl': 'http://ecx.images-amazon.com/images/I/51MKP0T4DBL.jpg', 'categories': [['Books']], 'title': 'The Crucible: Performed by Stuart Pankin, Jerome Dempsey &amp; Cast'}
{'asin': '0000143561', 'categories': [['Movies & TV', 'Movies']], 'description': '3Pack DVD set - Italian Classics, Parties and Holidays.', 'title': 'Everyday Italian (with Giada de Laurentiis), Volume 1 (3 Pack): Italian Classics, Parties, Holidays', 'price': 12.99, 'salesRank': {'Movies & TV': 376041}, 'imUrl': 'http://g-ecx.images-amazon.com/images/G/01/x-site/icons/no-img-sm._CB192198896_.gif', 'related': {'also_viewed': ['B0036FO6SI', 'B000KL8ODE', '000014357X', 'B0037718RC', 'B002I5GNVU', 'B000RBU4BM'], 'buy_after_viewing': ['B0036FO6SI', 'B000KL8ODE', '000014357X', 'B0037718RC']}}
{'asin': '0000037214', 'related': {'also_viewed': ['B00JO8II76', 'B00DGN4R1Q', 'B00E1YRI4C']}, 'title': 'Purple Sequin Tiny Dancer Tutu Ballet Dance Fairy Princess Costume Accessory', 'price': 6.99, 'salesRank': {'Clothing': 1233557}, 'imUrl': 'http://ecx.images-amazon.com/images/I/31mCncNuAZL.jpg', 'brand': 'Big Dreams', 'categories': [['Clothing, Shoes & Jewelry', 'Girls'], ['Clothing, Shoes & Jewelry', 'Novelty, Costumes & More', 'Costumes & Accessories', 'More Accessories', 'Kids & Baby']]}
{ "asin": "0000031852", "title": "Girls Ballet Tutu Zebra Hot Pink", "price": 3.17, "imUrl": "http://ecx.images-amazon.com/images/I/51fAmVkTbyL._SY300_.jpg", "related": { "also_bought": ["B00JHONN1S", "B002BZX8Z6", "B00D2K1M3O", "0000031909", "B00613WDTQ", "B00D0WDS9A", "B00D0GCI8S", "0000031895", "B003AVKOP2", "B003AVEU6G", "B003IEDM9Q", "B002R0FA24", "B00D23MC6W", "B00D2K0PA0", "B00538F5OK", "B00CEV86I6", "B002R0FABA", "B00D10CLVW", "B003AVNY6I", "B002GZGI4E", "B001T9NUFS", "B002R0F7FE", "B00E1YRI4C", "B008UBQZKU", "B00D103F8U", "B007R2RM8W"], "also_viewed": ["B002BZX8Z6", "B00JHONN1S", "B008F0SU0Y", "B00D23MC6W", "B00AFDOPDA", "B00E1YRI4C", "B002GZGI4E", "B003AVKOP2", "B00D9C1WBM", "B00CEV8366", "B00CEUX0D8", "B0079ME3KU", "B00CEUWY8K", "B004FOEEHC", "0000031895", "B00BC4GY9Y", "B003XRKA7A", "B00K18LKX2", "B00EM7KAG6", "B00AMQ17JA", "B00D9C32NI", "B002C3Y6WG", "B00JLL4L5Y", "B003AVNY6I", "B008UBQZKU", "B00D0WDS9A", "B00613WDTQ", "B00538F5OK", "B005C4Y4F6", "B004LHZ1NY", "B00CPHX76U", "B00CEUWUZC", "B00IJVASUE", "B00GOR07RE", "B00J2GTM0W", "B00JHNSNSM", "B003IEDM9Q", "B00CYBU84G", "B008VV8NSQ", "B00CYBULSO", "B00I2UHSZA", "B005F50FXC", "B007LCQI3S", "B00DP68AVW", "B009RXWNSI", "B003AVEU6G", "B00HSOJB9M", "B00EHAGZNA", "B0046W9T8C", "B00E79VW6Q", "B00D10CLVW", "B00B0AVO54", "B00E95LC8Q", "B00GOR92SO", "B007ZN5Y56", "B00AL2569W", "B00B608000", "B008F0SMUC", "B00BFXLZ8M"], "bought_together": ["B002BZX8Z6"] }, "salesRank": {"Toys & Games": 211836}, "brand": "Coxlures", "categories": [["Sports & Outdoors", "Other Sports", "Dance"]] } 

import json
import re 
with open('metadata.json') as f:
    x = f.read()
    data = re.sub("\'related\': {","",x)
    data = re.sub("]},", "]", data)
    data = re.sub("}\n{", "},\n{", data)
    print("1")
    print(type(data))
    data = json.dumps(data)
    print("2")
    print(type(data))
    data = json.loads("[" + data + "]")
    print("3")
    print(type(data))
    print("done")


import pandas 
pandas.read_json(data.to_csv('metadata.csv')

我希望有一个正确的 CSV 文件，其中包含字段 [asin, title, categories, price, also_viewed, also_bought, brand]

【问题讨论】：

请在问题中添加代码，而不是在评论中！
我不知道你在问什么......“将其存储在单个单元格中”是什么意思？你想达到什么目的？您能否提供一个小（！）JSON 文件的内容、您的实际（部分）工作代码、您的预期输出和实际输出？如果您不确定如何操作，请阅读How to create a Minimal, Reproducible Example。
0 {'asin': '0001048791', 'salesRank': {'Books': 6334800}, 'imUrl': 'ecx.images-amazon.com/images/I/51MKP0T4DBL.jpg', 'categories': [['Books'] ]，“标题”：“坩埚：由 Stuart Pankin、Jerome Dempsey 和Cast'}, {'asin': '0000143561', 'categories': [['Movies & TV', 'Movies']], 'description': '3Pack DVD set - Italian Classics, Party and Holidays.', the csv 格式的数据进入单个单元格，不创建列和行
{'asin': '0132793040', 'imUrl': 'http/images/IL.jpg', 'description': '需要', 'categories': [['Electronics' , 'Computer', 'Cables & Accessories', 'Monitor Accessories']], 'title': 'Kelby Training DVD: Mastering Blend Modes in Adobe Photoshop CS5 By Corey Barker'} {'asin': '0321732944', 'imUrl ': 'ecx.images-amazon.com/images/I/31uogm6YcrL.jpg', 'categories': [['Electronics', 'Computers & Accessories', 'Cables & Accessories', 'Monitor Accessories']], 'title': 'Kelby Training DVD: Adobe Photoshop CS5 By Matt Kloskowski'} 我需要具有上述字段的数据的 csv 格式
请在 Q(Que/Problem) 中至少显示第一个 2-3 个 json 对象。不在 cmets 中。您的文件似乎不是有效的 JSON 文件。由于所有对象/字典都不在数组/列表中（我可以从您上面粘贴的内容中获取）。如果您粘贴了错误的 JSON，您能否在 Q 中添加它，可能是您忘记在其中的第一个或最后添加括号。代替 {... }{... }，它应该是 [{...}{...}] 或 {0: {... }, 1: {... }} 或任何其他有效的 JSON 格式。

标签： python json python-3.x csv

【解决方案1】：

好的，如果我理解正确，您有一个 JSON 文件，其中包含一个包含上述字段的对象数组。您希望将这些字段中的一些存储到 CSV 文件中。 JSON 数据包含一个字段“related”，其中包含子字段“also_viewed”和“also_bought”。您希望后两个子字段作为 CSV 文件中的单独列，并且不希望在 CSV 文件中有“相关”列。对？如果我误解了你，请尝试澄清你的问题。

Python 可以使用标准库模块（json 和 csv）处理 JSON 和 CSV 数据。但是，您必须自己编写“粘合逻辑”。我不建议对 JSON 数据执行正则表达式，因为它非常容易出错。此外，您当前的代码对数据执行多个操作，并且在内存中有多个数据副本，因此它可能会很慢并且您很快就会耗尽内存。

下面是一个基于标准 Python 模块的工作示例。但是，我不知道您的输入数据的格式（您问题中的 JSON 数据不是有效的 JSON 格式！）而且我不知道您想以哪种格式输出列表。我做了一些假设，但如果您期望不同的输出格式，您可能需要更改代码。但是代码确实输出了有效的 CSV 数据，可以在电子表格应用程序（例如 LibreOffice Calc）中打开。

请注意，此版本还一次加载 JSON 文件。它比您当前的实现更节省内存，但如果 JSON 文件为 3 GB，仍会消耗大量内存。如果您有足够的内存，这不是问题（只要您使用的是 64 位版本的 Python）。但是，您也可以实现基于流式 JSON 解析器的解析器（我已经使用 ijson 让它工作，但它的代码要多一些，我不确定是否应该将它作为答案发布）。这将大大减少内存消耗。 CSV 文件已经逐行写入，因此不需要太多内存。

所以，现在是代码:-)

我假设你的 JSON 数据是这样的：

[
{"asin": "0001048791", "salesRank": {"Books": 6334800}, "imUrl": "http://ecx.images-amazon.com/images/I/51MKP0T4DBL.jpg", "categories": [["Books"]], "title": "The Crucible: Performed by Stuart Pankin, Jerome Dempsey &amp; Cast"},
{"asin": "0000143561", "categories": [["Movies & TV", "Movies"]], "description": "3Pack DVD set - Italian Classics, Parties and Holidays.", "title": "Everyday Italian (with Giada de Laurentiis), Volume 1 (3 Pack): Italian Classics, Parties, Holidays", "price": 12.99, "salesRank": {"Movies & TV": 376041}, "imUrl": "http://g-ecx.images-amazon.com/images/G/01/x-site/icons/no-img-sm._CB192198896_.gif", "related": {"also_viewed": ["B0036FO6SI", "B000KL8ODE", "000014357X", "B0037718RC", "B002I5GNVU", "B000RBU4BM"], "buy_after_viewing": ["B0036FO6SI", "B000KL8ODE", "000014357X", "B0037718RC"]}},
{"asin": "0000037214", "related": {"also_viewed": ["B00JO8II76", "B00DGN4R1Q", "B00E1YRI4C"]}, "title": "Purple Sequin Tiny Dancer Tutu Ballet Dance Fairy Princess Costume Accessory", "price": 6.99, "salesRank": {"Clothing": 1233557}, "imUrl": "http://ecx.images-amazon.com/images/I/31mCncNuAZL.jpg", "brand": "Big Dreams", "categories": [["Clothing, Shoes & Jewelry", "Girls"], ["Clothing, Shoes & Jewelry", "Novelty, Costumes & More", "Costumes & Accessories", "More Accessories", "Kids & Baby"]]},
{"asin": "0000031852", "title": "Girls Ballet Tutu Zebra Hot Pink", "price": 3.17, "imUrl": "http://ecx.images-amazon.com/images/I/51fAmVkTbyL._SY300_.jpg", "related": { "also_bought": ["B00JHONN1S", "B002BZX8Z6", "B00D2K1M3O", "0000031909", "B00613WDTQ", "B00D0WDS9A", "B00D0GCI8S", "0000031895", "B003AVKOP2", "B003AVEU6G", "B003IEDM9Q", "B002R0FA24", "B00D23MC6W", "B00D2K0PA0", "B00538F5OK", "B00CEV86I6", "B002R0FABA", "B00D10CLVW", "B003AVNY6I", "B002GZGI4E", "B001T9NUFS", "B002R0F7FE", "B00E1YRI4C", "B008UBQZKU", "B00D103F8U", "B007R2RM8W"], "also_viewed": ["B002BZX8Z6", "B00JHONN1S", "B008F0SU0Y", "B00D23MC6W", "B00AFDOPDA", "B00E1YRI4C", "B002GZGI4E", "B003AVKOP2", "B00D9C1WBM", "B00CEV8366", "B00CEUX0D8", "B0079ME3KU", "B00CEUWY8K", "B004FOEEHC", "0000031895", "B00BC4GY9Y", "B003XRKA7A", "B00K18LKX2", "B00EM7KAG6", "B00AMQ17JA", "B00D9C32NI", "B002C3Y6WG", "B00JLL4L5Y", "B003AVNY6I", "B008UBQZKU", "B00D0WDS9A", "B00613WDTQ", "B00538F5OK", "B005C4Y4F6", "B004LHZ1NY", "B00CPHX76U", "B00CEUWUZC", "B00IJVASUE", "B00GOR07RE", "B00J2GTM0W", "B00JHNSNSM", "B003IEDM9Q", "B00CYBU84G", "B008VV8NSQ", "B00CYBULSO", "B00I2UHSZA", "B005F50FXC", "B007LCQI3S", "B00DP68AVW", "B009RXWNSI", "B003AVEU6G", "B00HSOJB9M", "B00EHAGZNA", "B0046W9T8C", "B00E79VW6Q", "B00D10CLVW", "B00B0AVO54", "B00E95LC8Q", "B00GOR92SO", "B007ZN5Y56", "B00AL2569W", "B00B608000", "B008F0SMUC", "B00BFXLZ8M"], "bought_together": ["B002BZX8Z6"] }, "salesRank": {"Toys & Games": 211836}, "brand": "Coxlures", "categories": [["Sports & Outdoors", "Other Sports", "Dance"]] }
]

Python 解决方案：

import csv
import json

CSV_COLUMNS = ['asin', 'title', 'categories', 'price',
               'also_bought', 'also_viewed', 'brand']


class DataConverter:

    def __init__(self, filename):
        self.filename = filename
        self.writer = None

    def _get_categories(self, item):
        groups = item.get('categories', [[]])
        formatted_groups = []
        for group in groups:
            formatted_groups.append('(' + ', '.join(group) + ')')
        return ', '.join(formatted_groups)

    def _get_also_bought(self, item):
        related = item.get('related', {})
        also_bought = related.get('also_bought', [])
        return ', '.join(also_bought)

    def _get_also_viewed(self, item):
        related = item.get('related', {})
        also_viewed = related.get('also_viewed', [])
        return ', '.join(also_viewed)

    def _process(self, item):
        """Process one data item.

        Expects a Python representation of a JSON object.
        Returns a list of strings (one row of the CSV file).
        """
        asin = item.get('asin', '')
        title = item.get('title', '')
        categories = self._get_categories(item)
        price = item.get('price', '')
        also_bought = self._get_also_bought(item)
        also_viewed = self._get_also_viewed(item)
        brand = item.get('brand', '')
        return [asin, title, categories, price, also_bought, also_viewed,
                brand]

    def convert(self):
        """Convert the JSON file to a CSV file."""
        # Open the JSON file.
        with open(self.filename + '.json', 'rb') as json_file:
            # Parse the JSON data.
            data = json.load(json_file)
            # Open the CSV file.
            with open(self.filename + '.csv', 'wt', newline='') as csv_file:
                # Create the CSV writer object.
                self.writer = csv.writer(csv_file, delimiter=',')
                # Write the header row.
                self.writer.writerow(CSV_COLUMNS)
                # Write CSV rows, one by one.
                for item in data:
                    row = self._process(item)
                    self.writer.writerow(row)


if __name__ == '__main__':
    DataConverter('metadata').convert()

【讨论】：