【问题标题】:JSON nested list to Pandas dataframeJSON 嵌套列表到 Pandas 数据框
【发布时间】:2021-09-30 14:35:16
【问题描述】:

我有一个如下所示的 json 文件:

    "Aveiro": {
        "Albergaria-a-Velha": {
            "candidates": [
                {
                    "effectiveCandidates": [
                        "JOSÉ OLIVEIRA SANTOS"
                    ],
                    "party": "B.E.",
                    "votes": {
                        "absoluteMajority": 0,
                        "acronym": "B.E.",
                        "constituenctyCounter": 1,
                        "mandates": 0,
                        "percentage": 1.34,
                        "presidents": 0,
                        "validVotesPercentage": 1.4,
                        "votes": 179
                    }
                },
                {
                    "effectiveCandidates": [
                        "ANTÓNIO AUGUSTO AMARAL LOUREIRO E SANTOS"
                    ],
                    "party": "CDS-PP",
                    "votes": {
                        "absoluteMajority": 1,
                        "acronym": "CDS-PP",
                        "constituenctyCounter": 1,
                        "mandates": 5,
                        "percentage": 59.7,
                        "presidents": 1,
                        "validVotesPercentage": 62.5,
                        "votes": 7970
                    }
                },
                {
                    "effectiveCandidates": [
                        "CARLOS MANUEL DA COSTA SERVEIRA VASQUES"
                    ],
                    "party": "CH",
                    "votes": {
                        "absoluteMajority": 0,
                        "acronym": "CH",
                        "constituenctyCounter": 1,
                        "mandates": 0,
                        "percentage": 1.87,
                        "presidents": 0,
                        "validVotesPercentage": 1.95,
                        "votes": 249
                    }
                },
                {
                    "effectiveCandidates": [
                        "RODRIGO MANUEL PEREIRA MARQUES LOURENÇO"
                    ],
                    "party": "PCP-PEV",
                    "votes": {
                        "absoluteMajority": 0,
                        "acronym": "PCP-PEV",
                        "constituenctyCounter": 1,
                        "mandates": 0,
                        "percentage": 1.57,
                        "presidents": 0,
                        "validVotesPercentage": 1.65,
                        "votes": 210
                    }
                },
                {
                    "effectiveCandidates": [
                        "DELFINA LISBOA MARTINS DA CUNHA"
                    ],
                    "party": "PPD/PSD",
                    "votes": {
                        "absoluteMajority": 0,
                        "acronym": "PPD/PSD",
                        "constituenctyCounter": 1,
                        "mandates": 2,
                        "percentage": 24.23,
                        "presidents": 0,
                        "validVotesPercentage": 25.37,
                        "votes": 3235
                    }
                },
                {
                    "effectiveCandidates": [
                        "JESUS MANUEL VIDINHA TOMÁS"
                    ],
                    "party": "PS",
                    "votes": {
                        "absoluteMajority": 0,
                        "acronym": "PS",
                        "constituenctyCounter": 1,
                        "mandates": 0,
                        "percentage": 6.82,
                        "presidents": 0,
                        "validVotesPercentage": 7.14,
                        "votes": 910
                    }
                }
            ],
            "parentTerritoryName": "Aveiro",
            "territoryKey": "LOCAL-010200",
            "territoryName": "Albergaria-a-Velha",
            "total_votes": {
                "availableMandates": 0,
                "blankVotes": 377,
                "blankVotesPercentage": 2.82,
                "displayMessage": null,
                "hasNoVoting": false,
                "nullVotes": 221,
                "nullVotesPercentage": 1.66,
                "numberParishes": 6,
                "numberVoters": 13351,
                "percentageVoters": 59.48
            }
        },

完整文件为here供参考

我认为这段代码可以工作

import pandas as pd 
from pandas import json_normalize
import json


with open('autarquicas_2021.json') as f:
    data = json.load(f)

df = pd.json_normalize(data)

但是这会返回以下内容:

df.head()
Aveiro.Albergaria-a-Velha.candidates  ... Évora.Évora.total_votes.percentageVoters
0  [{'effectiveCandidates': ['JOSÉ OLIVEIRA SANTO...  ...                                    49.84

[1 rows x 4312 columns]

df.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 1 entries, 0 to 0
Columns: 4312 entries, Aveiro.Albergaria-a-Velha.candidates to Évora.Évora.total_votes.percentageVoters
dtypes: bool(308), float64(924), int64(1540), object(1540)
memory usage: 31.7+ KB
None

由于某种原因,代码无法正常工作,我的研究导致我没有找到解决方案,因为似乎每个 json 文件都有自己的想法。

任何帮助将不胜感激。提前感谢您!

免责声明:这是一个开源项目,旨在提高葡萄牙地方选举的透明度。它不会用于商业或盈利项目。

【问题讨论】:

  • 您希望数据看起来如何?
  • pd.read_json("your_json_file.json") 工作吗?
  • @HaleemurAli 我也应该发布它。在这里你可以看到一个例子,目标是docs.google.com/spreadsheets/d/e/…
  • @zabop pd.read_json 返回 ``` Aveiro Açores Beja Braga Bragança ... Setúbal Viana do Castelo Vila Real Viseu Évora Albergaria-a-Velha {'candidates': [{'effectiveCandidates': [ 'JOSÉ... NaN NaN NaN NaN ... NaN NaN NaN NaN NaN``` 带有 df.info() 返回 [2 rows x 20 columns] &lt;class 'pandas.core.frame.DataFrame'&gt; Index: 306 entries, Albergaria-a-Velha to Évora Data columns (total 20 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 Aveiro 19 non-null object
  • 所以,我使用了一个在线工具,这是生成的 CSV 文件:docs.google.com/spreadsheets/d/e/…

标签: python json pandas dataframe


【解决方案1】:

您可以使用json_normalize,对原始 JSON 格式稍作转换。

  1. 将 JSON 转换为列表格式。 我假设“Aveiro”为城市,“Albergaria-a-Velha”为地区。抱歉我不熟悉该地区,所以如果有错误,请重命名密钥。
res = [{**z, **{'city': x, 'district': y}} for x, y in data.items() for y, z in y.items()]

这会将键值样式的原始 JSON 转换为对象列表。

[{
    "city": "Aveiro",
    "district": "Albergaria-a-Velha",
    "candidates": [{
        ...
}]
  1. 然后使用json_normalize
df = pd.json_normalize(res, record_path=['candidates'], meta=['total_votes', 'city', 'district'])
  1. 进一步扩展嵌套对象total_votes
df = pd.concat([df, pd.json_normalize(df['total_votes'])], axis=1)
>>> df.iloc[0]
effectiveCandidates                                      [JOSÉ OLIVEIRA SANTOS]
party                                                                      B.E.
votes.absoluteMajority                                                        0
votes.acronym                                                              B.E.
votes.constituenctyCounter                                                    1
votes.mandates                                                                0
votes.percentage                                                           1.34
votes.presidents                                                              0
votes.validVotesPercentage                                                  1.4
votes.votes                                                                 179
total_votes                   {'availableMandates': 0, 'blankVotes': 377, 'b...
city                                                                     Aveiro
district                                                     Albergaria-a-Velha
availableMandates                                                             0
blankVotes                                                                  377
blankVotesPercentage                                                       2.82
displayMessage                                                             None
hasNoVoting                                                               False
nullVotes                                                                   221
nullVotesPercentage                                                        1.66
numberParishes                                                                6
numberVoters                                                              13351
percentageVoters                                                          59.48
Name: 0, dtype: object

【讨论】:

    【解决方案2】:

    递归方法:

    我通常使用(a recursive approach)这个函数来做那种事情:

    # Function for flattening 
    # json
    def flatten_json(y):
        out = {}
      
        def flatten(x, name =''):
              
            # If the Nested key-value 
            # pair is of dict type
            if type(x) is dict:
                  
                for a in x:
                    flatten(x[a], name + a + '_')
                      
            # If the Nested key-value
            # pair is of list type
            elif type(x) is list:
                  
                i = 0
                  
                for a in x:                
                    flatten(a, name + str(i) + '_')
                    i += 1
            else:
                out[name[:-1]] = x
      
        flatten(y)
        return out
    

    您可以调用 flatten_json 来扁平化您的嵌套 json。

    # Driver code
    print(flatten_json(data))
    

    基于库的方法:

    from flatten_json import flatten
    
    unflat_json = {'user' :
                   {'foo':
                    {'UserID':0123456,
                    'Email': 'foo@mail.com', 
                    'friends': ['Johnny', 'Mark', 'Tom']
                    }
                   }
                  }
      
    flat_json = flatten(unflat_json)
      
    print(flat_json)
    

    【讨论】:

      猜你喜欢
      • 1970-01-01
      • 1970-01-01
      • 1970-01-01
      • 1970-01-01
      • 1970-01-01
      • 1970-01-01
      • 2020-09-27
      • 2021-04-20
      • 2018-03-06
      相关资源
      最近更新 更多