从 INSEE API 解析多层 JSON

Question

我正在向 API 发送请求。这个API是INSEE，是注册公司的法国官方仓库。

这是我的要求：

headers = {
    'Accept': 'application/json',
    'Authorization': 'xxx'
}

params = (
    ('q', 'siren:530085802'),
    ('date', '2021-10-01'),
    ('champs', 'siret, denominationUniteLegale, codePostalEtablissement, libelleCommuneEtablissement, denominationUsuelleEtablissement'),
    ('debut', 1)
    # ('nombre', 1)
)

response = requests.get('https://api.insee.fr/entreprises/sirene/V3/siret', headers=headers, params=params)

对于这个例子，我请求 Facebook 的法人实体编号：siren:530085802（这是开放数据，这里没有任何机密信息）。

现在我得到了回应：

reponse.text

这个：

'{"header":{"statut":200,"message":"OK","total":3,"debut":1,"nombre":2},"etablissements":[{"siret":"53008580200011","uniteLegale":{"denominationUniteLegale":"FACEBOOK FRANCE"},"adresseEtablissement":{"codePostalEtablissement":"75116","libelleCommuneEtablissement":"PARIS 16"},"periodesEtablissement":[{"dateFin":null,"dateDebut":"2012-06-14","denominationUsuelleEtablissement":null},{"dateFin":"2012-06-13","dateDebut":"2011-02-17","denominationUsuelleEtablissement":null},{"dateFin":"2011-02-16","dateDebut":"2011-01-01","denominationUsuelleEtablissement":null}]},{"siret":"53008580200037","uniteLegale":{"denominationUniteLegale":"FACEBOOK FRANCE"},"adresseEtablissement":{"codePostalEtablissement":"75002","libelleCommuneEtablissement":"PARIS 2"},"periodesEtablissement":[{"dateFin":null,"dateDebut":"2018-06-26","denominationUsuelleEtablissement":null},{"dateFin":"2018-06-25","dateDebut":"2016-04-18","denominationUsuelleEtablissement":null}]}]}'

根据官方文档，有多个层：

• 法人实体：由siren 标识
• 建立：由siret 标识——即SIREN + 5 个字符。基本上在这里我得到了一个机构列表，因为我只要求一个警报器
• Periods ：这是几个属性的存档/更改跟踪。带有dateDebut（开始日期）和dateFin（结束日期）和属性，其中denominationUsuelleEtablissement 我想要当前值。

尽管在我的请求中添加了日期，但 API 响应了多个句点，而不仅仅是包含我提供的日期的句点。因此添加了一个图层。

我正在尝试做的是展平响应并将其转换为数据框以便于使用。

我所做的如下。使用json.normalize：

import json
import pandas as pd

contenuReponse = json.loads(response.text)
etablissements = contenuReponse['etablissements']
 
df = pd.json_normalize(etablissements)
df

我明白了：

	siret	periodesEtablissements	uniteLegale.denominationUniteLegale
0	53008580200011	[{'dateFin': None, 'dateDebut': '2012-06-14', ...	FACEBOOK FRANCE
1	53008580200037	[{'dateFin': None, 'dateDebut': '2018-06-26', ...	FACEBOOK FRANCE

json_normalize 忽略periodesEtablissement下面的层。

有没有办法成功地展平整个响应。或者，仅使用与我的请求相关的 periodesEtablissement。意思是没有dateFin的那个？

Answer 1

你可以试试这个功能：

def flatten_nested_json_df(df):
    df = df.reset_index()
    s = (df.applymap(type) == list).all()
    list_columns = s[s].index.tolist()
    
    s = (df.applymap(type) == dict).all()
    dict_columns = s[s].index.tolist()

    
    while len(list_columns) > 0 or len(dict_columns) > 0:
        new_columns = []

        for col in dict_columns:
            horiz_exploded = pd.json_normalize(df[col]).add_prefix(f'{col}.')
            horiz_exploded.index = df.index
            df = pd.concat([df, horiz_exploded], axis=1).drop(columns=[col])
            new_columns.extend(horiz_exploded.columns) # inplace

        for col in list_columns:
            print(f"exploding: {col}")
            df = df.drop(columns=[col]).join(df[col].explode().to_frame())
            new_columns.append(col)

        s = (df[new_columns].applymap(type) == list).all()
        list_columns = s[s].index.tolist()

        s = (df[new_columns].applymap(type) == dict).all()
        dict_columns = s[s].index.tolist()
    return df

然后这样做：

flatten_nested_json_df(df)

给出：

  index           siret uniteLegale.denominationUniteLegale  \
0      0  53008580200011                     FACEBOOK FRANCE   
0      0  53008580200011                     FACEBOOK FRANCE   
0      0  53008580200011                     FACEBOOK FRANCE   
1      1  53008580200037                     FACEBOOK FRANCE   
1      1  53008580200037                     FACEBOOK FRANCE   

  adresseEtablissement.codePostalEtablissement  \
0                                        75116   
0                                        75116   
0                                        75116   
1                                        75002   
1                                        75002   

  adresseEtablissement.libelleCommuneEtablissement  \
0                                         PARIS 16   
0                                         PARIS 16   
0                                         PARIS 16   
1                                          PARIS 2   
1                                          PARIS 2   

  periodesEtablissement.dateFin periodesEtablissement.dateDebut  \
0                          None                      2012-06-14   
0                    2012-06-13                      2011-02-17   
0                    2011-02-16                      2011-01-01   
1                          None                      2018-06-26   
1                    2018-06-25                      2016-04-18   

  periodesEtablissement.denominationUsuelleEtablissement  
0                                               None      
0                                               None      
0                                               None      
1                                               None      
1                                               None