访问 JSON 文件中的嵌套数据以构建多个数据框答案

【问题标题】：Accessing nested data in JSON file to build multiple dataframes访问 JSON 文件中的嵌套数据以构建多个数据框
【发布时间】：2020-05-06 19:18:00
【问题描述】：

我目前正在访问一个 JSON 文件以收集有关公司、分支机构和员工的信息。在这个 JSON 中有多个公司，每个公司都有自己的分支机构，每个分支机构都有自己的工人。下面是我正在处理的 JSON 结构的示例。

{
  "company": [
    {
      "companyName": "MyCompany",
      "branches": [
        {
          "branchName": "My First Branch",
          "workers": [
            {
              "workerName": "John",
              "wage": "10"
            },
            {
              "workerName": "Lucas",
              "wage": "20"
            }
          ]
        },
        {
          "branchName": "My Second Branch",
          "workers": [
            {
              "workerName": "Mary",
              "wage": "30"
            },
            {
              "workerName": "Jack",
              "wage": "40"
            }
          ]
        }
      ]
    },
    {
      "companyName": "YourCompany",
      "branches": [
        {
          "branchName": "Your First Branch",
          "workers": [
            {
              "workerName": "George",
              "wage": "15"
            },
            {
              "workerName": "Harry",
              "wage": "25"
            }
          ]
        },
        {
          "branchName": "Your Second Branch",
          "workers": [
            {
              "workerName": "Wayne",
              "wage": "35"
            },
            {
              "workerName": "Rose",
              "wage": "45"
            }
          ]
        }
      ]
    }
  ]
}

我的目标是在一个数据框中为每个公司、分支机构和员工收集有关所有公司、分支机构和员工的信息。为此，我目前正在使用循环，如下所示。

Companies = my_json['Companies'] #accessing list of companies
Branches = None
Workers = None
for i in range(len(Companies)):
  company_branches = Companies[i]['Branches'] #accessing branches for that company
  if(Branches is None):
    Branches = pd.DataFrame(company_branches)
  else:
    Branches = pd.concat([Branches,pd.DataFrame(company_branches)])
  for j in range(len(company_branches)):
    branch_workers = company_branches[j]['Workers'] #accessing workers for that branch
    if(Workers is None):
      Workers = pd.DataFrame(branch_workers)
    else:
      Workers = pd.concat([Workers,pd.DataFrame(branch_workers)])
Companies = pd.DataFrame(Companies)

这解决了我的问题，创建了三个所需的数据框（为公司和分支机构添加了一些额外的列，我仍然会删除），但我目前遇到了性能问题。我一直试图在没有循环的情况下解决这个问题，但我无法正确创建数据帧。如果我尝试

Companies = pd.DataFrame(my_json['companies'])

它会正确创建数据框，但如果我尝试

Branches = pd.DataFrame(Companies.branches.values)

它没有正确创建。它不会引发错误，但它基本上是从 Companies 数据框中复制列，为每个公司的分支机构创建一个带有 JSON 代码的列。在这种情况下，我想要的是与分支具有的属性数量一样多的列。

关于如何有效解决此问题的任何提示？

【问题讨论】：

标签： python json pandas dataframe

【解决方案1】：

您可能应该将所有数据保存在一个数据框中（整洁的数据）。如果需要，您可以获取公司或分支机构的数据框，例如pd.DataFrame(df.company.unique(), columns='company name').

以下方法使用嵌套列表推导来展平您的数据。它还将wage 记录从字符串转换为浮点数。

df = pd.DataFrame(
    [(company.get('companyName'), branch.get('branchName'), 
      worker.get('workerName'), float(worker.get('wage', 0))) 
     for company in my_json['company']
     for branch in company['branches'] 
     for worker in branch.get('workers')
], columns=['company', 'branch', 'worker', 'wage'])

>>> df
       company              branch  worker  wage
0    MyCompany     My First Branch    John  10.0
1    MyCompany     My First Branch   Lucas  20.0
2    MyCompany    My Second Branch    Mary  30.0
3    MyCompany    My Second Branch    Jack  40.0
4  YourCompany   Your First Branch  George  15.0
5  YourCompany   Your First Branch   Harry  25.0
6  YourCompany  Your Second Branch   Wayne  35.0
7  YourCompany  Your Second Branch    Rose  45.0

【讨论】：

【解决方案2】：

这是我对问题的通用解决方案。鉴于我将您提供的整个json 定义为data，那么：

to_df = {'companyName':[],'branchName':[],'workerName':[],'wage':[]}
for i in range(len(data['company'])):
    for j in range(len(data['company'][i]['branches'])):
        for k in range(len(data['company'][i]['branches'][j]['workers'])):
            to_df['companyName'].append(data['company'][i]['companyName'])
            to_df['branchName'].append(data['company'][i]['branches'][j]['branchName'])
            to_df['workerName'].append(data['company'][i]['branches'][j]['workers'][k]['workerName'])
            to_df['wage'].append(data['company'][i]['branches'][j]['workers'][k]['wage'])
df = pd.DataFrame(to_df)
print(df)

输出：

   companyName          branchName workerName wage
0    MyCompany     My First Branch       John   10
1    MyCompany     My First Branch      Lucas   20
2    MyCompany    My Second Branch       Mary   30
3    MyCompany    My Second Branch       Jack   40
4  YourCompany   Your First Branch     George   15
5  YourCompany   Your First Branch      Harry   25
6  YourCompany  Your Second Branch      Wayne   35
7  YourCompany  Your Second Branch       Rose   45

您可以查看此答案以获取有关嵌套 json 的更多信息：How to extract nested JSON data?

【讨论】：