如何构建循环 API 调用以使 JSON 结果可用？

Question

我有一个 API，我正在调用它限制每个请求 50 个结果以及每分钟 180 个请求。但是，一个对象总共可能有 10 到 50,000 个结果。我需要以可用的格式构建生成的 JSON，或者将其转换为我可以更轻松地访问的东西，比如 pandas，我只是不确定哪个是更好的选择。计划是遍历我需要数据的每个对象 ID。我当前的 API 调用如下所示：

import requests, time
import json, pandas as pd

#object id's to iterate through
# object_id = ['13','16','95','93','77']
object_id = ['13']
#this block finds the count of entries
for obj in object_id:
  data = []
  info_url = 'https://api.ontraport.com/1/objects/getInfo?objectID={}'.format(obj)
  headers = {
    'Api-Appid': 'XXXXXXXXX',
    'Api-Key': 'XXXXXXXXX'
  }
  response = requests.get(info_url, headers=headers).json()
  obj_count = int(response['data']['count'])
#this block ensures all entries are pulled as it iterates in groups of 50
  start = 1
  while start / obj_count <= 1:
    url = 'https://api.ontraport.com/1/objects?objectID={}&start={}'.format(obj, start)
    headers = {
      'Api-Appid': 'XXXXXXXXX',
      'Api-Key': 'XXXXXXXXX'
    }
    response = requests.get(url, headers=headers).json()
    data.append(response)
    start += 50
    time.sleep(0.34)
  with open('C:/Desktop/data{}.txt'.format(obj), 'w') as outfile:
    json.dump(data, outfile)

API 产生以下结果：

{
   "code": 0,
   "data": [{
              "id":"111",
              "date":"1441326063"
           },
           {
              "id":"132",
              "date":"1441526112"
           }],
   "account_id": 0
}

我只关心“数据”标签内的部分。但是，当我将结果附加在一起时，它会成为每个拉取的 JSON 对象的大型 JSON 数组，并且在每个对象中，“数据”是另一个数组。是否可以将其结构化为仅附加数据结果，然后格式化为 pandas 数据框？

Answer 1

我希望这项工作，有关更多信息，请查看List comprehension

import pandas as pd
import json

json_str = """[
{
   "code": 0,
   "data": [{
              "id":"111",
              "date":"1441326063"
           },
           {
              "id":"132",
              "date":"1441526112"
           }],
   "account_id": 0
},
{
   "code": 0,
   "data": [{
              "id":"111",
              "date":"1441326063"
           },
           {
              "id":"132",
              "date":"1441526112"
           }],
   "account_id": 0
}
]"""

json_dict = json.loads(json_str)

df = pd.DataFrame([d for y in json_dict for d in y['data']])

Answer 2

如果只有 1 个数据块，您可以像这样简单地保存它，使用 json 字典格式。

data.extend(response['Data'])

例如：

import pandas as pd

Data = {
"code": 0,
"data": [{
            "id":"111",
            "date":"1441326063"
        },
        {
            "id":"132",
            "date":"1441526112"
        }],
"account_id": 0
}

Data2 = {
"code": 0,
"data": [{
            "id":"134311",
            "date":"1441326063"
        },
        {
            "id":"133432",
            "date":"1441526112"
        }],
"account_id": 0
}

data = []

data.extend(Data['data'])
data.extend(Data2['data'])

print(data)
>>> [{'id': '111', 'date': '1441326063'}, {'id': '132', 'date': '1441526112'}, {'id': '134311', 'date': '1441326063'}, {'id': '133432', 'date': '1441526112'}]

df = pd.DataFrame(data)
df.to_csv('example.csv', index=False)
print(df)

>>>          date      id
     0  1441326063     111
     1  1441526112     132
     2  1441326063  134311
     3  1441526112  133432