批处理 API 请求答案

【问题标题】：Batching API Requests批处理 API 请求
【发布时间】：2023-02-09 03:38:04
【问题描述】：

我有一个包含 1,000 个机场的列表，我正在发送到 API 以获取每个机场的航班数据。即使我延迟调用，API 也无法立即处理整个列表。我需要将机场列表分成 100 个批次，以便 API 调用正常工作。我下面的代码遍历机场列表并将它们一一发送到 API。我想分解 API 调用（机场列表）并以 100 个为一组调用它们，因为当我使用整个 1,000 个时它会导致数据格式错误。当我只用 100 个机场测试 API 时，所有数据都正确返回。我不确定将批处理代码放在我的 API 调用循环中的什么位置。

# Sample dataset for this post
airport = [['HLZN'], ['HLLQ'],['HLLB'],['HLGT'],['HLMS'],['HLLS'],['HLTQ'],['HLLT'],['HLLM']] 

payload = {'max_pages': 500, 'type':'Airline'}
seconds = 1
count = 1

#Create an empty list to hold responses
json_responses = []

#Iterate through list
for airports in airport:
    response = requests.get(apiUrl + f"airports/{airports[0]}/flights",params=payload,
               headers=auth_header)
    if response.status_code == 200:
        print(count, airports)
        count +=1
        for i in trange(100):
            time.sleep(0.01)
    else:
        pass
    results = response.json()
    json_responses.append(response.json())
    sleep(seconds)

我不确定将批处理代码放在 API 调用循环中的什么位置。我一般不熟悉批处理 API 调用和循环，因此我们将不胜感激。

total_count = len(airport)

#Iterate through list
for airports in airport:
    response = requests.get(apiUrl + f"airports/{airports[0]}/flights",params=payload,
               headers=auth_header)
    chunks = (total_count - 1) // 100 + 1
    for i in range(chunks):
        batch = airport[i*100:(i+1)*100] #Tried batch code here
        if response.status_code == 200:
            print(count, airports)
            count +=1
            for i in trange(100):
                time.sleep(0.01)
        else:
            pass
        results = response.json()
        json_responses.append(response.json())
        sleep(seconds)

【问题讨论】：

标签： python api loops rest batching

【解决方案1】：

我相信这就是你想要做的：

# Sample dataset for this post
airports = [['HLZN'], ['HLLQ'],['HLLB'],['HLGT'],['HLMS'],['HLLS'],['HLTQ'],['HLLT'],['HLLM']] 

payload = {'max_pages': 500, 'type':'Airline'}
seconds = 1

#Create an empty list to hold responses
json_responses = []

# Counter variable
counter = 0

# Chunk size
chunk_size = 100

#Iterate through list
for airport in airports:
    response = requests.get(apiUrl + f"airports/{airports[0]}/flights",params=payload,
               headers=auth_header)
    results = response.json()
    json_responses.append(response.json())

    # Increment counter and check if it is a multiple of the chunk size, if yes, sleep for a defined number of seconds
    counter += 1
    if counter % chunk_size == 0:
        sleep(seconds)

【讨论】：