Instagram API 在一些调用后开始返回加载页面

Question

我正在使用下面的代码使用 asycnio 获取一千个 instagram 帐户的帐户信息。在初始请求中，输出是正确的，但在 10-20 次调用后，instagram 开始返回加载页面的 HTML 代码。我在这里做错了什么？下面是python代码。

import random
import asyncio
from aiohttp import ClientSession
import urllib.request
import  aiohttp
async def fetch(url, session,sem):
    print("------")
    print(url)
    async with session.get(url = url) as response:
        print(await response.text())
        await  response.text()
        # exit()
        if response.status == 200:
            await sem.acquire()
            fname = url[22:]
            fname = fname.split('/')
            fname = fname[0] + '.txt'
            f = open(fname , 'w')
            f.write(str(await response.text()))
            sem.release()

        # return (await response.text())


async def run(url_list):
    tasks = []

    # create instance of Semaphore
    sem = asyncio.Semaphore(2)
    # Create client session that will ensure we dont open new connection
    # per each request.
    async with ClientSession() as session:
        for url in url_list:
            task = asyncio.ensure_future(fetch(url, session,sem))
            tasks.append(task)
        responses = asyncio.gather(*tasks)
        await responses


# making the url list here
url_list = []
file = open('url.txt', 'r')
for url in file:
    url_list.append(url)

print(url_list)
import time
old = time.time()
loop = asyncio.get_event_loop()
future = asyncio.ensure_future(run(url_list))
loop.run_until_complete(future)

print(time.time() - old)

这里是一些来自 url.txt 文件的 URL

https://instagram.com/johanna_kre/?__a=1
https://instagram.com/channie_f/?__a=1
https://instagram.com/lilakuh68/?__a=1
https://instagram.com/nataliacallisto/?__a=1
https://instagram.com/edbastian/?__a=1
https://instagram.com/sylvana.h/?__a=1
https://instagram.com/munich_bombon/?__a=1
https://instagram.com/younotus/?__a=1
https://instagram.com/meet.herbert/?__a=1
https://instagram.com/inaaogo/?__a=1
https://instagram.com/dennisaogo/?__a=1
https://instagram.com/mrslight__/?__a=1
https://instagram.com/reneturrek/?__a=1
https://instagram.com/_eeasyyy/?__a=1
https://instagram.com/sentinobln/?__a=1
https://instagram.com/eri.ka_g/?__a=1

Answer 1

您的信号量并未按照您的意愿限制请求；您应该在发出请求之前获取它，而不是在处理内容之前。

使用您当前的实现，您正在发出 100 个并发请求（aiohttp 的客户端默认限制），但一次只处理两个响应（但是从服务器的角度来看，此时请求已经被处理）。

用途：

async def fetch(url, session,sem):
    print("------")
    print(url)
    await sem.acquire()
    async with session.get(url = url) as response:
        print(await response.text())
        await  response.text()
        ...
    sem.release()
    ...