带有请求的 Python3 Concurrent.Futures

【问题标题】：Python3 Concurrent.Futures with Requests带有请求的 Python3 Concurrent.Futures
【发布时间】：2021-06-27 11:44:40
【问题描述】：

我正在尝试实现并发请求以加快对 URL 列表的检查，但它似乎不适用于我的代码，因为它仍在逐一检查它们。

for domain in list:
    try:
        follow_url = requests.head(f'http://{domain}', allow_redirects=True, timeout=60)

        with concurrent.futures.ThreadPoolExecutor(max_workers=20) as executor:
            executor.submit(follow_url)

        with open("alive.txt", "a") as file:
            file.write(f'{domain}\n')

    except Exception as e:
        print(e)

【问题讨论】：

标签： python-3.x concurrent.futures

【解决方案1】：

您没有正确应用它。您正在迭代中创建并行流程。正确的方式可能是这样的：

def parallel_req(domain):
    try:
        follow_url = requests.head(f'http://{domain}', allow_redirects=True, timeout=60)
        with open("alive.txt", "a") as file:
            file.write(f'{domain}\n')
    except requests.exceptions.RequestException as e:
        print(e)

with ThreadPoolExecutor() as e:
    e.map(parallel_req, domains)

【讨论】：