如何在请求中使用线程？ [复制]答案

【问题标题】：How can I use threading with requests? [duplicate]如何在请求中使用线程？ [复制]
【发布时间】：2019-12-08 13:57:30
【问题描述】：

您好，我正在使用请求模块，我想提高速度，因为我有很多 url，所以我想我可以使用线程来获得更好的速度。这是我的代码：

import requests

urls = ["http://www.google.com", "http://www.apple.com", "http://www.microsoft.com", "http://www.amazon.com", "http://www.facebook.com"]
for url in urls:
    reponse = requests.get(url)
    value = reponse.json()

但我不知道如何将请求与线程一起使用......

你能帮帮我吗？

谢谢！

【问题讨论】：

标签： python python-3.x multithreading python-2.7 python-multithreading

【解决方案1】：

您可以使用concurrent 模块。

    pool = concurrent.futures.thread.ThreadPoolExecutor(max_workers=DEFAULT_NUMBER_OF_THREADS)
    pool.map(lambda x : requests.get(x), urls)

这允许受控并发。

这是来自threadpool 文档的direct example

import concurrent.futures
import urllib.request

URLS = ['http://www.foxnews.com/',
        'http://www.cnn.com/',
        'http://europe.wsj.com/',
        'http://www.bbc.co.uk/',
        'http://some-made-up-domain.com/']

# Retrieve a single page and report the URL and contents
def load_url(url, timeout):
    with urllib.request.urlopen(url, timeout=timeout) as conn:
        return conn.read()

# We can use a with statement to ensure threads are cleaned up promptly
with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
    # Start the load operations and mark each future with its URL
    future_to_url = {executor.submit(load_url, url, 60): url for url in URLS}
    for future in concurrent.futures.as_completed(future_to_url):
        url = future_to_url[future]
        try:
            data = future.result()
        except Exception as exc:
            print('%r generated an exception: %s' % (url, exc))
        else:
            print('%r page is %d bytes' % (url, len(data)))

【讨论】：

【解决方案2】：

只需从 bashrc 添加，您也可以将它与请求一起使用。你不需要使用 urllib.request 方法。

应该是这样的：

from concurrent import futures

URLS = ['http://www.foxnews.com/',
        'http://www.cnn.com/',
        'http://europe.wsj.com/',
        'http://www.bbc.co.uk/',
        'http://some-made-up-domain.com/']
with futures.ThreadPoolExecutor(max_workers=5) as executor: ## you can increase the amount of workers, it would increase the amount of thread created
    res = executor.map(requests.get,URLS)
responses = list(res) ## the future is returning a generator. You may want to turn it to list.

然而，我喜欢做的是创建一个函数，该函数直接从响应中返回 json（或者如果你想 scrape 则返回文本）。并在线程池中使用该函数

import requests
from concurrent import futures
URLS = ['http://www.foxnews.com/',
        'http://www.cnn.com/',
        'http://europe.wsj.com/',
        'http://www.bbc.co.uk/',
        'http://some-made-up-domain.com/']

def getData(url):
   res = requests.get(url)
   try:
       return res.json()
   except:
       return res.text
with futures.ThreadPoolExecutor(max_workers=5) as executor:
    res = executor.map(getData,URLS)
responses = list(res) ## your list will already be pre-formated

【讨论】：