多线程python请求[重复]

Question

对于我的学士论文，我需要从大约 40000 个网站中获取一些数据。因此，我正在使用 python 请求，但目前从服务器获得响应真的很慢。

有没有办法加快速度并保持我当前的标题设置？我找到的所有教程都没有标题。

这是我的代码片段：

def parse(url):
    headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) '
                         'Chrome/39.0.2171.95 Safari/537.36'}
    r = requests.get(url, headers=headers)

    for line in r.iter_lines():
        ...

Answer 1

我认为使用mutil-thread 像threading 或multiprocess 是个好主意，或者由于gevent，您可以使用grequests（异步请求）

Answer 2

你可以使用线程，因为这是一个I/O Bound问题。使用内置的threading 库是您的最佳选择。我使用Semaphore 对象来限制可以同时运行的线程数。

import time
import threading

# Number of parallel threads
lock = threading.Semaphore(2)


def parse(url):
   """
   Change to your logic, I just use sleep to mock http request.
   """

    print 'getting info', url
    sleep(2)

    # After we done, subtract 1 from the lock
    lock.release()


def parse_pool():
    # List of all your urls
    list_of_urls = ['website1', 'website2', 'website3', 'website4']

    # List of threads objects I so we can handle them later
    thread_pool = []

    for url in list_of_urls:
        # Create new thread that calls to your function with a url
        thread = threading.Thread(target=parse, args=(url,))
        thread_pool.append(thread)
        thread.start()

        # Add one to our lock, so we will wait if needed.
        lock.acquire()

    for thread in thread_pool:
        thread.join()

    print 'done'

Answer 3

您可以使用 asyncio 并发运行任务。您可以使用 asyncio.wait() 的返回值列出 url 响应（已完成和待处理的响应）并异步调用协程。结果将以意想不到的顺序排列，但这是一种更快的方法。

import asyncio
import functools


async def parse(url):
    print('in parse for url {}'.format(url))

    info = await #write the logic for fetching the info, it waits for the responses from the urls

    print('done with url {}'.format(url))
    return 'parse {} result from {}'.format(info, url)


async def main(sites):
    print('starting main')
    parses = [
        parse(url)
        for url in sites
    ]
    print('waiting for phases to complete')
    completed, pending = await asyncio.wait(parses)

    results = [t.result() for t in completed]
    print('results: {!r}'.format(results))


event_loop = asyncio.get_event_loop()
try:
    websites = ['site1', 'site2', 'site3']
    event_loop.run_until_complete(main(websites))
finally:
    event_loop.close()