如何在 Python 中同时或并行执行此操作（检查请求状态代码）？

Question

这就是我正在做的事情：

• 从文本文件中获取单词 - 每个单词都在单独的行中。
• 将http://www. 和.com 添加到文字以创建网址。
• 获取带有请求的 URL。
• 确定它是否是免费域（基于状态代码和 连接错误/其他错误）。
• 将免费域添加到文本文件中。
• 计时。

到目前为止，我已经让它工作了，但是速度很慢。文本文件有 350 000 字。我将如何同时或并行执行此操作？还有哪个更适合这项任务？

这是我的代码：

import requests, time

start = time.time()

with open('words1.txt','r') as f:
    words = []
    for item in f:
        words.append(item.strip())

for w in words:
    url = 'http://www.'+w+'.com'
    try:
        header = {'User-Agent': 'Mozilla/5.0'}
        r = requests.get(url, headers=header)
        codes = [200,201,202,203,204,205,206,300,301,302,303,307,308,400,401,402,403,404,405,406,500,501,502,503]
        if r.status_code in codes:
            print(url,': Known Status Code > Unavailable')
        else:
            print(url,': Unknown Status Code > Probably Free')
            with open('available.txt','a') as myfile:
                myfile.write(url+'\n')
    except requests.exceptions.ConnectionError:
        print(url,' : Connection Error > Probably Free')
        with open('available.txt','a') as myfile:
            myfile.write(url+'\n')
    except requests.exceptions.HTTPError:
        print('http error')
    except requests.exceptions.Timeout:
        print('timeout error')
    except requests.exceptions.TooManyRedirects:
        print('too many redirects')

end = time.time()
print('\n')
print(end-start, 'seconds')
print((end-start)/60,'minutes')
print(((end-start)/60)/60,'hours')

谢谢！

编辑：我让它工作了。感谢 Kendas 和 DeepSpace 的帮助！ 这是一个快速测试：

100 字 - 22 秒

1000 字 - 285 秒

不是太快，但比我第一次尝试要快。

似乎 gevent + socket 是要走的路。

如果您对改进/更快有任何建议，请告诉我。

代码如下：

import gevent,time
from gevent import socket

start = time.time()

words = []
with open('words1000.txt','r') as f:
    for item in f:
        words.append(item.strip())

urls = ['www.{}.com'.format(w) for w in words]

jobs = [gevent.spawn(socket.gethostbyname, url) for url in urls]

gevent.joinall(jobs)

values = {url:job.value for (url,job) in zip(urls,jobs)}
freeDomains = []
for (v,job,url) in zip(values,jobs,urls):
    if job.value == None:
        freeDomains.append(url)
        with open('availableds.txt','a') as myFile:
            myFile.write(url+'\n')

print(freeDomains)
end = time.time()
print(end-start,'seconds')
print((end-start)/60,'minutes')
print((end-start)/3600,'hours')

Answer 1

grequests（请求的并发版本）使这很容易。 使用.format 而不是每次迭代都重新定义header 也会有所帮助

import grequests

def exception_handler(request, exception):
    print(exception)

with open('words1.txt','r') as f:
    words = []
    for item in f:
        words.append(item.strip())
urls = ['http://www.{}.com'.format(w) for w in words]

header = {'User-Agent': 'Mozilla/5.0'}

requests = [grequests.get(url) for url in urls]

responses = grequests.map(requests, exception_handler=exception_handler)

for resp in responses:
    if resp:
         print(resp.status_code)