gevent和线程的效率比较

Question

最近，我正在做一个 gevent 演示，并尝试比较 gevent 和线程之间的效率。一般来说，gevent 代码应该比线程代码更高效。但是当我使用 time 命令分析程序时，我得到了不寻常的结果（我的命令是time python FILENAME.py 50 1000，最后两个参数表示池号或线程号，所以我更改了下表中的两个数字）。结果表明线程比gevent代码效率更高，所以我想知道为什么会发生这种情况以及我的程序有什么问题？谢谢。

gevent VS 线程

我的代码如下（主要思想是使用线程或gevent发送多个HTTP请求）：

******这是线程版本代码******

# _*_ coding: utf-8 _*_
import sys
reload(sys)
sys.setdefaultencoding("utf8")
import requests
import threading
import time
import urllib2

finished = 0


def GetUrl(pagenum):
    url = 'http://opendata.baidu.com/zhaopin/s?p=mini&wd=%B0%D9%B6%C8&pn=' + \
        str(pagenum*20) + '&rn=20'
    return url


def setUrlSet():
    for i in xrange(requestnum):
        urlnum = i % 38
        urlset.append(GetUrl(urlnum))


def GetResponse(pagenum):
    try:
        r = requests.get(urlset[pagenum])
    except Exception, e:
        print e
    pass


def DigJobByPagenum(pagenum, requestnum):
    init_num = pagenum
    print '%d begin' % init_num
    while pagenum < requestnum:
        GetResponse(pagenum)
        pagenum += threadnum
    print '%d over' % init_num


def NormalThread(threadnum):
    startime = time.time()
    print "%s is running..." % threading.current_thread().name
    threads = []
    global finished, requestnum
    for i in xrange(threadnum):
        thread = threading.Thread(target=DigJobByPagenum, args=(i, requestnum))
        threads.append(thread)
    for t in threads:
        t.daemon = True
        t.start()
    for t in threads:
        t.join()
        finished += 1
    endtime = time.time()
    print "%s is stop.The total time is %0.2f" % \
        (threading.current_thread().name, (endtime - startime))


def GetAvageTime(array):
    alltime = 0.0
    for i in array:
        alltime += i
    avageTime = alltime/len(array)
    return avageTime

if __name__ == '__main__':
    threadnum = int(sys.argv[1])
    requestnum = int(sys.argv[2])
    print 'threadnum : %s,requestnum %s ' % (threadnum, requestnum)
    originStartTime = time.time()
    urlset = []
    setUrlSet()
    NormalThread(threadnum)

******这是gevent的版本代码******

# _*_ coding: utf-8 _*_
import sys
reload(sys)
sys.setdefaultencoding("utf8")
from gevent import monkey
monkey.patch_all()
import gevent
from gevent import pool
import requests
import time

finished = 0


def GetUrl(pagenum):
    url = 'http://opendata.baidu.com/zhaopin/s?p=mini&wd=%B0%D9%B6%C8&pn=' + \
        str(pagenum*20) + '&rn=20'
    return url


def setUrlSet():
    for i in xrange(requestnum):
        urlnum = i % 38
        urlset.append(GetUrl(urlnum))


def GetResponse(url):
    startime = time.time()
    r = requests.get(url)
    print url
    endtime = time.time()
    spendtime = endtime - startime
    NormalSpendTime.append(spendtime)
    global finished
    finished += 1
    print finished


def GetAvageTime(array):
    alltime = 0.0
    for i in array:
        alltime += i
    avageTime = alltime/len(array)
    return avageTime


def RunAsyncJob():
    jobpool = pool.Pool(concurrent)
    for url in urlset:
        jobpool.spawn(GetResponse, url)
    jobpool.join()
    endtime = time.time()
    allSpendTime = endtime - originStartime
    print 'Total spend time is %0.3f, total request num is %s within %s \
            seconds' % (allSpendTime, finished, timeoutNum)
    print 'Each request time is %0.3f' % (GetAvageTime(NormalSpendTime))


if __name__ == '__main__':
    concurrent = int(sys.argv[1])
    requestnum = int(sys.argv[2])
    timeoutNum = 100
    NormalSpendTime = []
    urlset = []
    urlActionList = []
    setUrlSet()
    originStartime = time.time()
    RunAsyncJob()

Answer 1

试试

gevent.monkey.patch_all(httplib=True)

似乎默认情况下 gevent 不会修补 httplib（看看 http://www.gevent.org/gevent.monkey.html : httplib=False）所以你实际上是在做阻塞请求，你失去了异步框架的所有优点。虽然我不确定 requests 是否使用 httplib。

如果这不起作用，那么看看这个库：

https://github.com/kennethreitz/grequests

Answer 2

回复：httplib=False

您已经在使用requests 库进行网络调用。它有gevent 风格，称为grequests：

https://github.com/kennethreitz/grequests

总体而言，如果您的池非常小，我不会立即看到选择一种线程样式而不是另一种线程样式的理由。当然，真正的线程相对较重（从 8MB 堆栈开始），但您必须将其与工作的大小成比例。

我的看法，两者都尝试（完成），确认你都做对了（要做），让数字说话。