urllib2.urlopen 读取的最佳块大小是多少？

Question

我正在使用这段代码下载 mp3 播客。

req = urllib2.urlopen(item)
CHUNK = 16 * 1024
with open(local_file, 'wb') as fp:
    while True:
        chunk = req.read(CHUNK)
        if not chunk: break
        fp.write(chunk)

哪个效果很好 - 但我想知道最佳下载性能的最佳块大小是多少？

如果有什么不同，我使用的是 6mbit adsl 连接。

Answer 1

一个好的缓冲区大小将与您的操作系统内核用于套接字缓冲区的大小相同。这样，您就不会执行过多的读取操作。

在 GNU/Linux 上，可以在 /proc/sys/net/core/rmem_default 文件中看到套接字缓冲区的大小（大小以字节为单位）。 您可以增加套接字的缓冲区大小，使用setsockopt 设置SO_RCVBUF 参数。但是，此大小受您的系统 (/proc/sys/net/core/rmem_max) 限制，您需要管理员权限 (CAP_NET_ADMIN) 才能超出该限制。

此时，您可能会做一些特定于平台的事情以获取少量收益。

但是，最好查看套接字的选项（请参阅man 7 socket、online version）以执行微优化和学习内容。 :)

由于没有真正的最佳位置总是最有效的，因此您应该始终对任何调整进行基准测试，以检查您的更改是否真的有益。玩得开心！

Answer 2

进一步扩展我对@giant_teapot 的评论

我用来基准测试的代码是...

#!/usr/bin/env python

import time
import os
import urllib2

#5mb mp3 file
testdl = "http://traffic.libsyn.com/timferriss/Arnold_5_min_-_final.mp3" 

chunkmulti = 1
numpass = 5

while (chunkmulti < 207):
    passtime = 0
    passattempt = 1
    while (passattempt <= numpass):
        start = time.time()
        req = urllib2.urlopen(testdl)
        CHUNK = chunkmulti * 1024
        with open("test.mp3", 'wb') as fp:
            while True:
                chunk = req.read(CHUNK)
                if not chunk: break
                fp.write(chunk)
        end = time.time()
        passtime += end - start
        os.remove("test.mp3")
        passattempt += 1
    print "Chunk size multiplier ", chunkmulti , " took ", passtime / passattempt, " seconds"
    chunkmulti += 1

结果不是决定性的。这是第一批结果...

Chunk size multiplier  1  took  13.9629709721  seconds
Chunk size multiplier  2  took  8.01173728704  seconds
Chunk size multiplier  3  took  10.3750542402  seconds
Chunk size multiplier  4  took  7.11076325178  seconds
Chunk size multiplier  5  took  11.3685477376  seconds
Chunk size multiplier  6  took  6.86864703894  seconds
Chunk size multiplier  7  took  14.2680369616  seconds
Chunk size multiplier  8  took  7.93746650219  seconds
Chunk size multiplier  9  took  6.81188523769  seconds
Chunk size multiplier  10  took  7.54047352076  seconds
Chunk size multiplier  11  took  6.84347498417  seconds
Chunk size multiplier  12  took  7.88792568445  seconds
Chunk size multiplier  13  took  7.37244099379  seconds
Chunk size multiplier  14  took  8.15134423971  seconds
Chunk size multiplier  15  took  7.1664044857  seconds
Chunk size multiplier  16  took  10.9474172592  seconds
Chunk size multiplier  17  took  7.23868894577  seconds
Chunk size multiplier  18  took  7.66610199213  seconds

结果以这种方式持续到 207kb 的块大小

所以我将块大小设置为 6kb。可能会尝试将其与 wget next 进行基准测试...