使用具有服务器限制的python流下载文件

Question

我尝试使用python从服务器下载文件，有时文件非常大，我想要一些进度条，我可以想出的一种方法是在流中下载，这样我可以打印进度。目前我已经尝试了标准的 urlopen、urlretrieve 和 requests 模块（开启了流）。

很明显，urlopen 不能下载流中的文件，请求模块支持这个，但是服务器对我一次可以下载的文件有限制（它的限制是 1）。所以每次，我尝试使用请求，它只会让网页告诉我等待，还有其他方法吗？

Answer 1

我最近使用此功能下载了许多类型的媒体：

import sys
import requests
import time
def download_resource(domain, url, file_name = None, download = True):
    cookies = {}
    s = requests.Session()

    s.config['keep_alive'] = True
    #add your own cookies here, I have a specific function I call
    #for my application but yours is different
    r = s.get(url, cookies = cookies, stream = True)

    if not r.ok:
        print "error in downloading"
        return -1

    file_size = int(r.headers['content-length']) 

    if not file_name:
        try:
            temp = r.headers['content-disposition']
        except Exception as e:
            pass
            #failing download
            return -1
        else:
            if not temp:
                return -1
            else:
                file_name = temp.split("filename=")[-1]
                return_obj["filename"] = file_name
    #print "File size:", file_size
    #print "\n", str(self.entire_size / float(1024*1024*1024)), "\n"
    print "Downloading:", file_name
    if download:
        with open(file_name, "wb") as fh:
            count = 1
            chunk_size = 1048576
            start_time = time.time()
            try:
                for block in r.iter_content(chunk_size):
                    total_time = time.time() - start_time
                    percent = count*chunk_size/float(file_size) * 100.0
                    fraction = int(percent/5)

                    download_speed = 1.0 / total_time

                    sys.stdout.write('\r')
                    sys.stdout.write("[%-20s] %d%%  %3.2f MB/s       " % ('='* fraction , percent, download_speed))
                    sys.stdout.flush()
                    if not block:
                        break
                    fh.write(block)
                    count += 1
                    start_time = time.time()
            except Exception as e:
                print e
            finally:    
                #close up the stream
                r.close()