使用请求通过 http 下载文件时的进度条

Question

我需要下载一个相当大的 (~200MB) 文件。我想出了如何使用here 下载和保存文件。最好有一个进度条来知道已经下载了多少。我找到了ProgressBar，但我不确定如何将两者结合在一起。

这是我尝试过的代码，但它不起作用。

bar = progressbar.ProgressBar(max_value=progressbar.UnknownLength)
with closing(download_file()) as r:
    for i in range(20):
        bar.update(i)

Answer 1

我建议你试试tqdm，它非常好用。 使用requests库下载的示例代码：

from tqdm import tqdm
import requests

url = "http://www.ovh.net/files/10Mb.dat" #big file test
# Streaming, so we can iterate over the response.
response = requests.get(url, stream=True)
total_size_in_bytes= int(response.headers.get('content-length', 0))
block_size = 1024 #1 Kibibyte
progress_bar = tqdm(total=total_size_in_bytes, unit='iB', unit_scale=True)
with open('test.dat', 'wb') as file:
    for data in response.iter_content(block_size):
        progress_bar.update(len(data))
        file.write(data)
progress_bar.close()
if total_size_in_bytes != 0 and progress_bar.n != total_size_in_bytes:
    print("ERROR, something went wrong")

Answer 2

您似乎需要获取远程文件大小 (answered here) 来计算您的距离。

然后，您可以在处理每个块时更新进度条...如果您知道块的总大小和大小，您可以确定何时更新进度条。

Answer 3

Progress Bar Usage 页面上的示例与代码实际需要的内容之间似乎存在脱节。

在以下示例中，请注意使用maxval 而不是max_value。还要注意使用.start() 来初始化栏。这已在Issue 中注明。

n_chunk 参数表示在遍历请求迭代器时一次流式传输多少个 1024 kb 块。

import requests
import time

import numpy as np

import progressbar


url = "http://wikipedia.com/"

def download_file(url, n_chunk=1):
    r = requests.get(url, stream=True)
    # Estimates the number of bar updates
    block_size = 1024
    file_size = int(r.headers.get('Content-Length', None))
    num_bars = np.ceil(file_size / (n_chunk * block_size))
    bar =  progressbar.ProgressBar(maxval=num_bars).start()
    with open('test.html', 'wb') as f:
        for i, chunk in enumerate(r.iter_content(chunk_size=n_chunk * block_size)):
            f.write(chunk)
            bar.update(i+1)
            # Add a little sleep so you can see the bar progress
            time.sleep(0.05)
    return

download_file(url)

编辑：解决了关于代码清晰度的评论。
EDIT2：固定逻辑，因此栏在完成时报告 100%。感谢 leovp's answer 使用 1024 kb 块大小。

Answer 4

tqdm 有答案。

def download(url, fname):
    resp = requests.get(url, stream=True)
    total = int(resp.headers.get('content-length', 0))
    with open(fname, 'wb') as file, tqdm(
            desc=fname,
            total=total,
            unit='iB',
            unit_scale=True,
            unit_divisor=1024,
    ) as bar:
        for data in resp.iter_content(chunk_size=1024):
            size = file.write(data)
            bar.update(size)

Gits：https://gist.github.com/yanqd0/c13ed29e29432e3cf3e7c38467f42f51

Answer 5

tqdm 包现在包含一个专门为这种情况设计的函数：wrapattr。您只需包装对象的read（或write）属性，其余的由tqdm 处理；没有弄乱块大小或类似的东西。这是一个简单的下载功能，将所有内容与requests 放在一起：

def download(url, filename):
    import functools
    import pathlib
    import shutil
    import requests
    from tqdm.auto import tqdm
    
    r = requests.get(url, stream=True, allow_redirects=True)
    if r.status_code != 200:
        r.raise_for_status()  # Will only raise for 4xx codes, so...
        raise RuntimeError(f"Request to {url} returned status code {r.status_code}")
    file_size = int(r.headers.get('Content-Length', 0))

    path = pathlib.Path(filename).expanduser().resolve()
    path.parent.mkdir(parents=True, exist_ok=True)

    desc = "(Unknown total file size)" if file_size == 0 else ""
    r.raw.read = functools.partial(r.raw.read, decode_content=True)  # Decompress if needed
    with tqdm.wrapattr(r.raw, "read", total=file_size, desc=desc) as r_raw:
        with path.open("wb") as f:
            shutil.copyfileobj(r_raw, f)

    return path

Answer 6

也可以使用python库enlighten，功能强大，提供彩色进度条，在Linux、Windows下都能正常运行。

以下是代码 + 现场截屏。这段代码可以运行here on repl.it。

import math
import requests, enlighten

url = 'https://upload.wikimedia.org/wikipedia/commons/a/ae/Arthur_Streeton_-_Fire%27s_on_-_Google_Art_Project.jpg?download'
fname = 'image.jpg'

# Should be one global variable
MANAGER = enlighten.get_manager()

r = requests.get(url, stream = True)
assert r.status_code == 200, r.status_code
dlen = int(r.headers.get('Content-Length', '0')) or None

with MANAGER.counter(color = 'green', total = dlen and math.ceil(dlen / 2 ** 20), unit = 'MiB', leave = False) as ctr, \
     open(fname, 'wb', buffering = 2 ** 24) as f:
    for chunk in r.iter_content(chunk_size = 2 ** 20):
        print(chunk[-16:].hex().upper())
        f.write(chunk)
        ctr.update()

输出（+ascii-video）

Answer 7

用您已经下载的大小计算文件大小会找到您的距离。或者你可以使用 tqdm。

Answer 8

由于某种原因，我在处理 zip 文件时无法通过请求获取文件大小，所以我使用 urllib 来获取它

# A simple downloader with progress bar

import requests
from tqdm import tqdm
import zipfile
from urllib.request import urlopen

url = "https://web.cs.dal.ca/~juanr/downloads/malnis_dataset.zip"
block_size = 1024 #1 Kibibyte

filename = url.split("/")[-1]
print(f"Downloading {filename}...")
site = urlopen(url)
meta = site.info()
# Streaming, so we can iterate over the response.
response = requests.get(url, stream=True)
total_size_in_bytes = int(meta["Content-Length"])
progress_bar = tqdm(total = total_size_in_bytes, unit='iB', unit_scale=True)
with open('test.dat', 'wb') as file:
    for data in response.iter_content(block_size):
        progress_bar.update(len(data))
        file.write(data)
progress_bar.close()
print("Download complete")
print(f"Extracting {filename}...")
zip = zipfile.ZipFile(filename, "r")
zip.extractall()
zip.close()
print("Extracting complete")