无法从某个网站下载文件

Question

我在 python 中编写了一些代码来从网页下载文件。由于我不知道如何从任何站点下载文件，所以我只能从该站点抓取文件链接。如果有人可以帮助我实现这一目标，我将非常感谢他。提前非常感谢。

链接到该网站：web_link

这是我的尝试：

from bs4 import BeautifulSoup
import requests

response = requests.get("http://usda.mannlib.cornell.edu/MannUsda/viewDocumentInfo.do?documentID=1194")
soup = BeautifulSoup(response.text,"lxml")
for item in soup.select("#latest a"):
    print(item['href'])

在执行时，上述脚本会为这些文件生成四个不同的 url。

Answer 1

你可以使用request.get:

import requests
from bs4 import BeautifulSoup

response = requests.get("http://usda.mannlib.cornell.edu/MannUsda/"
                        "viewDocumentInfo.do?documentID=1194")
soup = BeautifulSoup(response.text, "lxml")
for item in soup.select("#latest a"):
    filename = item['href'].split('/')[-1]
    with open(filename, 'wb') as f:
        f.write(requests.get(item['href']).content)

Answer 2

您可以使用标准库的urllib.request.urlretrieve()，但是，由于您已经在使用requests，您可以在此处重新使用会话（download_file 主要取自this answer）：

from bs4 import BeautifulSoup
import requests


def download_file(session, url):
    local_filename = url.split('/')[-1]

    r = session.get(url, stream=True)
    with open(local_filename, 'wb') as f:
        for chunk in r.iter_content(chunk_size=1024):
            if chunk: # filter out keep-alive new chunks
                f.write(chunk)

    return local_filename


with requests.Session() as session:
    response = session.get("http://usda.mannlib.cornell.edu/MannUsda/viewDocumentInfo.do?documentID=1194")
    soup = BeautifulSoup(response.text,"lxml")
    for item in soup.select("#latest a"):
        local_filename = download_file(session, item['href'])
        print(f"Downloaded {local_filename}")