无法使用 python 请求库下载压缩的 csv 文件

Question

我正在尝试从 https://www1.nseindia.com/products/content/derivatives/equities/archieve_fo.htm 下载压缩的 csv 文件。

要查看文件，请选择报告：Bhavcopy & 和日期：18-06-2021。它提供了 zip 文件 fo18JUN2021bhav.csv.zip 与 url https://www1.nseindia.com/content/historical/DERIVATIVES/2021/JUN/fo18JUN2021bhav.csv.zip

现在，当我使用 requests 库对压缩的 csv url（带有适当的标头）进行 get 调用时，我收到 [404] 错误。

有没有办法以编程方式下载它？谢谢！

Answer 1

要获取文件，请设置User-Agent 和Referer HTTP 标头：

import requests
from bs4 import BeautifulSoup

# change filetype and date you want to search for:
url = "https://www1.nseindia.com/ArchieveSearch?h_filetype=fobhav&date=18-06-2021&section=FO"

headers = {
    "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:89.0) Gecko/20100101 Firefox/89.0",
    "Referer": "https://www1.nseindia.com/products/content/derivatives/equities/archieve_fo.htm",
}

soup = BeautifulSoup(requests.get(url, headers=headers).content, "html.parser")
csv_url = "https://www1.nseindia.com" + soup.a["href"]

print("Downloading {}...".format(csv_url))
with open(csv_url.split("/")[-1], "wb") as f_out:
    f_out.write(requests.get(csv_url, headers=headers).content)
print("Done.")

下载fo18JUN2021bhav.csv.zip文件：

Downloading https://www1.nseindia.com/content/historical/DERIVATIVES/2021/JUN/fo18JUN2021bhav.csv.zip...
Done.

$ ls -alF fo18JUN2021bhav.csv.zip 
-rw-r--r-- 1 root root 632963 june 20 23:11 fo18JUN2021bhav.csv.zip