【问题标题】:download files from website with cloudflare using Python使用 Python 从带有 cloudflare 的网站下载文件
【发布时间】:2016-08-25 05:15:30
【问题描述】:
我正在尝试使用 python 从具有 Cloud Flare 的网站下载 mp3 文件。我知道 python 的“cfscrape”模块,但是如何使用它从 url 下载文件。
【问题讨论】:
标签:
python
web-scraping
cloudflare
download
【解决方案1】:
这里用于从包含链接的“csv”文件中下载多个文件。
注意:我从这里得到了帮助:Python download files by links stored in csv
import csv
import os
import sys
import cfscrape
scraper = cfscrape.create_scraper()
filename = 'nazm_urls.csv'
with open(filename, 'rb') as f:
reader = csv.reader(f)
try:
for row in reader:
if 'http' in row[0]:
reverse = row[0][::-1]
i = reverse.index('/')
tmp = reverse[0:i]
cfurl = scraper.get(row[0]).content
if not os.path.exists("./"+tmp[::-1]):
with open(tmp[::-1], 'wb') as f:
f.write(cfurl)
f.close()
else:
print("file: ", tmp[::-1], "already exists")
except csv.Error as e:
sys.exit('file %s, line %d: %s' % (filename, reader.line_num, e))
【解决方案2】:
我明白了。
import cfscrape
scraper = cfscrape.create_scraper()
url = 'the website url'
cfurl = scraper.get(url).content
name = url.split('/')[-1]
with open(name, 'wb') as f:
f.write(cfurl)