【发布时间】：2013-05-16 18:16:36
【问题描述】：
我在使用 Python mechanize 循环下载多个文件时遇到问题。我也在使用 Beautiful Soup 4。任何一个包的文档似乎都没有答案。
这是我的代码 - 请跳至实际循环。我将所有内容都包括在内以供参考：
import mechanize, cookielib, os, time
from bs4 import BeautifulSoup
fcList = ['abandoned mine land inventory points', 'abandoned mine land inventory polygons', \
'abandoned mine land inventory sites', 'coal mining operations', 'coal pillar location-mining', \
'industrial mineral mining operations', 'longwall mining panels', 'mine drainage treatment/land recycling project locations', \
'mined out areas', 'residual waste operations', 'underground mining permit']
dlLink = 'FTP Download'
dloadPath = 'C:\\Users\\SomeGuy\\Downloads'
# Browser
br = mechanize.Browser()
# Cookie Jar
cj = cookielib.LWPCookieJar()
br.set_cookiejar(cj)
# Select the first (index zero) form
br.select_form(nr=0)
# Input form data
br.form['Keyword']='mining'
br.submit()
html = br.response().read()
# Pass html to beautiful soup for parse
soup = BeautifulSoup(html)
htmlinks = soup.findAll("a")
# Find links with desired text
for htmlink in htmlinks:
string = str(htmlink.string)
if string.lower() in fcList:
print "Matched link!", string + ". attempting download...\n"
try:
req = br.click_link(text = string)
br.open(req)
print "URL: " + str(br.geturl)
html = br.response().read()
soup = BeautifulSoup(html)
the_tag = soup.find('a', text=dlLink)
fileURL = the_tag.get('href')
print fileURL
# attempt download
fnam = string.replace(" ", "_")
fnam = fnam.replace("/", "_")
f = br.retrieve(fileURL, os.path.join(dloadPath, fnam + ".zip"))
print f + "\n"
br.back()
except:
print "An unknown error occurred."
输出：
>>>
Matched link! Abandoned Mine Land Inventory Points. attempting download...
URL: <bound method Browser.geturl of <mechanize._mechanize.Browser instance at 0x02D9D7B0>>
http://www.pasda.psu.edu/data/dep/AMLInventoryPoints2013_04.zip
An unknown error occurred.
Matched link! Abandoned Mine Land Inventory Polygons. attempting download...
An unknown error occurred.
Matched link! Abandoned Mine Land Inventory Sites. attempting download...
An unknown error occurred.
Matched link! Coal Mining Operations. attempting download...
An unknown error occurred.
Matched link! Coal Pillar Location-Mining. attempting download...
An unknown error occurred.
Matched link! Industrial Mineral Mining Operations. attempting download...
An unknown error occurred.
Matched link! Longwall Mining Panels. attempting download...
An unknown error occurred.
Matched link! Mine Drainage Treatment/Land Recycling Project Locations. attempting download...
An unknown error occurred.
Matched link! Mined Out Areas. attempting download...
An unknown error occurred.
Matched link! Residual Waste Operations. attempting download...
An unknown error occurred.
Matched link! Underground Mining Permit. attempting download...
An unknown error occurred.
>>>
我认为问题可能是由于下载之间没有等待时间。无论我选择哪个，此代码都会成功下载循环中的第一个文件。或者可能是我不知道的其他错误 - 我昨天刚刚下载了 mechanize 和 beautifulsoup！
【问题讨论】：
标签： python loops download beautifulsoup mechanize