【发布时间】:2018-07-09 00:58:20
【问题描述】:
我使用此脚本的目标是导入 csv 文件。然后使用该信息运行搜索查询并使用美丽的汤 4 抓取结果,然后将抓取的信息导出到不同的 csv 文件中。
我有 2 个问题,虽然第一个我无法让 While:True 语句起作用,第二个也是最重要的。我在增加行变量方面没有成功。 (即 Run1 (stl[0]+1) Run2 (stl[1])+1) Run3 (stl[2]+1)。
import mechanize
import cookielib
from bs4 import BeautifulSoup
import csv
While:True
ifile = open('license.csv', "rb")
reader = csv.reader(ifile)
sl1 = []
sl2 = []
for row in reader:
sl1.append((row[0]))
sl2.append((row[1]))
ifile.close()
# Browser
br = mechanize.Browser()
# Cookie Jar
cj = cookielib.LWPCookieJar()
br.set_cookiejar(cj)
# Browser options
br.set_handle_equiv(True)
br.set_handle_gzip(True)
br.set_handle_redirect(True)
br.set_handle_referer(True)
br.set_handle_robots(False)
# Follows refresh 0 but not hangs on refresh > 0
br.set_handle_refresh(mechanize._http.HTTPRefreshProcessor(), max_time=1)
# Want debugging messages?
#br.set_debug_http(True)
#br.set_debug_redirects(True)
#br.set_debug_responses(True)
# User-Agent (this is cheating, ok?)
br.addheaders = [('User-agent', 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.1) Gecko/2008071615 Fedora/3.0.1-1.fc9 Firefox/3.0.1')]
# Open some site, let's pick a random one, the first that pops in mind:
r = br.open('https://secure.utah.gov/llv/search/index.html')
html = r.read()
# Show the source
print html
# or
print br.response().read()
# Show the html title
print br.title()
# Show the response headers
print r.info()
# or
print br.response().info()
# Show the available forms
for f in br.forms():
print f
# Select the first (index zero) form
br.select_form(nr=1)
# Let's search
br.form['licenseNumberCore']=sl1[0]+1
br.form['licenseNumberFourDigit']=sl2[0]+1
br.submit()
print br.response().read()
# Looking at some results in link format
for l in br.links(url_regex=sl1[0]+1):
print l
# Testing presence of link (if the link is not found you would have to
# handle a LinkNotFoundError exception)
br.find_link(text_regex=sl1[0]+1)
print br.find_link()
# Actually clicking the link
req = br.click_link(text_regex=sl1[0]+1)
br.open(req)
print br.response().read()
print br.geturl()
soup = BeautifulSoup(br.response().read(), 'html.parser')
soup.find("td", text="Name:").find_next_sibling("td").text
pname = soup.find("td", text="Name:").find_next_sibling("td").text
print(pname)
soup.find("td", text="License Number:").find_next_sibling("td").text
plic = soup.find("td", text="License Number:").find_next_sibling("td").text
print(plic)
soup.find("td", text="License Status:").find_next_sibling("td").text
pstat = soup.find("td", text="License Status:").find_next_sibling("td").text
print(pstat)
soup.find("td", text="Expiration Date:").find_next_sibling("td").text
pexp = soup.find("td", text="Expiration Date:").find_next_sibling("td").text
print(pexp)
# open a csv file with append, so old data will not be erased
with open('license_check.csv', 'a') as csv_file:
writer = csv.writer(csv_file)
writer.writerow([pname, plic, pstat, pexp])
if sh1[0]+1 == 23
break
else
continue
【问题讨论】:
-
您的 python 存在一些基本问题。通常认为最好的做法是发布一个最小的工作示例来演示您要解决的问题。在这种情况下,您还没有生成一个工作示例。我鼓励你修复它。
-
欢迎来到 Stack Overflow。请使用tour 并阅读有关如何创建How to Ask 的信息,尤其是在其他 cmets 中提到的如何创建minimal reproducible example
-
python while 循环的语法是 while True:你的代码有 while:True...更正它
标签: python csv beautifulsoup mechanize