【发布时间】：2020-10-06 12:27:42
【问题描述】：
我正在尝试从该页面返回结果的每一页中获取数据。
https://www.azjobconnection.gov/ada/mn_warn_dsp.cfm?def=false&securitysys=on
很难验证我是否已获取所有内容，因为当您点击下一页按钮时，所有内容都出现故障。唯一按年份排序的页面是第一页。后续页面的数据超出了最初选择的范围。例如，如果您在搜索页面输入 01/01/2020，则返回的第一页将只有 2020 年 1 月及以后的条目。但是一旦你点击下一页，你会得到 2016 年、2018 年、....的条目。
我只想能够在 2020 年 1 月 1 日输入并获取日期范围（2020 年 1 月 - 今天）内的所有数据。我尝试输入结束日期，但没有帮助。我知道我需要做更多的工作才能获得我所寻找的数据；但现在我只需要确保从所有返回的页面中获取每个条目。该网站显示，从 2020 年 1 月 1 日到今天，大约有 134 个 obs。我的输出只有约 50 个唯一值。
我对网络抓取非常陌生，所以如果您能保持建议非常简单，我将不胜感激。谢谢
# Imports
from bs4 import BeautifulSoup
from requests import Session
# Session Object
session = Session()
# Add a user agent, so the request looks more human like.
session.headers.update({
"user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36"
})
# Initial sesssion, you need to fetch the url first, so the authenticity
# token can be parsed out of the html
init_session = session.get(url="https://www.azjobconnection.gov/ada/mn_warn_dsp.cfm?def=false")
# Beautiful soup object, used for HTML parsing
soup = BeautifulSoup(init_session.content, "html.parser")
# Get all of the input tags
inputs = soup.findAll('input')
# Upon running, we see that the authenticity token, is the first element in the array.
authenticty_token = inputs[0]['value']
# Now we can make our request!
data = {
"authenticity_token" : authenticty_token,
"coname": "",
"coName_ADAdefault": "",
"coName_verify_char[0|50]": "The value you have supplied for Company Name is too long.",
"city": "",
"city_ADAdefault": "",
"city_verify_char[0|45]": "The value you have supplied for City is too long.",
"zip": "",
"zip_ADAdefault": "",
"zip_verify_char[0|10]": "The value you have supplied for Zip/Postal Code is too long.",
"sda": "",
"startdate": "01/01/2020",
"startDate_ADAdefault": "mm/dd/yyyy",
"startDate_verify_date4": "",
"startDate_verify_char[0|45]": "The value you have supplied for Start Date is too long.",
"enddate": "mm/dd/yyyy",
"endDate_ADAdefault": "mm/dd/yyyy",
"endDate_verify_date4": "",
"endDate_verify_char[0|45]": "The value you have supplied for End Date is too long.",
"layoffType": "y",
"search": "Search",
"old_choice": 1,
"ZIP_prev": "",
"def_prev": "false",
"CITY_prev": "",
"SDA_prev": "",
"STARTDATE_prev": "",
"CONAME_prev": "",
"ENDDATE_prev": "",
"FormName": "Form0",
}
# Get the data
get_warn_data = session.post("https://www.azjobconnection.gov/ada/mn_warn_dsp.cfm?securitysys=on&start_row=1&max_rows=25&FormID=0", data=data)
soup = BeautifulSoup(get_warn_data.content, "html.parser")
# These are all the hash taags you need to go to to get data and the links
# for the other pages you possibly need to go to to get data.
targets = soup.find_all('a', href = True)
import re
regex = re.compile("(?=\").*(?<=\")")
targets2 = [re.search(regex, str(a)).group(0) for a in targets]
# These are the url parts to which you need to append the url_head then run a
# request on the entire url; also, python shows & where it should just be
# &; making that substitution here
# FIRST SET OF IDs to pull data on; will append businesses gathered from
# the other pages
bus1 = [a for a in targets2 if 'mn_warn_dsp' in a and 'hash' in a]
bus1 = [re.sub(r"\"", "", a) for a in bus1]
bus1 = [re.sub(r"&", "&", a) for a in bus1]
# Most queries will return multiple pages of business; need to loop through the pages to get
# all of the businesses; business from here will be combined with business from first
# page above;
more_pages = [a for a in targets2 if 'start_row' in a and 'max_row' in a and 'orderby' in a]
more_pages = [re.sub(r"\"", "", a) for a in more_pages]
more_pages = [re.sub(r"&", "&", a) for a in more_pages]
# Getting rid of ada... part from all additional page url parts; will attach
# to all below
more_pages = [re.sub(r"/ada/mn_warn_dsp.cfm", "", a) for a in more_pages]
# url prefixes for businesses already have the mn_warn_dsp part; the additional
# page urls do not; for url parts in "more_pages", append url_head_pages; for
# businesses, append the url_head
url_head = "https://www.azjobconnection.gov/ada/"
url_head_pages = "https://www.azjobconnection.gov/ada/mn_warn_dsp.cfm"
# Going to additional pages and getting all the ids/hash number
# Here, I'm just repeating on subsequent pages what I did on the first page; no
# need to check for additional pages here. Just going through each page
# and grabbing the hash marks
hash_hold = []
for page in more_pages:
test123 = url_head_pages+page # url of the page with additional businesses
work_now = session.get(url = test123) # getting html to parse
soup = BeautifulSoup(work_now.content, "html.parser")
targets = soup.find_all('a', href = True) # finding all ids/hash values
regex = re.compile("(?=\").*(?<=\")") # getting stuff between double quotes
targets2 = [re.search(regex, str(a)).group(0) for a in targets]
bus2 = [a for a in targets2 if 'mn_warn_dsp' in a and 'hash' in a]
bus2 = [re.sub(r"\"", "", a) for a in bus1]
bus2 = [re.sub(r"&", "&", a) for a in bus1]
hash_hold.append(bus2)
# hash_hold has hash/ids from subsequent pages and the bus1 has hash/ids
# from the first page; joining them all together here to get all hash/ids
# we need
hash_hold.append(bus1)
# These are all of the hash/ids/businesses I captured; notice it is much smaller than the number of returned results if you search from Jan. 1, 2020 to today
from pandas.core.common import flatten
businesses_use = list(flatten(hash_hold))
【问题讨论】：
标签： python-3.x web-scraping beautifulsoup python-requests