使用漂亮的汤刮掉具有多个链接但没有“下一步”按钮的网站

Question

我对 python 很陌生（三天），我偶然发现了一个我无法用 google/youtube 解决的问题。我想从National Governors Association 获取所有美国州长的背景数据，并将其保存到 csv 文件中。

我已经设法抓取了所有调控器的列表，但要获得更多详细信息，我需要单独进入每个调控器的页面并保存数据。我在网上找到了代码建议，它利用“下一步”按钮或 url 结构来遍历多个站点。但是，该网站没有下一步按钮，并且 url 链接不遵循可循环的结构。所以我被困住了。

如果我能得到任何帮助，我将不胜感激。我想提取每个州长页面中主要文本（“地址”标签中的办公日期、学校等）上方的信息，例如this one。

这是我目前得到的：

import bs4 as bs
import urllib.request
import pandas as pd

url = 'https://www.nga.org/cms/FormerGovBios?begincac77e09-db17-41cb-9de0-687b843338d0=10&endcac77e09-db17-41cb-9de0-687b843338d0=9999&pagesizecac77e09-db17-41cb-9de0-687b843338d0=10&militaryService=&higherOfficesServed=&religion=&lastName=&sex=Any&honors=&submit=Search&college=&firstName=&party=&inOffice=Any&biography=&warsServed=&'

sauce = urllib.request.urlopen(url).read()
soup = bs.BeautifulSoup(sauce, "html.parser")

#dl list of all govs
dfs = pd.read_html(url, header=0)
for df in dfs:
    df.to_csv('governors.csv')

#dl links to each gov
table = soup.find('table', 'table table-striped table-striped')
links = table.findAll('a')
with open ('governors_links.csv', 'w') as r:
    for link in links:
        r.write(link['href'])
        r.write('\n')
    r.close()

#enter each gov page and extract data in the "address" tag(s)
#save this in a csv file

Answer 1

我假设您已在名为 links 的列表中找到所有链接。
你可以这样来一一获取你想要的所有Governors的数据：

for link in links:
    r = urllib.request.urlopen(link).read()
    soup = bs.BeautifulSoup(r, 'html.parser')
    print(soup.find('h2').text)  # Name of Governor
    for p in soup.find('div', {'class': 'col-md-3'}).findAll('p'):
        print(p.text.strip())  # Office dates, address, phone, ...
    for p in soup.find('div', {'class': 'col-md-7'}).findAll('p'):
        print(p.text.strip())  # Family, school, birth state, ...

编辑：

将您的 links 列表更改为

links = ['https://www.nga.org' + x.get('href') for x in table.findAll('a')]

Answer 2

这可能有效。自从我在工作以来，我还没有完全完成测试，但这应该是你的起点。

import bs4 as bs
import requests
import re
def is_number(s):
    try:
        int(s)
        return True
    except ValueError:
        return False

def main():
    url = 'https://www.nga.org/cms/FormerGovBios?inOffice=Any&state=Any&party=&lastName=&firstName=&nbrterms=Any&biography=&sex=Any&religion=&race=Any&college=&higherOfficesServed=&militaryService=&warsServed=&honors=&birthState=Any&submit=Search'

    sauce = requests.get(url).text
    soup = bs.BeautifulSoup(sauce, "html.parser")
    finished = False
    csv_data = open('Govs.csv', 'a')
    csv_data.write('Name,Address,OfficeDates,Success,Address,Phone,Fax,Born,BirthState,Party,Schooling,Email')
    try:
        while not finished:
        #dl links to each gov
            table = soup.find('table', 'table table-striped table-striped')
            links = table.findAll('a')
            for link in links:
                info_array = []
                gov = {}
                name = link.string
                gov_sauce =  requests.get(r'https://nga.org'+link.get('href')).text
                gov_soup = bs.BeautifulSoup(gov_sauce, "html.parser")
                #print(gov_soup)
                office_and_stuff_info = gov_soup.findAll('address')
                for address in office_and_stuff_info:
                    infos = address.findAll('p')
                    for info in infos:
                        tex = re.sub('[^a-zA-Z\d:]','',info.text)
                        tex = re.sub('\\s+',' ',info.text)
                        tex = tex.strip()
                        if tex: 
                            info_array.append(tex)
                info_array = list(set(info_array))
                gov['Name'] = name
                secondarry_address = ''
                gov['Address'] = ''
                for line in info_array:
                    if 'OfficeDates:' in line:
                        gov['OfficeDates'] = line.replace('OfficeDates:','').replace('-','')
                    elif 'Succ' or 'Fail' in line:
                        gov['Success'] = line
                    elif 'Address' in line:
                        gov['Address'] = line.replace('Address:','')
                    elif 'Phone:' or 'Phone ' in line:
                        gov['Phone'] = line.replace('Phone ','').replace('Phone: ','')
                    elif 'Fax:' in line:
                        gov['Fax'] = line.replace('Fax:','')
                    elif 'Born:' in line:
                        gov['Born'] = line.replace('Born:','')
                    elif 'Birth State:' in line:
                        gov['BirthState'] = line.replace('BirthState:','')
                    elif 'Party:' in line:
                        gov['Party'] =  line.replace('Party:','')
                    elif 'School(s)' in line:
                        gov['Schooling'] = line.replace('School(s):','').replace('School(s) ')
                    elif 'Email:' in line:
                        gov['Email'] = line.replace('Email:','')
                    else:
                        secondarry_address = line
                gov['Address'] = gov['Address'] + secondarry_address
                data_line = gov['Name'] +','+gov['Address'] +','+gov['OfficeDates'] +','+gov['Success'] +','+gov['Address'] +','+ gov['Phone'] +','+ gov['Fax'] +','+gov['Born'] +','+gov['BirthState'] +','+gov['Party'] +','+gov['Schooling'] +','+gov['Email']
                csv_data.write(data_line)
            next_page_link = soup.find('ul','pagination center-blockdefault').find('a',{'aria-label':'Next'})
            if next_page_link.parent.get('class') == 'disabled':
                finished = True
            else:

                url = r'https://nga.org'+next_page_link.get('href')
                sauce = requests.get(url).text
                soup = bs.BeautifulSoup(sauce,'html.parser')
    except:
        print('Code failed.')
    finally:
        csv_data.close()
if __name__ == '__main__':
    main()