无法在 CSV 中存储信息（Python Web Scraping）

Question

我的代码没有将结果正确存储到我创建的 csv 文件中。

我需要从U.S. Congress website 中提取每张账单的号码、赞助商和派对的数据。

在解释器中运行代码时，它可以正常工作并给我想要的结果。但是，在我创建的 csv 文件中，我遇到了以下问题之一：

• 同一赞助方每张帐单（正确的帐单编号，但他们都共享同一个赞助方）

SPONS  PARTY NBILL
Name   D     7402
Name   D     7401
...

有趣的是，我找到的名字（Grijalva，Raul）对应于 Bill 7302。

• 正确的Sponsor-Party，但只是第100个账单，即每100个Sponsor-Party我有7402；7302等等。

如上所述，不同的赞助商和参与方，但账单数量仅每 100 对赞助商/参与方发生变化，并且变为 100 乘 100（前 100 对为 7402，第二对为 7302，依此类推）

• 正确的赞助方但没有账单，以下代码会发生这种情况

编辑：如果我将Congress=[-]+[-]+[-] 放在代码的末尾，我就属于第一种情况。

 with open('115congress.csv', 'w') as f:
        fwriter=csv.writer(f, delimiter=';')
        fwriter.writerow(['SPONS', 'PARTY', 'NBILL'])
        BillN=[]
        Spons=[]
        Party=[]
        for j in range(1, 114):
            hrurl='https://www.congress.gov/search?q=%7B%22source%22%3A%22legislation%22%2C%22congress%22%3A%22115%22%2C%22type%22%3A%22bills%22%7D&page='+str(j)
            hrpage=requests.get(hrurl, headers=headers)
            data=hrpage.text
            soup=BeautifulSoup(data, 'lxml')
            for q in soup.findAll('span', {'class':'result-item'}):
                for a in q.findAll('a', href=True, text=True, target='_blank'):
                    secondindex=secondindex+1
                    if (secondindex/2).is_integer():
                        continue
                    Spons=a.text
                    print(Spons)
                    SPONS=Spons
                    if 'R' in Spons:
                        Party='Republican'
                    if 'D' in Spons:
                        Party='Democratic'
                    print(Party)
                    PARTY=Party
                    Congress115=[SPONS]+[PARTY]
                    fwriter.writerow(Congress115)
            for r in soup.findAll('span', {'class':'result-heading'}):
                index=index+1
                if (index/2).is_integer():
                    continue
                Bill=r.findNext('a')
                BillN=Bill.text
                print(BillN)
                NBILL=BillN
                Congress115= [SPONS]+[PARTY]+[NBILL]
                fwriter.writerow(Congress115)

    f.close()

如何修复写入 CSV 的代码，以免出现这些问题？

Answer 1

我不明白您对代码提出的所有问题，因为我无法重现您的错误。但是，我认为您的代码存在一些问题，我想向您展示另一种可能的方法。

我认为您的主要错误之一是将变量多次写入您的 csv 文件。此外，如果您只在包含派对缩写和名称的字符串中查找单个字符，则会收到很多关于派对的虚假条目。

假设您要从每个条目中提取 bill_nr、spons 和 party，您可以执行以下操作（请参阅代码中的 cmets）：

import csv
import requests
from bs4 import BeautifulSoup

for j in range(1,114):
  hrurl=f'https://www.congress.gov/search?q=%7B%22source%22%3A%22legislation%22%2C%22congress%22%3A%22115%22%2C%22type%22%3A%22bills%22%7D&page={j}'
  hrpage=requests.get(hrurl)
  data=hrpage.text
  soup=BeautifulSoup(data, 'html5lib')

  # get the main div, that contains all entries on the page
  main_div = soup.find('div', {'id':'main'})
  # every entry is within a <li> element
  all_li = main_div.findAll('li', {'class':'expanded'})

  # iterate over <li>-elements
  for li in all_li:
    # get BILL_NR
    bill_nr_raw = li.find('span', {'class':'result-heading'}).text
    # I assume only the first part is the Nr, so you could extract it with the following
    bill_nr = bill_nr_raw.split()[0]
    # get SPONS
    spons_raw = li.find('span', {'class':'result-item'})
    spons = spons_raw.find('a').text

    # get PARTY
    # check if the string starts with one of the following to ensure you pick the right party
    if spons.startswith('Rep'):
      party = 'Republican'
    elif spons.startswith('Dem'):
      party = 'Democratic'

    # put all the information you extracted from this single entry (=<li>-element) into a list and write that list (=one row) to the csv file
    entry = [bill_nr, spons, party]
    with open('output.csv', 'a') as out_file:
      out = csv.writer(out_file)
      out.writerow(entry)

请注意，仅在 Python >3.6 中支持使用 f 字符串（在主循环的开头）。

Answer 2

更好的方法是遍历不同的元素，例如<li> 然后在里面找到需要的元素。

要获得共同赞助商，您首先需要通过检查数量来测试是否有。如果这不是0，则首先获取指向子页面的链接。使用单独的 BeautifulSoup 对象请求此子页面。然后可以解析包含共同发起人的表，并将所有共同发起人添加到列表中。如果需要，您可以在此处添加额外的处理。然后将该列表组合成一个字符串，以便将其保存到 CSV 文件的单个列中。

from bs4 import BeautifulSoup
import csv
import requests
import string

headers = None

with open('115congress.csv', 'w', newline='') as f:
    fwriter = csv.writer(f, delimiter=';')
    fwriter.writerow(['SPONS', 'PARTY', 'NBILL', 'TITLE', 'COSPONSORS'])

    for j in range(1, 3):  #114):
        print(f'Getting page {j}')

        hrurl = 'https://www.congress.gov/search?q=%7B%22source%22%3A%22legislation%22%2C%22congress%22%3A%22115%22%2C%22type%22%3A%22bills%22%7D&page='+str(j)
        hrpage = requests.get(hrurl, headers=headers)
        soup = BeautifulSoup(hrpage.content, 'lxml')

        for li in soup.find_all('li', class_='expanded'):
            bill_or_law = li.span.text
            sponsor = li.find('span', class_='result-item').a.text
            title = li.find('span', class_='result-title').text
            nbill = li.find('a').text.strip(string.ascii_uppercase + ' .')

            if '[R' in sponsor:
                party = 'Republican'
            elif '[D' in sponsor:
                party = 'Democratic'
            else:
                party = 'Unknown'

            # Any cosponsors?
            cosponsor_link = li.find_all('a')[2]

            if cosponsor_link.text == '0':
                cosponsors = "No cosponsors"
            else:
                print(f'Getting cosponsors for {sponsor}')
                # Get the subpage containing the cosponsors
                hr_cosponsors = requests.get(cosponsor_link['href'], headers=headers)
                soup_cosponsors = BeautifulSoup(hr_cosponsors.content, 'lxml')
                table = soup_cosponsors.find('table', class_="item_table")

                # Create a list of the cosponsors
                cosponsor_list = []

                for tr in table.tbody.find_all('tr'):
                    cosponsor_list.append(tr.td.a.text)

                # Join them together into a single string
                cosponsors = ' - '.join(cosponsor_list)

            fwriter.writerow([sponsor, party, nbill, f'{bill_or_law} - {title}', cosponsors])

给你一个输出 CSV 文件开始：

SPONS;PARTY;NBILL;TITLE;COSPONSORS
Rep. Ellison, Keith [D-MN-5];Democratic;7401;BILL - Strengthening Refugee Resettlement Act;No cosponsors
Rep. Wild, Susan [D-PA-15];Democratic;7400;BILL - Making continuing appropriations for the Coast Guard.;No cosponsors
Rep. Scanlon, Mary Gay [D-PA-7];Democratic;7399;BILL - Inaugural Fund Integrity Act;No cosponsors
Rep. Foster, Bill [D-IL-11];Democratic;7398;BILL - SPA Act;No cosponsors
Rep. Hoyer, Steny H. [D-MD-5];Democratic;7397;BILL - To provide further additional continuing appropriations for fiscal year 2019, and for other purposes.;No cosponsors
Rep. Torres, Norma J. [D-CA-35];Democratic;7396;BILL - Border Security and Child Safety Act;Rep. Vargas, Juan [D-CA-51]* - Rep. McGovern, James P. [D-MA-2]*
Rep. Meadows, Mark [R-NC-11];Republican;7395;BILL - To direct the Secretary of Health and Human Services to allow delivery of medical supplies by unmanned aerial systems, and for other purposes.;No cosponsors
Rep. Luetkemeyer, Blaine [R-MO-3];Republican;7394;"BILL - To prohibit the Federal financial regulators from requiring compliance with the accounting standards update of the Financial Accounting Standards Board related to current expected credit loss (""CECL""), to require the Securities and Exchange Commission to take certain impacts of a proposed accounting principle into consideration before accepting the principle, and for other purposes.";Rep. Budd, Ted [R-NC-13]*
Rep. Faso, John J. [R-NY-19];Republican;7393;BILL - Medicaid Quality Care Act;No cosponsors
Rep. Babin, Brian [R-TX-36];Republican;7392;BILL - TRACED Act;No cosponsors
Rep. Arrington, Jodey C. [R-TX-19];Republican;7391;BILL - Rural Hospital Freedom and Flexibility Act of 2018;No cosponsors
Rep. Jackson Lee, Sheila [D-TX-18];Democratic;7390;BILL - Violence Against Women Extension Act of 2018;Rep. Hoyer, Steny H. [D-MD-5] - Rep. Clyburn, James E. [D-SC-6]

使用csv.writer() 时，应始终使用newline='' 参数打开文件。这样可以避免在 CSV 文件中出现双倍行距。

我建议在文本中搜索[D 或[R，因为文本的其余部分可能已经有D 或R。