遗漏的数据没有从 Hansard 中被刮掉

Question

我正在尝试从 Hansard 中抓取数据，这是英国议会大厦内所有发言的官方逐字记录。这是我要抓取的precise link：简而言之，我想抓取此页面上的每个“提及”容器以及之后的 50 页。

但我发现，当我的爬虫“完成”时，它只收集了 990 个容器的数据，而不是完整的 1010 个容器。缺少 20 个容器的数据，就好像它在跳过一页一样。当我只将页面范围设置为 (0,1) 时，它无法收集任何值。当我将它设置为 (0,2) 时，它只收集第一页的值。要求它收集 52 页的数据并没有帮助。我认为这可能是因为我没有给 URL 足够的时间来加载，所以我在爬虫的抓取中添加了一些延迟。这并没有解决任何问题。

谁能告诉我我可能遗漏了什么？我想确保我的抓取工具正在收集所有可用数据。

pages = np.arange(0, 52)

for page in pages:

     hansard_url = "https://hansard.parliament.uk/search/Contributions? searchTerm=%22civilian%20casualties%22&startDate=01%2F01%2F1988%2000%3A00%3A00&endDate=07%2F14%2F2020%2000%3A00%3A00"

     full_url = hansard_url + "&page=" + str(page) + "&partial=true"
     page = get(full_url)
     html_soup = BeautifulSoup(page.text, 'html.parser')
     mention_containers = html_soup.find_all('div', class_="result contribution")

     time.sleep(randint(2,10))


     for mention in mention_containers:

          topic = mention.div.span.text
          topics.append(topic)

          house = mention.find("img")["alt"]

          if house == "Lords Portcullis":
                 houses.append("House of Lords")
          elif house == "Commons Portcullis":
                 houses.append("House of Commons")
          else:
                 houses.append("N/A")

          name = mention.find('div', class_="secondaryTitle").text
          names.append(name)

          date = mention.find('div', class_="").text
          dates.append(date)

          time.sleep(randint(2,10))


 hansard_dataset = pd.DataFrame(
   {'Date': dates, 'House': houses, 'Speaker': names, 'Topic': topics})
  )

  print(hansard_dataset.info())
  print(hansard_dataset.isnull().sum())
  hansard_dataset.to_csv('hansard.csv', index=False, sep="#")

感谢任何帮助我解决此问题的帮助。

Answer 1

服务器在第 48 页返回空容器，因此从第 1 页到第 51 页（含）总共有 1000 个结果：

import requests
import pandas as pd
from bs4 import BeautifulSoup


url = 'https://hansard.parliament.uk/search/Contributions'

params = {
    'searchTerm':'civilian casualties',
    'startDate':'01/01/1988 00:00:00',
    'endDate':'07/14/2020 00:00:00',
    'partial':'True',
    'page':1,
}

all_data = []

for page in range(1, 52):
    params['page'] = page

    print('Page {}...'.format(page))

    soup = BeautifulSoup(requests.get(url, params=params).content, 'html.parser')
    mention_containers = soup.find_all('div', class_="result contribution")
    if not mention_containers:
        print('Empty container!')

    for mention in mention_containers:
        topic = mention.div.span.text
        house = mention.find("img")["alt"]

        if house == "Lords Portcullis":
             house = "House of Lords"
        elif house == "Commons Portcullis":
             house = "House of Commons"
        else:
             house = "N/A"

        name = mention.find('div', class_="secondaryTitle").text
        date = mention.find('div', class_="").get_text(strip=True)

        all_data.append({'Date': date, 'House': house, 'Speaker': name, 'Topic': topic})

df = pd.DataFrame(all_data)
print(df)

打印：

...

Page 41...
Page 42...
Page 43...
Page 44...
Page 45...
Page 46...
Page 47...
Page 48...
Empty container!    # <--- here is the server error
Page 49...
Page 50...
Page 51...
                 Date             House                                            Speaker                                         Topic
0        14 July 2014    House of Lords                                     Baroness Warsi                  Gaza debate in Lords Chamber
1        3 March 2016    House of Lords                                        Lord Touhig   Armed Forces Bill debate in Grand Committee
2     2 December 2015  House of Commons                                   Mr David Cameron       ISIL in Syria debate in Commons Chamber
3        3 March 2016    House of Lords                                                      Armed Forces Bill debate in Grand Committee
4       27 April 2016    House of Lords                                                        Armed Forces Bill debate in Lords Chamber
..                ...               ...                                                ...                                           ...
995      18 June 2003    House of Lords                               Lord Craig of Radley        Defence Policy debate in Lords Chamber
996  7 September 2004    House of Lords                                           Lord Rea                  Iraq debate in Lords Chamber
997  14 February 1994    House of Lords  The Parliamentary Under-Secretary of State, Mi...             Landmines debate in Lords Chamber
998   12 January 2000  House of Commons  The Minister of State, Foreign and Commonwealt...  Serbia And Kosovo debate in Westminster Hall
999  26 February 2003    House of Lords                                           Lord Rea                  Iraq debate in Lords Chamber

[1000 rows x 4 columns]