如何使用 Beautiful Soup 让网络爬虫遍历多页搜索结果？ [复制]

Question

我正在尝试编写一个爬虫来从以下页面获取结果：

https://www.co.dutchess.ny.us/CountyClerkDocumentSearch/Search.aspx?q=nco1%253d2%2526name1%253da&page=1

我试图获得所有结果，而不仅仅是“A”结果，但我想我可以从一个字母开始，然后遍历整个字母表。如果有人能在这部分提供帮助，那就太好了。

无论如何，我想将所有派对名称归零，即具有属性类派对名称的元素。

我有以下代码：

from urllib.request import urlopen
from bs4 import BeautifulSoup
html = urlopen("https://www.co.dutchess.ny.us/CountyClerkDocumentSearch/Search.aspx?q=nco1%253d2%2526name1%253da&page=1")
bsObj = BeautifulSoup(html)
nameList = bsObj.findAll("td", {"class":"party-name"})
for name in nameList:
 print(name.get_text())

但是，这只适用于一页。结果跨越多个页面。如何为多个页面完成此操作？

此外，如果您能帮助获得所有结果，而不仅仅是 A，那就太好了。

编辑 我现在改进了我的代码，可以浏览所有搜索。但是，我仍然无法转到下一页。我曾尝试使用 page_number++，但由于页面结果数量不同，因此不知道在哪里停止。我怎么能让它在最后一页进入下一页？

新代码：

from urllib.request import urlopen
from bs4 import BeautifulSoup

all_letters = ["a", "b", "c", "d", "e", "f", "g", "h", "i", "j", "k", "l", "m", "n", "o","p","q","r","s","t","u","v", "w", "x", "y", "z", "0", "1", "2", "3", "4", "5", "6", "7", "8", "9"]
for letter in all_letters:

    page_number = 1
    url = "https://www.co.dutchess.ny.us/CountyClerkDocumentSearch/Search.aspx?q=nco1%253d2%2526name1%253d" + letter + "&page=" + str (page_number)
    html = urlopen(url)
    bsObj = BeautifulSoup(html)
    nameList = bsObj.findAll("td", {"class":"party-name"})

    for name in nameList:
        print(name.get_text())

Answer 1

据我了解，您想更改页面上的“starts_with”参数并遍历所有字母。如果我对问题的理解是正确的，那么这可能会有所帮助。

如果您分析 url，您将得到答案。

url = "https://www.co.dutchess.ny.us/CountyClerkDocumentSearch/Search.aspx?q=nco1%253d2%2526name1%253da&page=1"

“%253d”后面的字母表示“starts_with”术语。目前它是'a'，因此如果你想迭代，它会以'a'开头返回，只需更改url

url = 'https://www.co.dutchess.ny.us/CountyClerkDocumentSearch/Search.aspx?q=nco1%253d2%2526name1%253d' + starts_with + '&page=1'

starts_with 可以是任何字符 (a,b,c,...) 或字符串 (abc,asde,...)

Answer 2

我会这样解决（伪代码）

for letter in all_letters:
   page = 1
   while True:
       url = letter + page
       # scrape the page
       # check with bs if there is an a-element with id "NextLink1"
       if not link_to_next_page_found:
           break
       page += 1