从分页下一步按钮中提取内容答案

【问题标题】：Extracting content from pagination next button从分页下一步按钮中提取内容
【发布时间】：2023-01-22 19:54:30
【问题描述】：

这是我要抓取的网站： (https://www.jurongpoint.com.sg/store-directory/?level=&cate=Food+%26+Beverage)

下面是我试过的代码，但它重复地返回第一页和第三页:(

from bs4 import BeautifulSoup
from urllib.request import urlopen

def parse():

    base_url = 'https://www.jurongpoint.com.sg/store-directory/?level=&cate=Food+%26+Beverage'
    url="https://www.jurongpoint.com.sg/store-directory/?level=&cate=Food+%26+Beverage&page=3"


    while True:
        html = urlopen(url)
        soup = BeautifulSoup(html ,"html.parser")

    
        for link in soup.find_all('div',class_='entry-content'):

            try:
                shops=soup.find_all('div',class_="col-9")
                names=soup.find_all('tr',class_="clickable")

                for n, k in zip(names, shops):
                    name = n.find_all('td')[1].text.replace(' ','')
                    desc = k.text.replace(' ','')
                    print(name + "\n")
                    print(desc)
                
            except AttributeError as e:
                print(e)

        
            next_button = soup.find('a', href=True)

            if next_button:
                url = base_url + next_button['href']
            else:
                break

parse()

提前谢谢你:)我对美丽的汤很陌生。

【问题讨论】：

标签： python web-scraping beautifulsoup pagination request

【解决方案1】：

更具体地选择您的元素，在这里使用 css selectors 来获取 <a> ，它是带有 class="PagedList-skipToNext" 的元素的子元素：

next_button = soup.select_one('.PagedList-skipToNext a')

还要检查你选择的结果，这里不需要base_url：

url = next_button.get('href')

from bs4 import BeautifulSoup
import requests

def parse():
    url = 'https://www.jurongpoint.com.sg/store-directory/?level=&cate=Food+%26+Beverage'

    while True:
        print(url)
        soup = BeautifulSoup(requests.get(url).text)
        next_button = soup.select_one('.PagedList-skipToNext a')

        if next_button:
            url = next_button.get('href')
        else:
            break

parse()

【讨论】：