Beautifulsoup 分页使用下一个按钮答案

【问题标题】：Beautifulsoup Pagination using next buttonBeautifulsoup 分页使用下一个按钮
【发布时间】：2020-11-13 00:12:24
【问题描述】：

我正在尝试通过此链接收集有关 2020 年全球最高薪运动员收入的信息 https://www.forbes.com/profile/roger-federer/?list=athletes 这是第一页的代码

import requests
from bs4 import BeautifulSoup
import csv

page = requests.get('https://www.forbes.com/profile/roger-federer/?list=athletes')
soup = BeautifulSoup(page.text, 'html.parser')
profile = soup.find(class_ = 'profile-content')
name = soup.find(class_='profile-heading__rank').next_sibling

value = profile.find(class_ = "profile-info__item-value").get_text()

stats = profile.find_all(class_ = "profile-stats__text")
age = stats[0].get_text()
sport = stats[1].get_text()
citizenship = stats[5].get_text()

photo = profile.find(class_ = "profile-photo")
image = photo.find("img")
source=image.get("src")

print(name+" " +" "+ value+" "+ source+" "+age+" "+sport+" "+citizenship)

如何通过点击next的分页获取剩余99名运动员的详细信息

image for showing next button

【问题讨论】：

从按钮中提取超链接并下载它，“点击”按钮在beautifulsoup中不起作用。
使用 class "profile-nav__next" 的 href 加载 a 元素，这将为您提供“下一个”要获取的 URL。
@NathanChampion 代码示例
我不懂 Python。但这似乎很简单，使用 profile.find(class_ = "profile-nav__next").get("href")。将其存储在一个变量中，然后是 requests.get(NextPageVariable)。您可能需要预先添加 URI，并且需要某种结束条件。

标签： python html web-scraping beautifulsoup pagination

【解决方案1】：

如果找不到更多的运动员，这个人会跑和摔：

import requests
from bs4 import BeautifulSoup
import csv
main = 'https://www.forbes.com'
athlet = None
while True:
    if athlet:
        page = requests.get(main + athlet)
    else:
        page = requests.get('https://www.forbes.com/profile/roger-federer/?list=athletes')
    soup = BeautifulSoup(page.text, 'html.parser')
    profile = soup.find(class_ = 'profile-content')
    name = soup.find(class_='profile-heading__rank').next_sibling

    value = profile.find(class_ = "profile-info__item-value").get_text()

    stats = profile.find_all(class_ = "profile-stats__text")
    age = stats[0].get_text()
    sport = stats[1].get_text()
    citizenship = stats[5].get_text()

    photo = profile.find(class_ = "profile-photo")
    image = photo.find("img")
    source=image.get("src")

    print(name+" " +" "+ value+" "+ source+" "+age+" "+sport+" "+citizenship)
    athlet_link = soup.find('a',class_='profile-nav__next')
    if athlet_link:
        athlet = athlet_link.get('href')
    else:
        break

【讨论】：

【解决方案2】：

试试这个：

import requests
from bs4 import BeautifulSoup
import csv
main_url = 'https://www.forbes.com'
for x in range(100):
    if x == 0 :
        url = main_url + '/profile/roger-federer/?list=athletes'
    page = requests.get(url)
    soup = BeautifulSoup(page.text, 'html.parser')
    profile = soup.find(class_ = 'profile-content')
    name = soup.find(class_='profile-heading__rank').next_sibling

    value = profile.find(class_ = "profile-info__item-value").get_text()

    stats = profile.find_all(class_ = "profile-stats__text")
    age = stats[0].get_text()
    sport = stats[1].get_text()
    citizenship = stats[5].get_text()

    photo = profile.find(class_ = "profile-photo")
    image = photo.find("img")
    source=image.get("src")

    print(name+" " +" "+ value+" "+ source+" "+age+" "+sport+" "+citizenship)
    
    url = soup.find('a',class_='profile-nav__next')
    if url :
        url = main_url + url.get('href')
    else :
        break

输出：

Roger Federer  $106.3M https://thumbor.forbes.com/thumbor/fit-in/416x416/filters%3Aformat%28jpg%29/https%3A%2F%2Fspecials-images.forbesimg.com%2Fimageserve%2F5ed53e8fa40c3d0007ed25b3%2F0x0.jpg%3Fbackground%3D000000%26cropX1%3D509%26cropX2%3D1693%26cropY1%3D78%26cropY2%3D1262 38 Tennis Switzerland
Cristiano Ronaldo  $105M https://thumbor.forbes.com/thumbor/fit-in/416x416/filters%3Aformat%28jpg%29/https%3A%2F%2Fspecials-images.forbesimg.com%2Fimageserve%2F5ec593cc431fb70007482137%2F0x0.jpg%3Fbackground%3D000000%26cropX1%3D1321%26cropX2%3D3300%26cropY1%3D114%26cropY2%3D2093 35 Soccer Portugal
Lionel Messi  $104M https://thumbor.forbes.com/thumbor/fit-in/416x416/filters%3Aformat%28jpg%29/https%3A%2F%2Fspecials-images.forbesimg.com%2Fimageserve%2F5ec595d45f39760007b05c07%2F0x0.jpg%3Fbackground%3D000000%26cropX1%3D989%26cropX2%3D2480%26cropY1%3D74%26cropY2%3D1564 33 Soccer Argentina
...

【讨论】：

它可以工作，但一段时间后就坏了，那么我在哪里可以引入睡眠？
我认为这个脚本有两个主要问题 1 最后你会得到一个错误。 2 如果列表是 101 或任何其他数字，您也会遇到问题。更好的方法是在找不到更多运动员时休息。
如果你想要@PASCAL，你可以试试我的解决方案。我不认为你会从中得到错误
我的代码最后出现错误。你可以试试@PASCAL MAGONA 的回答。我已经更新了我的答案，现在它确实在最后给出了错误，并且一直工作到 100 页。