如何获取所有页面的 URL？

Question

我有一个代码可以从“oddsportal”网站收集一个页面的所有 URL：

from bs4 import BeautifulSoup
import requests

headers = {
    'User-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36'}
source = requests.get("https://www.oddsportal.com/soccer/africa/africa-cup-of-nations/results/",headers=headers)

soup = BeautifulSoup(source.text, 'html.parser')
main_div=soup.find("div",class_="main-menu2 main-menu-gray")
a_tag=main_div.find_all("a")
for i in a_tag:
    print(i['href'])

返回这些结果：

/soccer/africa/africa-cup-of-nations/results/
/soccer/africa/africa-cup-of-nations-2019/results/
/soccer/africa/africa-cup-of-nations-2017/results/
/soccer/africa/africa-cup-of-nations-2015/results/
/soccer/africa/africa-cup-of-nations-2013/results/
/soccer/africa/africa-cup-of-nations-2012/results/
/soccer/africa/africa-cup-of-nations-2010/results/
/soccer/africa/africa-cup-of-nations-2008/results/

我希望将网址返回为：

https://www.oddsportal.com/soccer/africa/africa-cup-of-nations/results/
https://www.oddsportal.com/soccer/africa/africa-cup-of-nations/results/#/page/2/
https://www.oddsportal.com/soccer/africa/africa-cup-of-nations/results/#/page/3/

对于为results 生成的所有父 url。

我可以看到可以从检查元素中看到 URL 可以附加如下div id = "pagination"

Answer 1

id="pagination"下的数据是动态加载的，所以requests不支持。

但是，您可以通过向以下地址发送GET 请求来获取所有这些页面 (1-3) 的表格：

https://fb.oddsportal.com/ajax-sport-country-tournament-archive/1/MN8PaiBs/X0/1/0/{page}/?_={timestampe}"

其中{page}对应页码（1-3），{timestampe}是当前时间

您还需要添加：

"Referer": "https://www.oddsportal.com/"

到您的headers。 另外，使用lxml 解析器而不是html.parser 来避免RecursionError。

import re
import requests
from datetime import datetime
from bs4 import BeautifulSoup

headers = {
    "User-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36",
    "Referer": "https://www.oddsportal.com/",
}


with requests.Session() as session:
    session.headers.update(headers)
    for page in range(1, 4):
        response = session.get(
            f"https://fb.oddsportal.com/ajax-sport-country-tournament-archive/1/MN8PaiBs/X0/1/0/{page}/?_={datetime.now().timestamp()}"
        )

        table_data = re.search(r'{"html":"(.*)"}', response.text).group(1)
        soup = BeautifulSoup(table_data, "lxml")
        print(soup.prettify())