如何在使用 Beautiful Soup 和 Requests 按下按钮后获取 HTML 更改答案

【问题标题】：How to get HTML changes after pressing button with Beautiful Soup and Requests如何在使用 Beautiful Soup 和 Requests 按下按钮后获取 HTML 更改
【发布时间】：2021-12-05 10:19:11
【问题描述】：

我想在按下按钮“更多”[+] 次以加载所有游戏后获取此站点https://www.forebet.com/en/football-predictions 的 HTML。每次页面底部的更多[+] 按钮时，HTML 都会更改并显示更多足球比赛。如何获取加载所有足球比赛的页面的请求？

from bs4 import BeautifulSoup
import requests

leagues = {"EPL","UCL","Es1","De1","Fr1","Pt1","It1","UEL"}

class ForeBet:

#gets all games from the leagues on leagues returning the games on a string list
#game format is League|Date|Hour|Home Team|Away Team|Prob Home|Prob Tie| Prob Away
def get_games_and_probs(self):

    response=requests.get('https://www.forebet.com/en/football-prediction')
    soup = BeautifulSoup(response.text, 'html.parser')
    results=list()

    games = soup.findAll(class_='rcnt tr_0')+soup.findAll(class_='rcnt tr_1')

    for game in games:
        if(leagues.__contains__(game.find(class_='shortTag').text.strip())):
            game=game.find(class_='shortTag').text+"|"+\
                game.find(class_='date_bah').text.split(" ")[0]+"|"+ \
                game.find(class_='date_bah').text.split(" ")[1]+"|"+ \
                game.find(class_='homeTeam').text+"|"+\
                game.find(class_='awayTeam').text+"|"+\
                game.find(class_='fprc').findNext().text+"|"+\
                game.find(class_='fprc').findNext().findNext().text+"|"+\
                game.find(class_='fprc').findNext().findNext().findNext().text
            print(game)
            results.append(game)

    return results

【问题讨论】：

问题到底是什么？
我想在按下按钮 x 次后获取 HTML
如果您查看源代码，您会看到More [+] 按钮调用了一个JavaScript 函数。你不能用 bs4 重新创建它——你需要一个（无头）浏览器，比如 Selenium 或 PhantomJS

标签： python web-scraping beautifulsoup request

【解决方案1】：

如前所述，requests 和 beautfulsoup 用于解析数据，而不是与站点交互。为此，您需要 Selenium。

您的另一个选择是看您是否可以直接获取数据，并查看是否有参数可以发出另一个请求，就像您单击获取更多一样。这对你有用吗？

import pandas as pd
import requests

results = pd.DataFrame()
i=0
while True:
    print(i)
    url = 'https://m.forebet.com/scripts/getrs.php'
    payload = {
    'ln': 'en',
    'tp': '1x2',
    'in': '%s' %(i+11),
    'ord': '0'}
    
    jsonData = requests.get(url, params=payload).json()
    results = results.append(pd.DataFrame(jsonData[0]), sort=False).reset_index(drop=True)

    if max(results['id'].value_counts()) <=1:
        i+=1
    else:
        results = results.drop_duplicates()
        break

输出：

print(results)
          id  pr_under  ...    country         full_name
0    1473708        31  ...    England   Isthmian League
1    1473713        35  ...    England   Isthmian League
2    1473745        28  ...    England   Isthmian League
3    1473710        35  ...    England   Isthmian League
4    1473033        28  ...    England  Premier League 2
..       ...       ...  ...        ...               ...
515  1419208        47  ...  Argentina  Torneo Federal A
516  1419156        57  ...  Argentina  Torneo Federal A
517  1450589        50  ...    Armenia    Premier League
518  1450590        35  ...    Armenia    Premier League
519  1450591        52  ...    Armenia    Premier League

[518 rows x 73 columns]

【讨论】：