Beautifulsoup4 find_all 没有得到我需要的结果

Question

我正在尝试从 flashscore.com 获取数据到我正在做的项目中，这是我自学 Python 研究的一部分：

import requests
from bs4 import BeautifulSoup

res = requests.get("https://www.flashscore.com/")
soup = BeautifulSoup(res.text, "lxml")
games = soup.find_all("div", {'class':['event__match', 'event__match--scheduled', 'event__match--twoLine']})
print(games)

当我运行它时，它会得到一个空列表[]

为什么？

Answer 1

失败是由于网站使用了一套 Ajax 技术，特别是借助 JavaScript 客户端脚本语言动态添加的内容。脚本语言的客户端代码在浏览器本身中执行，而不是在 Web 服务器级别。此类代码的成功取决于浏览器正确解释和执行它的能力。借助您编写的程序中的 BeatifulSoup 库，您只需检查 HTML 代码。 JavaScript 代码可以打开，例如，借助 Selenium 库：https://www.selenium.dev/。以下是我想您感兴趣的数据的完整代码：

# crawler_her_sel.py
# -*- coding: utf-8 -*-

import time

from selenium.webdriver import Firefox
from selenium.webdriver.firefox.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import pandas as pd

# Variable with the URL of the website.
my_url = "https://www.flashscore.com/"

# Preparing of the browser for the work.
options = Options()
options.add_argument("--headless")
driver = Firefox(options=options)
driver.get(my_url)

# Prepare the blank dictionary to fill in for pandas.
dictionary_of_matches = {}

# Preparation of lists with scraped data.
list_of_countries = []
list_of_leagues = []
list_of_home_teams = []
list_of_scores_for_home = []
list_of_scores_for_away = []
list_of_away_teams = []

# Wait for page to fully render
try:
    element = WebDriverWait(driver, 20).until(
        EC.presence_of_element_located((By.ID, "box-over-content-a")))
finally:
    # Determining the number of the football matches based on 
    # the attribute.
    soccer = driver.find_element(By.XPATH , "/html/body/div[5]/div/div[1]/a[1]")
    matches_soccer = soccer.get_attribute("data-sport-count")

    # Determining the number of the countries for the given football 
    # matches.
    countries = driver.find_elements(By.CLASS_NAME , "event__title--type")

    # Determination of the number that determines the number of 
    # the loop iterations.
    sum_to_iterate = int(matches_soccer) + len(countries)
    
    for ind in range(1, (sum_to_iterate+1)):
        # Scraping of the country names.
        try:
            country = driver.find_element(By.XPATH ,\
             '//div[@class="sportName soccer"]/div['+str(ind)+\
             ']/div[1]/div/span[1]').text
            list_of_countries.append(country)
        except:
            country = ""
            list_of_countries.append(country)

        # Scraping of the league names.
        try:
            league = driver.find_element(By.XPATH ,\
             '//div[@class="sportName soccer"]/div['+str(ind)+\
             ']/div[1]/div/span[2]').text
            list_of_leagues.append(league)
        except:
            league = ""
            list_of_leagues.append(league)

        # Scraping of the home team names.
        try:
            home_team = driver.find_element(By.XPATH ,\
             '//div[@class="sportName soccer"]/div['+str(ind)+']/div[3]').text
            list_of_home_teams.append(home_team)
        except:
            home_team = ""
            list_of_home_teams.append(home_team)

        # Scraping of the home team scores.
        try:
            score_for_home_team = driver.find_element(By.XPATH ,\
             '//div[@class="sportName soccer"]/div['+str(ind)+']/div[5]').text
            list_of_scores_for_home.append(score_for_home_team)
        except:
            score_for_home_team = ""
            list_of_scores_for_home.append(score_for_home_team)

        # Scraping of the away team scores.
        try: 
            score_for_away_team = driver.find_element(By.XPATH ,\
             '//div[@class="sportName soccer"]/div['+str(ind)+']/div[6]').text
            list_of_scores_for_away.append(score_for_away_team)
        except:
            score_for_away_team = ""
            list_of_scores_for_away.append(score_for_away_team)

        # Scraping of the away team names.
        try:
            away_team = driver.find_element(By.XPATH ,\
             '//div[@class="sportName soccer"]/div['+str(ind)+']/div[4]').text
            list_of_away_teams.append(away_team)
        except:
            away_team = ""
            list_of_away_teams.append(away_team)

    # Add lists with the scraped data to the dictionary in the correct 
    # order.
    dictionary_of_matches["countries"] = list_of_countries
    dictionary_of_matches["leagues"] = list_of_leagues
    dictionary_of_matches["home_teams"] = list_of_home_teams
    dictionary_of_matches["scores_for_home_teams"] = list_of_scores_for_home
    dictionary_of_matches["scores_for_away_teams"] = list_of_scores_for_away
    dictionary_of_matches["away_teams"] = list_of_away_teams

    # Creating of the frame for the data with the help of the pandas 
    # package.
    df_res = pd.DataFrame(dictionary_of_matches)

    # Saving of the properly formatted data to the csv file. The date 
    # and the time of the scraping are hidden in the file name.
    name_of_file = lambda: "flashscore{}.csv".format(time.strftime(\
        "%Y%m%d-%H.%M.%S"))
    df_res.to_csv(name_of_file(), encoding="utf-8")

    driver.quit()

脚本的结果是一个 csv 文件，当它作为数据加载到 Excel 中时，会给出下表，例如：

这里值得一提的是为您的浏览器下载必要的驱动程序：https://www.selenium.dev/documentation/webdriver/getting_started/install_drivers/。
此外，我为您提供了与从 https://www.flashscore.com/ 门户网站抓取相关的另外两个有趣脚本的链接，即：How can i scrape a football results from flashscore using python 和 Scraping stats with Selenium。
我还想在这里提出法律问题。从https://www.flashscore.com/robots.txt 网站下载的 robots.txt 文件如下所示：

显示可以抓取首页。但“一般使用条款”指出，“未经提供商事先书面授权，访问者无权复制、修改、篡改、分发、传输、显示、复制、传输、上传、下载或以其他方式使用或更改应用程序的任何内容。 ”
不幸的是，这引入了歧义，最终不清楚所有者真正想要什么。因此，我建议您不要经常使用此脚本，当然也不要用于商业目的，我会向其他访问本网站的访问者询问。我自己写这个脚本的目的是为了学习刮，我根本不打算使用它。 完成的脚本可以从我的 GitHub 下载。

Answer 2

当find_all()返回一个空列表时，表示找不到你指定的元素。

确保您尝试抓取的内容不是动态添加的，例如某些情况下的 iframe