Beautifulsoup 并非每次都一致地抓取数据

Question

我正在尝试在此网站上抓取玩家的姓名和他们的评分： https://www.whoscored.com/Matches/1549539/LiveStatistics/England-Premier-League-2021-2022-Brentford-Arsenal.

抓取后，我将数据放入 csv 中。但是，它不会始终如一地刮擦。我可能必须多次运行脚本（2-5 次）才能让它抓取数据。当我尝试抓取其他匹配项时，也会发生这种情况。例如，如果我从 3 个匹配项中获取数据，它可能只会抓取第一个匹配项，而不会抓取其他页面的剩余数据。这是我的代码：

from bs4 import BeautifulSoup
from selenium import webdriver

match_link='https://www.whoscored.com/Matches/1549539/Live/England-Premier-League-2021-2022-Brentford-Arsenal.'

driver=webdriver.Chrome('C:\\Program Files (x86)\\chromedriver.exe')
driver.get(match_link)
soup=BeautifulSoup(driver.page_source,'html.parser')

Players_list=[]
Player_rating=[]

try:
            player_name=soup.select('a.player-link span.iconize.iconize-icon-left')
            player_rating=soup.select('td.rating')

            #print('------------getting player name and ratings-----------')
            for nme in player_name:
                #print(nme.text)
                Players_list.append(nme.text)
            for rat in player_rating:
                #print(rat.text)
                Player_rating.append(rat.text)
except:
            print('NO ELEMENT')

Players_list=pd.DataFrame(Players_list)
Player_rating=pd.DataFrame(Player_rating)

        
df=pd.concat([Players_list,Player_rating],axis=1)
df.to_csv('brentford-arsenal.csv')

它不会引发错误。它只返回一个空结果（意味着数据没有被抓取）。元素选择正确，但问题是脚本不一致。

Empty DataFrame
Columns: []
Index: []

Answer 1

您需要为要呈现的页面添加显式wait。

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

url = 'https://www.whoscored.com/Matches/1549539/LiveStatistics/England-Premier-League-2021-2022-Brentford-Arsenal'

driver = webdriver.Chrome('C:\\Program Files (x86)\\chromedriver.exe')
driver.get(url)

WebDriverWait(driver, 15).until(
        EC.presence_of_element_located((By.ID, 'player-table-statistics-body'))
    )

如果您总是想使用最新的 chrome 驱动程序，那么新的web manager 会自动检测新驱动程序何时可用、下载并缓存它。安装管理器：pip install webdriver-manager

您可以直接在 Selenium 中进行选择，因此无需使用 BeautifulSoup 重新解析 HTML。这是使用 ChromeDriverManager 的完整解决方案。

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import pandas as pd

url = 'https://www.whoscored.com/Matches/1549539/LiveStatistics/England-Premier-League-2021-2022-Brentford-Arsenal'

s = Service(ChromeDriverManager().install())
driver = webdriver.Chrome(service=s)

# web driver goes to page
driver.get(url)

WebDriverWait(driver, 15).until(
        EC.presence_of_element_located((By.ID, 'player-table-statistics-body'))
    )

Players_list = []
Player_rating = []

for nme in driver.find_elements(By.CSS_SELECTOR, 'a.player-link span.iconize.iconize-icon-left'):
    Players_list.append(nme.text)
for rat in driver.find_elements(By.CSS_SELECTOR, 'td.rating'):
    Player_rating.append(rat.text)
data = {'name': Players_list, 'rating': Player_rating}
df = pd.DataFrame(data)
print(df)
df.to_csv('brentford-arsenal.csv')

输出：

                      name rating
0               David Raya   7.60
1            Ethan Pinnock   7.57
2          Kristoffer Ajer   6.71
3           Pontus Jansson   6.92
4              Sergi Canós   8.78
5               Rico Henry   6.79
6       Christian Nørgaard   7.67
7            Vitaly Janelt   6.74
...
33            Reiss Nelson   6.04
34          Mohamed Elneny      -
35               Karl Hein      -
36           Cédric Soares      -
37         Héctor Bellerín      -
38  Ainsley Maitland-Niles      -
39             Rob Holding      -

如果页面渲染仍然返回不一致的结果，那么可以在 WebDriverWait 调用之后添加 driver.implicitly_wait(5) 或创建一个自定义 EC 条件，等待找到相同数量的玩家和排名。