尝试从表中抓取数据，bs4 给出 None 尽管值存在答案

【问题标题】：Trying to webscrape data from a table and bs4 gives None though value is there尝试从表中抓取数据，bs4 给出 None 尽管值存在
【发布时间】：2020-10-22 14:15:05
【问题描述】：

我正在尝试为每个玩家抓取 Fantasy Football Games Week 数据，即使选择了正确的元素，我的 beautifulsoup (playerStatTable) 也会不断返回“None”。

import requests
from bs4 import BeautifulSoup

url = 'https://fantasy.premierleague.com/statistics'
res = requests.get(url)
res.raise_for_status()#catch errors
print(res.status_code)

playerSoup = BeautifulSoup(res.text, "html.parser")

playerStatTable = playerSoup.find('table', attrs={'class': 'Table-ziussd-1 ElementDialog__HistoryTable-gmefnd-16 hraaSw'})

print(playerStatTable)

#if playerStatTable is not None:
for gw in playerStatTable.find_all("tbody"):
    rows = gw.find_all('tr')

我几乎是一个初学者，真的需要掌握使用请求和美丽汤的窍门。可以在下面看到一个屏幕截图。感谢您的帮助。

【问题讨论】：

标签： python web-scraping beautifulsoup python-requests

【解决方案1】：

您在页面上看到的数据是通过 JavaScript 动态呈现的。您可以使用selenium 提取数据或使用requests/json 直接从他们的API 加载数据。例如：

import json
import requests 


url = 'https://fantasy.premierleague.com/api/bootstrap-static/'
ajax_url = 'https://fantasy.premierleague.com/api/element-summary/{}/'

data = requests.get(url).json()

# uncomment this to print all data:
# print(json.dumps(data, indent=4))

for e in data['elements']:
    print(e['first_name'], e['second_name'])
    player_data = requests.get(ajax_url.format(e['id'])).json()
    
    # uncomment this to print all player data:
    # print(json.dumps(player_data, indent=4))    

    # print only first row from history:
    print(json.dumps(player_data['history'][0], indent=4))

    print('-' * 80)

打印：

Mesut Özil
{
    "element": 1,
    "fixture": 2,
    "opponent_team": 8,
    "total_points": 0,
    "was_home": false,
    "kickoff_time": "2020-09-12T11:30:00Z",
    "team_h_score": 0,
    "team_a_score": 3,
    "round": 1,
    "minutes": 0,
    "goals_scored": 0,
    "assists": 0,
    "clean_sheets": 0,
    "goals_conceded": 0,
    "own_goals": 0,
    "penalties_saved": 0,
    "penalties_missed": 0,
    "yellow_cards": 0,
    "red_cards": 0,
    "saves": 0,
    "bonus": 0,
    "bps": 0,
    "influence": "0.0",
    "creativity": "0.0",
    "threat": "0.0",
    "ict_index": "0.0",
    "value": 70,
    "transfers_balance": 0,
    "selected": 76656,
    "transfers_in": 0,
    "transfers_out": 0
}
--------------------------------------------------------------------------------
Sokratis Papastathopoulos
{
    "element": 2,
    "fixture": 2,
    "opponent_team": 8,
    "total_points": 0,
    "was_home": false,
    "kickoff_time": "2020-09-12T11:30:00Z",
    "team_h_score": 0,
    "team_a_score": 3,
    "round": 1,
    "minutes": 0,
    "goals_scored": 0,
    "assists": 0,
    "clean_sheets": 0,
    "goals_conceded": 0,
    "own_goals": 0,
    "penalties_saved": 0,
    "penalties_missed": 0,
    "yellow_cards": 0,
    "red_cards": 0,
    "saves": 0,
    "bonus": 0,
    "bps": 0,
    "influence": "0.0",
    "creativity": "0.0",
    "threat": "0.0",
    "ict_index": "0.0",
    "value": 50,
    "transfers_balance": 0,
    "selected": 12184,
    "transfers_in": 0,
    "transfers_out": 0
}
--------------------------------------------------------------------------------

...and so on.

【讨论】：

谢谢！这有很大帮助！我可以继续使用上面提供的代码进行抓取，我也会努力适应 selenium。感谢您的帮助。