抓取 NBA.com 个人球员对决头对头统计页面（涵盖多个页面）

Question

我正在尝试使用 Python 抓取以下页面（目前正在尝试使用 Requests 和 BeautifulSoup），但努力获得 a）表格格式的有意义的结果和 b）从每个页面抓取，因为大多数玩家的数据涵盖不同的页面（例如，以下播放器的数据跨越 7 页：https://www.nba.com/stats/player/203081/head-to-head/)

目前，我已经能够成功运行 GET & SOUP 请求，但我不确定继续进行的最佳方式。非常感谢任何帮助/建议/建议。

url = 'https://www.nba.com/stats/player/203081/head-to-head/'
r = requests.get(url)
if r.status_code==200:
    soup = BeautifulSoup(r.content, 'html.parser')
    print(soup)
    table = soup.find('table')
    if table:
        df = pd.read_html(str(table))[0]
        print(df)

Answer 1

我在浏览器中访问了该页面并记录了我的网络流量，并看到我的浏览器向 REST API 发出了几个 HTTP GET 请求。其中一个具有端点stats/leagueseasonmatchups，您可以查询特定的球员、联赛和赛季。响应是 JSON，其中包含您尝试抓取的所有表信息。通常，页面使用此 API 使用 JavaScript 异步填充 DOM。由于我们知道端点、查询字符串参数和请求标头，我们可以模仿 HTTP GET 请求，解析响应，并将其写入 CSV：

def get_matchups():

    import requests

    url = "https://stats.nba.com/stats/leagueseasonmatchups"

    params = {
        "DateFrom": "",
        "DateTo": "",
        "DefPlayerID": "203081",
        "LeagueID": "00",
        "Outcome": "",
        "PORound": "0",
        "PerMode": "Totals",
        "Season": "2020-21",
        "SeasonType": "Regular Season"
    }

    headers = {
        "Accept": "application/json",
        "Accept-Encoding": "gzip, deflate",
        "Referer": "https://www.nba.com/",
        "User-Agent": "Mozilla/5.0",
        "x-nba-stats-origin": "stats",
        "x-nba-stats-token": "true"
    }

    print("Getting matchups for player ID# {}...".format(params["DefPlayerID"]))

    response = requests.get(url, params=params, headers=headers)
    response.raise_for_status()

    data = response.json()
    
    fieldnames = data["resultSets"][0]["headers"]

    for row in data["resultSets"][0]["rowSet"]:
        yield dict(zip(fieldnames, row))

def main():

    from csv import DictWriter

    all_matchups = list(get_matchups())

    print("Writing to CSV file...")

    with open("output.csv", "w", newline="") as file:
        fieldnames = list(all_matchups[0]) # a bit lame
        writer = DictWriter(file, fieldnames=fieldnames)

        writer.writeheader()
        for matchup in all_matchups:
            writer.writerow(matchup)

    print("Done.")

    return 0


if __name__ == "__main__":
    import sys
    sys.exit(main())

输出（终端）：

Getting matchups for player ID# 203081...
Writing to CSV file...
Done.
>>> 

输出 (CSV)：

SEASON_ID,OFF_PLAYER_ID,OFF_PLAYER_NAME,DEF_PLAYER_ID,DEF_PLAYER_NAME,GP,MATCHUP_MIN,PARTIAL_POSS,PLAYER_PTS,TEAM_PTS,MATCHUP_AST,MATCHUP_TOV,MATCHUP_BLK,MATCHUP_FGM,MATCHUP_FGA,MATCHUP_FG_PCT,MATCHUP_FG3M,MATCHUP_FG3A,MATCHUP_FG3_PCT,HELP_BLK,HELP_FGM,HELP_FGA,HELP_FG_PERC,MATCHUP_FTM,MATCHUP_FTA,SFL
22020,202709,Cory Joseph,203081,Damian Lillard,5,17:34,68.6,4,82,1,1,0,2,10,0.2,0,3,0.0,0,0,0,0.0,0,0,0
22020,1628969,Mikal Bridges,203081,Damian Lillard,3,17:28,68.36,18,98,4,1,0,7,8,0.875,3,4,0.75,0,0,0,0.0,1,1,1
22020,1628366,Lonzo Ball,203081,Damian Lillard,3,16:34,65.98,17,77,6,2,1,6,13,0.462,5,11,0.455,0,0,0,0.0,0,0,0
22020,1626220,Royce O'Neale,203081,Damian Lillard,3,14:17,51.4,2,77,0,1,0,1,6,0.167,0,4,0.0,0,0,0,0.0,0,0,0
22020,1626196,Josh Richardson,203081,Damian Lillard,3,11:39,47.9,6,80,2,1,0,2,4,0.5,1,1,1.0,0,0,0,0.0,1,1,1
...