抓取json网页

Question

我对网络抓取非常陌生，在从 nba.com 抓取一些 NBA 球员数据时遇到了一些麻烦。我首先尝试使用 bs4 抓取页面，但遇到了一个问题，经过一些研究，我认为这是由于我阅读的文章中的“XHR”造成的。我能够找到 json 格式数据的网址，但我的 python 程序似乎陷入困境，并且从不加载数据。再次，我在网络抓取方面很新，但我想看看我是否在这里偏离轨道......有什么建议吗？谢谢！ （代码如下）

import requests
import json

url = "http://stats.nba.com/stats/leaguedashplayerstats?College=&Conference=&Country=&DateFrom=&DateTo=&Division=&DraftPick=&DraftYear=&GameScope=&GameSegment=&Height=&LastNGames=0&LeagueID=00&Location=&MeasureType=Base&Month=0&OpponentTeamID=0&Outcome=&PORound=0&PaceAdjust=N&PerMode=PerGame&Period=0&PlayerExperience=&PlayerPosition=&PlusMinus=N&Rank=N&Season=2017-18&SeasonSegment=&SeasonType=Regular+Season&ShotClockRange=&StarterBench=&TeamID=0&VsConference=&VsDivision=&Weight="

resp = requests.get(url=url)
data = json.loads(resp.text)
print(data)

Answer 1

试一试。它将根据我定义的标题从该页面生成所有类别。顺便说一句，您最初的尝试没有得到响应，因为网页在您的请求中期待User-Agent，以确保请求不是来自机器人，而是来自任何真正的浏览器。但是，我伪造了它并找到了响应。

import requests

url = "http://stats.nba.com/stats/leaguedashplayerstats?College=&Conference=&Country=&DateFrom=&DateTo=&Division=&DraftPick=&DraftYear=&GameScope=&GameSegment=&Height=&LastNGames=0&LeagueID=00&Location=&MeasureType=Base&Month=0&OpponentTeamID=0&Outcome=&PORound=0&PaceAdjust=N&PerMode=PerGame&Period=0&PlayerExperience=&PlayerPosition=&PlusMinus=N&Rank=N&Season=2017-18&SeasonSegment=&SeasonType=Regular+Season&ShotClockRange=&StarterBench=&TeamID=0&VsConference=&VsDivision=&Weight="
resp = requests.get(url,headers={'User-Agent':'Mozilla/5.0'})
data = resp.json()

storage = data['resultSets']
for elem in storage:
    all_list = elem['rowSet']

    for item in all_list:
        Player_Id = item[0]
        Player_name = item[1]
        Team_Id = item[2]
        Team_abbr = item[3]
        print("Player_Id: {} Player_name: {} Team_Id: {} Team_abbr: {}".format(
            Player_Id,Player_name,Team_Id,Team_abbr))

Answer 2

刚刚意识到这是因为用户代理标头不同...一旦添加它们就可以工作