如何用 BeautifulSoup 刮掉“影子内容”答案

【问题标题】：How to scrape "Shadow Content" with BeautifulSoup如何用 BeautifulSoup 刮掉“影子内容”
【发布时间】：2021-08-14 00:41:55
【问题描述】：

我一直在尝试使用 BeautifulSoup 收集玩家的 Madden 评分。我想从this player 获取统计信息。但是，当我检查 HTML 时，我看到评级在“影子内容”部分，我似乎无法抓取。

【问题讨论】：

标签： python web-scraping beautifulsoup

【解决方案1】：

您正在处理JavaScript 网站，该网站在页面完全加载后呈现内部 HTML。 requests 将无法为您渲染 JS。

在这种情况下，您必须查找提供内部HTML 的API，在您的情况下如下：

import requests
from pprint import pp


def main(url):
    params = {
        'filter': 'iteration:* AND primaryKey:9925',
        'sort': 'iteration:Asc'
    }
    r = requests.get(url, params=params)
    pp(r.json()['docs'][0])


main('https://ratings-api.ea.com/v2/entities/m22-ratings')

输出：

{'college': 'Rutgers',
 'awareness_rating': 94,
 'throwPower_rating': 30,
 'kickReturn_rating': 74,
 'leadBlock_rating': 22,
 'strength_rating': 62,
 'bCVision_rating': 65,
 'catchInTraffic_rating': 45,
 'playAction_rating': 6,
 'pursuit_rating': 85,
 'plyrAssetname': 'McCourtyDevin_9925',
 'mediumRouteRunning_rating': 13,
 'catching_rating': 80,
 'acceleration_rating': 91,
 'spinMove_rating': 75,
 'height': 70,
 'finesseMoves_rating': 45,
 'spectacularCatch_rating': 61,
 'runBlock_rating': 35,
 'tackle_rating': 74,
 'injury_rating': 95,
 'zoneCoverage_rating': 92,
 'weight': 195,
 'plyrBirthdate': '8/13/87',
 'runningStyle_rating': 'Short Stride Loose',
 'deepRouteRunning_rating': 10,
 'firstName': 'Devin',
 'yearsPro': 11,
 'totalSalary': 10000000,
 'trucking_rating': 28,
 'throwAccuracyShort_rating': 6,
 'position': 'FS',
 'jukeMove_rating': 77,
 'playRecognition_rating': 90,
 'shortRouteRunning_rating': 15,
 'status': 'published',
 'lastName': 'McCourty',
 'jerseyNum': 32,
 'breakSack_rating': 14,
 'speed_rating': 91,
 'runBlockPower_rating': 35,
 'jumping_rating': 92,
 'toughness_rating': 77,
 'throwOnTheRun_rating': 6,
 'manCoverage_rating': 85,
 'stiffArm_rating': 25,
 'powerMoves_rating': 35,
 'iteration': 'launch-ratings',
 'release_rating': 12,
 'hitPower_rating': 67,
 'throwAccuracyMid_rating': 6,
 'kickAccuracy_rating': 24,
 'passBlockPower_rating': 35,
 'impactBlocking_rating': 67,
 'stamina_rating': 97,
 'carrying_rating': 50,
 'breakTackle_rating': 62,
 'plyrPortrait': 2043,
 'kickPower_rating': 22,
 'plyrHandedness': 'Right',
 'throwUnderPressure_rating': 23,
 'team': 'Patriots',
 'signingBonus': 6550000,
 'passBlock_rating': 35,
 'changeOfDirection_rating': 87,
 'press_rating': 86,
 'throwAccuracyDeep_rating': 6,
 'archetype': 'S_Zone',
 'blockShedding_rating': 64,
 'runBlockFinesse_rating': 35,
 'teamId': 22,
 'agility_rating': 89,
 'fullNameForSearch': 'Devin McCourty',
 'overall_rating': 92,
 'passBlockFinesse_rating': 35,
 'age': 34,
 'primaryKey': 9925}

拿起你需要的任何东西:)。

【讨论】：