尝试从存在的 URL 抓取数据时出现 404 错误答案

【问题标题】：404 error when trying to scrape data from URL that exists尝试从存在的 URL 抓取数据时出现 404 错误
【发布时间】：2021-12-04 04:29:44
【问题描述】：

我正在尝试抓取以下页面：https://www.lacctic.com/leagues/4

我正在使用 python 3.8.5

当我尝试抓取主页 (https://www.lacctic.com/') 时，使用以下代码：

import requests
r = requests.get('https://www.lacctic.com/')
r.status_code

我得到一个 200 状态码。

但是，当我尝试抓取我想要抓取的页面时，我遇到了一些问题。代码如下：

import requests
r = requests.get('https://www.lacctic.com/leagues/4')
r.status_code

使用上面的代码，我得到一个 404 状态代码。但是，当我从浏览器打开它时，该页面存在。

我尝试为这个问题寻找解决方案，我发现我应该添加标题作为参数，作为假装是真实用户的一种方式。我尝试了以下代码：

import requests
headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'}
r = requests.get('https://www.lacctic.com/leagues/4', headers=headers)
r.status_code

我仍然收到 404 状态代码。我还尝试了其他标头变体，但不断收到 404 状态代码。

这里的网页有什么问题？他们是否以某种方式阻止了抓取，或者这是 URL 的问题？

【问题讨论】：

这里有一些更糟糕的消息：编号页面是动态生成的，您必须使用 Selenium 或类似的东西来获取它们的内容。
我看到来自 AWS API 调用的数据
虽然响应状态为404，但仍然发送页面数据（主要是Javascript）。也许他们这样做是为了混淆网络抓取工具。浏览器显然不在乎。
我没有使用硒就知道了，我在下面回答以防大家好奇

标签： python-3.x web-scraping python-requests

【解决方案1】：

如果您“检查元素”并导航到“网络”，然后刷新页面，您应该会看到标题为：?leagueids=4&nonull=true

如果你点击它，它会带你到一个这样的链接： https://c03mmwsf5i.execute-api.us-east-2.amazonaws.com/production/api_ranking/teams/?leagueids=4&nonull=true 有前 100 个结果。

{
    "count": 319,
    "next": "https://c03mmwsf5i.execute-api.us-east-2.amazonaws.com/production/api_ranking/teams/?leagueids=4&nonull=true&page=2",
    "previous": null,
    "results": [
        {
            "id": 796,
            "name": "Northern Arizona",
            "tfrrs_id": "AZ_college_m_Northern_Arizona",
            "top_5_ability_average": 812,
            "top_runner": 802,
            "fifth_runner": 825,
            "sex": "M"
        },
        {
            "id": 1340,
            "name": "BYU",
            "tfrrs_id": "UT_college_m_BYU",
            "top_5_ability_average": 814,
            "top_runner": 794,
            "fifth_runner": 829,
            "sex": "M"
        },
        {
            "id": 325,
            "name": "Oklahoma State",
            "tfrrs_id": "OK_college_m_Oklahoma_State",
            "top_5_ability_average": 815,
            "top_runner": 802,
            "fifth_runner": 821,
            "sex": "M"
        },
        {
            "id": 943,
            "name": "Notre Dame",
            "tfrrs_id": "IN_college_m_Notre_Dame_IN",
            "top_5_ability_average": 817,
            "top_runner": 804,
            "fifth_runner": 827,
            "sex": "M"
        },
        {
            "id": 320,
            "name": "Iowa State",
            "tfrrs_id": "IA_college_m_Iowa_State",
            "top_5_ability_average": 817,
            "top_runner": 797,
            "fifth_runner": 825,
            "sex": "M"
        },
        {
            "id": 1775,
            "name": "Stanford",
            "tfrrs_id": "CA_college_m_Stanford",
            "top_5_ability_average": 818,
            "top_runner": 806,
            "fifth_runner": 829,
            "sex": "M"
        },
        {
            "id": 710,
            "name": "Colorado",
            "tfrrs_id": "CO_college_m_Colorado",
            "top_5_ability_average": 819,
            "top_runner": 804,
            "fifth_runner": 825,
            "sex": "M"
        },
        {
            "id": 42,
            "name": "Arkansas",
            "tfrrs_id": "AR_college_m_Arkansas",
            "top_5_ability_average": 821,
            "top_runner": 811,
            "fifth_runner": 827,
            "sex": "M"
        },
        {
            "id": 135,
            "name": "Tulsa",
            "tfrrs_id": "OK_college_m_Tulsa",
            "top_5_ability_average": 822,
            "top_runner": 816,
            "fifth_runner": 828,
            "sex": "M"
        },
        {
            "id": 2436,
            "name": "Wisconsin",
            "tfrrs_id": "WI_college_m_Wisconsin",
            "top_5_ability_average": 825,
            "top_runner": 812,
            "fifth_runner": 833,
            "sex": "M"
        },
        {
            "id": 34,
            "name": "Washington",
            "tfrrs_id": "WA_college_m_Washington",
            "top_5_ability_average": 825,
            "top_runner": 814,
            "fifth_runner": 832,
            "sex": "M"
        },
        {
            "id": 1642,
            "name": "Oregon",
            "tfrrs_id": "OR_college_m_Oregon",
            "top_5_ability_average": 826,
            "top_runner": 805,
            "fifth_runner": 840,
            "sex": "M"
        },
        {
            "id": 946,
            "name": "Wake Forest",
            "tfrrs_id": "NC_college_m_Wake_Forest",
            "top_5_ability_average": 827,
            "top_runner": 813,
            "fifth_runner": 834,
            "sex": "M"
        },
        {
            "id": 1553,
            "name": "Gonzaga",
            "tfrrs_id": "WA_college_m_Gonzaga",
            "top_5_ability_average": 827,
            "top_runner": 816,
            "fifth_runner": 837,
            "sex": "M"
        },
        {
            "id": 752,
            "name": "Texas",
            "tfrrs_id": "TX_college_m_Texas",
            "top_5_ability_average": 827,
            "top_runner": 818,
            "fifth_runner": 838,
            "sex": "M"
        },
        {
            "id": 1084,
            "name": "Air Force",
            "tfrrs_id": "CO_college_m_Air_Force",
            "top_5_ability_average": 829,
            "top_runner": 824,
            "fifth_runner": 833,
            "sex": "M"
        },
        {
            "id": 1217,
            "name": "Portland",
            "tfrrs_id": "OR_college_m_Portland",
            "top_5_ability_average": 829,
            "top_runner": 824,
            "fifth_runner": 834,
            "sex": "M"
        },
        {
            "id": 250,
            "name": "Ole Miss",
            "tfrrs_id": "MS_college_m_Mississippi",
            "top_5_ability_average": 829,
            "top_runner": 820,
            "fifth_runner": 840,
            "sex": "M"
        },
        {
            "id": 695,
            "name": "Florida State",
            "tfrrs_id": "FL_college_m_Florida_State",
            "top_5_ability_average": 830,
            "top_runner": 801,
            "fifth_runner": 850,
            "sex": "M"
        },
        {
            "id": 1344,
            "name": "Butler",
            "tfrrs_id": "IN_college_m_Butler",
            "top_5_ability_average": 830,
            "top_runner": 816,
            "fifth_runner": 841,
            "sex": "M"
        },
        {
            "id": 775,
            "name": "Michigan",
            "tfrrs_id": "MI_college_m_Michigan",
            "top_5_ability_average": 831,
            "top_runner": 825,
            "fifth_runner": 836,
            "sex": "M"
        },
        {
            "id": 1345,
            "name": "Villanova",
            "tfrrs_id": "PA_college_m_Villanova",
            "top_5_ability_average": 831,
            "top_runner": 805,
            "fifth_runner": 846,
            "sex": "M"
        },
        {
            "id": 35,
            "name": "Syracuse",
            "tfrrs_id": "NY_college_m_Syracuse",
            "top_5_ability_average": 832,
            "top_runner": 825,
            "fifth_runner": 838,
            "sex": "M"
        },
        {
            "id": 2438,
            "name": "Princeton",
            "tfrrs_id": "NJ_college_m_Princeton",
            "top_5_ability_average": 832,
            "top_runner": 823,
            "fifth_runner": 840,
            "sex": "M"
        },
        {
            "id": 1837,
            "name": "Furman",
            "tfrrs_id": "SC_college_m_Furman",
            "top_5_ability_average": 833,
            "top_runner": 828,
            "fifth_runner": 842,
            "sex": "M"
        },
        {
            "id": 32,
            "name": "Harvard",
            "tfrrs_id": "MA_college_m_Harvard",
            "top_5_ability_average": 833,
            "top_runner": 814,
            "fifth_runner": 855,
            "sex": "M"
        },
        {
            "id": 1086,
            "name": "Boise State",
            "tfrrs_id": "ID_college_m_Boise_State",
            "top_5_ability_average": 833,
            "top_runner": 822,
            "fifth_runner": 841,
            "sex": "M"
        },
        {
            "id": 2440,
            "name": "Georgetown",
            "tfrrs_id": "DC_college_m_Georgetown_DC",
            "top_5_ability_average": 834,
            "top_runner": 827,
            "fifth_runner": 840,
            "sex": "M"
        },
        {
            "id": 949,
            "name": "North Carolina",
            "tfrrs_id": "NC_college_m_North_Carolina",
            "top_5_ability_average": 834,
            "top_runner": 819,
            "fifth_runner": 843,
            "sex": "M"
        },
        {
            "id": 739,
            "name": "Utah State",
            "tfrrs_id": "UT_college_m_Utah_State",
            "top_5_ability_average": 834,
            "top_runner": 821,
            "fifth_runner": 840,
            "sex": "M"
        },
        {
            "id": 2265,
            "name": "Southern Utah",
            "tfrrs_id": "UT_college_m_Southern_Utah",
            "top_5_ability_average": 834,
            "top_runner": 820,
            "fifth_runner": 847,
            "sex": "M"
        },
        {
            "id": 321,
            "name": "Minnesota",
            "tfrrs_id": "MN_college_m_Minnesota",
            "top_5_ability_average": 835,
            "top_runner": 818,
            "fifth_runner": 842,
            "sex": "M"
        },
        {
            "id": 2437,
            "name": "Michigan State",
            "tfrrs_id": "MI_college_m_Michigan_State",
            "top_5_ability_average": 836,
            "top_runner": 807,
            "fifth_runner": 848,
            "sex": "M"
        },
        {
            "id": 1342,
            "name": "Iona",
            "tfrrs_id": "NY_college_m_Iona",
            "top_5_ability_average": 836,
            "top_runner": 818,
            "fifth_runner": 852,
            "sex": "M"
        },
        {
            "id": 249,
            "name": "Alabama",
            "tfrrs_id": "AL_college_m_Alabama",
            "top_5_ability_average": 836,
            "top_runner": 815,
            "fifth_runner": 865,
            "sex": "M"
        },
        {
            "id": 738,
            "name": "Weber State",
            "tfrrs_id": "UT_college_m_Weber_State",
            "top_5_ability_average": 838,
            "top_runner": 815,
            "fifth_runner": 855,
            "sex": "M"
        },
        {
            "id": 43,
            "name": "Providence",
            "tfrrs_id": "RI_college_m_Providence",
            "top_5_ability_average": 840,
            "top_runner": 824,
            "fifth_runner": 852,
            "sex": "M"
        },
        {
            "id": 1347,
            "name": "Indiana",
            "tfrrs_id": "IN_college_m_Indiana_IN",
            "top_5_ability_average": 840,
            "top_runner": 829,
            "fifth_runner": 851,
            "sex": "M"
        },
        {
            "id": 1343,
            "name": "Montana State",
            "tfrrs_id": "MT_college_m_Montana_State",
            "top_5_ability_average": 840,
            "top_runner": 821,
            "fifth_runner": 864,
            "sex": "M"
        },
        {
            "id": 944,
            "name": "NC State",
            "tfrrs_id": "NC_college_m_North_Carolina_St",
            "top_5_ability_average": 841,
            "top_runner": 825,
            "fifth_runner": 848,
            "sex": "M"
        },
        {
            "id": 1554,
            "name": "Washington St.",
            "tfrrs_id": "WA_college_m_Washington_St",
            "top_5_ability_average": 842,
            "top_runner": 826,
            "fifth_runner": 853,
            "sex": "M"
        },
        {
            "id": 948,
            "name": "Duke",
            "tfrrs_id": "NC_college_m_Duke",
            "top_5_ability_average": 842,
            "top_runner": 839,
            "fifth_runner": 845,
            "sex": "M"
        },
        {
            "id": 709,
            "name": "Colorado St.",
            "tfrrs_id": "CO_college_m_Colorado_St",
            "top_5_ability_average": 842,
            "top_runner": 829,
            "fifth_runner": 846,
            "sex": "M"
        },
        {
            "id": 945,
            "name": "Virginia",
            "tfrrs_id": "VA_college_m_Virginia",
            "top_5_ability_average": 843,
            "top_runner": 828,
            "fifth_runner": 852,
            "sex": "M"
        },
        {
            "id": 638,
            "name": "Charlotte",
            "tfrrs_id": "NC_college_m_Charlotte",
            "top_5_ability_average": 844,
            "top_runner": 814,
            "fifth_runner": 864,
            "sex": "M"
        },
        {
            "id": 1348,
            "name": "Eastern Michigan",
            "tfrrs_id": "MI_college_m_Eastern_Michigan",
            "top_5_ability_average": 845,
            "top_runner": 831,
            "fifth_runner": 854,
            "sex": "M"
        },
        {
            "id": 1892,
            "name": "CBU",
            "tfrrs_id": "CA_college_m_California_Baptist",
            "top_5_ability_average": 845,
            "top_runner": 834,
            "fifth_runner": 859,
            "sex": "M"
        },
        {
            "id": 2486,
            "name": "Utah Valley",
            "tfrrs_id": "UT_college_m_Utah_Valley",
            "top_5_ability_average": 846,
            "top_runner": 837,
            "fifth_runner": 856,
            "sex": "M"
        },
        {
            "id": 1776,
            "name": "Purdue",
            "tfrrs_id": "IN_college_m_Purdue",
            "top_5_ability_average": 847,
            "top_runner": 830,
            "fifth_runner": 857,
            "sex": "M"
        },
        {
            "id": 251,
            "name": "Kentucky",
            "tfrrs_id": "KY_college_m_Kentucky",
            "top_5_ability_average": 850,
            "top_runner": 840,
            "fifth_runner": 863,
            "sex": "M"
        },
        {
            "id": 46,
            "name": "Penn",
            "tfrrs_id": "PA_college_m_Penn",
            "top_5_ability_average": 851,
            "top_runner": 831,
            "fifth_runner": 856,
            "sex": "M"
        },
        {
            "id": 1360,
            "name": "Charleston Southern",
            "tfrrs_id": "SC_college_m_Charleston_Southern",
            "top_5_ability_average": 851,
            "top_runner": 822,
            "fifth_runner": 869,
            "sex": "M"
        },
        {
            "id": 252,
            "name": "Missouri",
            "tfrrs_id": "MO_college_m_Missouri",
            "top_5_ability_average": 851,
            "top_runner": 839,
            "fifth_runner": 868,
            "sex": "M"
        },
        {
            "id": 694,
            "name": "North Florida",
            "tfrrs_id": "FL_college_m_North_Florida",
            "top_5_ability_average": 851,
            "top_runner": 835,
            "fifth_runner": 868,
            "sex": "M"
        },
        {
            "id": 639,
            "name": "Mid. Tenn. State",
            "tfrrs_id": "TN_college_m_Mid_Tenn_State",
            "top_5_ability_average": 851,
            "top_runner": 827,
            "fifth_runner": 868,
            "sex": "M"
        },
        {
            "id": 2608,
            "name": "Santa Clara",
            "tfrrs_id": "CA_college_m_Santa_Clara",
            "top_5_ability_average": 852,
            "top_runner": 835,
            "fifth_runner": 869,
            "sex": "M"
        },
        {
            "id": 194,
            "name": "Penn State",
            "tfrrs_id": "PA_college_m_Penn_State",
            "top_5_ability_average": 852,
            "top_runner": 840,
            "fifth_runner": 859,
            "sex": "M"
        },
        {
            "id": 326,
            "name": "Illinois",
            "tfrrs_id": "IL_college_m_Illinois",
            "top_5_ability_average": 852,
            "top_runner": 815,
            "fifth_runner": 874,
            "sex": "M"
        },
        {
            "id": 1916,
            "name": "UCLA",
            "tfrrs_id": "CA_college_m_UCLA",
            "top_5_ability_average": 852,
            "top_runner": 835,
            "fifth_runner": 869,
            "sex": "M"
        },
        {
            "id": 1314,
            "name": "Eastern Kentucky",
            "tfrrs_id": "KY_college_m_Eastern_Kentucky",
            "top_5_ability_average": 852,
            "top_runner": 833,
            "fifth_runner": 868,
            "sex": "M"
        },
        {
            "id": 44,
            "name": "UMass Lowell",
            "tfrrs_id": "MA_college_m_UMass_Lowell",
            "top_5_ability_average": 853,
            "top_runner": 844,
            "fifth_runner": 860,
            "sex": "M"
        },
        {
            "id": 2303,
            "name": "Columbia",
            "tfrrs_id": "NY_college_m_Columbia",
            "top_5_ability_average": 853,
            "top_runner": 840,
            "fifth_runner": 864,
            "sex": "M"
        },
        {
            "id": 134,
            "name": "Cincinnati",
            "tfrrs_id": "OH_college_m_Cincinnati",
            "top_5_ability_average": 854,
            "top_runner": 828,
            "fifth_runner": 866,
            "sex": "M"
        },
        {
            "id": 708,
            "name": "Wyoming",
            "tfrrs_id": "WY_college_m_Wyoming",
            "top_5_ability_average": 855,
            "top_runner": 846,
            "fifth_runner": 862,
            "sex": "M"
        },
        {
            "id": 691,
            "name": "Lipscomb",
            "tfrrs_id": "TN_college_m_Lipscomb",
            "top_5_ability_average": 855,
            "top_runner": 843,
            "fifth_runner": 863,
            "sex": "M"
        },
        {
            "id": 947,
            "name": "Virginia Tech",
            "tfrrs_id": "VA_college_m_Virginia_Tech",
            "top_5_ability_average": 855,
            "top_runner": 830,
            "fifth_runner": 867,
            "sex": "M"
        },
        {
            "id": 1203,
            "name": "Liberty",
            "tfrrs_id": "VA_college_m_Liberty",
            "top_5_ability_average": 856,
            "top_runner": 839,
            "fifth_runner": 866,
            "sex": "M"
        },
        {
            "id": 213,
            "name": "Drake",
            "tfrrs_id": "IA_college_m_Drake",
            "top_5_ability_average": 856,
            "top_runner": 840,
            "fifth_runner": 878,
            "sex": "M"
        },
        {
            "id": 334,
            "name": "Eastern Illinois",
            "tfrrs_id": "IL_college_m_Eastern_Illinois",
            "top_5_ability_average": 857,
            "top_runner": 836,
            "fifth_runner": 875,
            "sex": "M"
        },
        {
            "id": 36,
            "name": "Stony Brook",
            "tfrrs_id": "NY_college_m_Stony_Brook",
            "top_5_ability_average": 857,
            "top_runner": 849,
            "fifth_runner": 867,
            "sex": "M"
        },
        {
            "id": 951,
            "name": "Louisville",
            "tfrrs_id": "KY_college_m_Louisville",
            "top_5_ability_average": 857,
            "top_runner": 843,
            "fifth_runner": 883,
            "sex": "M"
        },
        {
            "id": 38,
            "name": "Army West Point",
            "tfrrs_id": "NY_college_m_Army_West_Point",
            "top_5_ability_average": 858,
            "top_runner": 845,
            "fifth_runner": 869,
            "sex": "M"
        },
        {
            "id": 1349,
            "name": "Cornell",
            "tfrrs_id": "NY_college_m_Cornell",
            "top_5_ability_average": 858,
            "top_runner": 842,
            "fifth_runner": 865,
            "sex": "M"
        },
        {
            "id": 327,
            "name": "South Dakota St.",
            "tfrrs_id": "SD_college_m_South_Dakota_St",
            "top_5_ability_average": 858,
            "top_runner": 854,
            "fifth_runner": 864,
            "sex": "M"
        },
        {
            "id": 952,
            "name": "Pittsburgh",
            "tfrrs_id": "PA_college_m_Pittsburgh",
            "top_5_ability_average": 858,
            "top_runner": 845,
            "fifth_runner": 868,
            "sex": "M"
        },
        {
            "id": 1346,
            "name": "Yale",
            "tfrrs_id": "CT_college_m_Yale",
            "top_5_ability_average": 858,
            "top_runner": 835,
            "fifth_runner": 870,
            "sex": "M"
        },
        {
            "id": 2439,
            "name": "Cal Poly",
            "tfrrs_id": "CA_college_m_Cal_Poly",
            "top_5_ability_average": 858,
            "top_runner": 842,
            "fifth_runner": 867,
            "sex": "M"
        },
        {
            "id": 136,
            "name": "Wichita State",
            "tfrrs_id": "KS_college_m_Wichita_State",
            "top_5_ability_average": 858,
            "top_runner": 854,
            "fifth_runner": 862,
            "sex": "M"
        },
        {
            "id": 1294,
            "name": "Toledo",
            "tfrrs_id": "OH_college_m_Toledo",
            "top_5_ability_average": 859,
            "top_runner": 842,
            "fifth_runner": 876,
            "sex": "M"
        },
        {
            "id": 33,
            "name": "Navy",
            "tfrrs_id": "MD_college_m_Navy",
            "top_5_ability_average": 859,
            "top_runner": 853,
            "fifth_runner": 866,
            "sex": "M"
        },
        {
            "id": 1219,
            "name": "Portland State",
            "tfrrs_id": "OR_college_m_Portland_State",
            "top_5_ability_average": 859,
            "top_runner": 844,
            "fifth_runner": 880,
            "sex": "M"
        },
        {
            "id": 258,
            "name": "Tennessee",
            "tfrrs_id": "TN_college_m_Tennessee",
            "top_5_ability_average": 860,
            "top_runner": 834,
            "fifth_runner": 875,
            "sex": "M"
        },
        {
            "id": 1351,
            "name": "Miami (Ohio)",
            "tfrrs_id": "OH_college_m_Miami_OH",
            "top_5_ability_average": 860,
            "top_runner": 849,
            "fifth_runner": 866,
            "sex": "M"
        },
        {
            "id": 1308,
            "name": "IUPUI",
            "tfrrs_id": "IN_college_m_IUPUI",
            "top_5_ability_average": 861,
            "top_runner": 853,
            "fifth_runner": 866,
            "sex": "M"
        },
        {
            "id": 1268,
            "name": "Arkansas State",
            "tfrrs_id": "AR_college_m_Arkansas_State",
            "top_5_ability_average": 861,
            "top_runner": 828,
            "fifth_runner": 879,
            "sex": "M"
        },
        {
            "id": 1085,
            "name": "New Mexico",
            "tfrrs_id": "NM_college_m_New_Mexico",
            "top_5_ability_average": 861,
            "top_runner": 821,
            "fifth_runner": 876,
            "sex": "M"
        },
        {
            "id": 980,
            "name": "La Salle",
            "tfrrs_id": "PA_college_m_La_Salle",
            "top_5_ability_average": 862,
            "top_runner": 852,
            "fifth_runner": 871,
            "sex": "M"
        },
        {
            "id": 328,
            "name": "Bradley",
            "tfrrs_id": "IL_college_m_Bradley",
            "top_5_ability_average": 862,
            "top_runner": 851,
            "fifth_runner": 871,
            "sex": "M"
        },
        {
            "id": 1917,
            "name": "UC Santa Barbara",
            "tfrrs_id": "CA_college_m_UC_Santa_Barbara",
            "top_5_ability_average": 864,
            "top_runner": 841,
            "fifth_runner": 878,
            "sex": "M"
        },
        {
            "id": 329,
            "name": "Kansas State",
            "tfrrs_id": "KS_college_m_Kansas_State",
            "top_5_ability_average": 864,
            "top_runner": 858,
            "fifth_runner": 871,
            "sex": "M"
        },
        {
            "id": 640,
            "name": "UTEP",
            "tfrrs_id": "TX_college_m_UTEP",
            "top_5_ability_average": 864,
            "top_runner": 852,
            "fifth_runner": 876,
            "sex": "M"
        },
        {
            "id": 323,
            "name": "Nebraska",
            "tfrrs_id": "NE_college_m_Nebraska",
            "top_5_ability_average": 864,
            "top_runner": 845,
            "fifth_runner": 895,
            "sex": "M"
        },
        {
            "id": 139,
            "name": "Temple",
            "tfrrs_id": "PA_college_m_Temple",
            "top_5_ability_average": 865,
            "top_runner": 838,
            "fifth_runner": 874,
            "sex": "M"
        },
        {
            "id": 37,
            "name": "Dartmouth",
            "tfrrs_id": "NH_college_m_Dartmouth",
            "top_5_ability_average": 865,
            "top_runner": 848,
            "fifth_runner": 875,
            "sex": "M"
        },
        {
            "id": 254,
            "name": "Texas A&M",
            "tfrrs_id": "TX_college_m_Texas_AM",
            "top_5_ability_average": 866,
            "top_runner": 838,
            "fifth_runner": 881,
            "sex": "M"
        },
        {
            "id": 1886,
            "name": "Cal St. Fullerton",
            "tfrrs_id": "CA_college_m_Cal_St_Fullerton",
            "top_5_ability_average": 866,
            "top_runner": 840,
            "fifth_runner": 886,
            "sex": "M"
        },
        {
            "id": 2014,
            "name": "Appalachian State",
            "tfrrs_id": "NC_college_m_Appalachian_State",
            "top_5_ability_average": 866,
            "top_runner": 860,
            "fifth_runner": 875,
            "sex": "M"
        },
        {
            "id": 1805,
            "name": "Grand Canyon",
            "tfrrs_id": "AZ_college_m_Grand_Canyon",
            "top_5_ability_average": 867,
            "top_runner": 861,
            "fifth_runner": 873,
            "sex": "M"
        },
        {
            "id": 690,
            "name": "East Tenn. St.",
            "tfrrs_id": "TN_college_m_East_Tenn_St",
            "top_5_ability_average": 867,
            "top_runner": 858,
            "fifth_runner": 876,
            "sex": "M"
        },
        {
            "id": 255,
            "name": "Florida",
            "tfrrs_id": "FL_college_m_Florida",
            "top_5_ability_average": 868,
            "top_runner": 863,
            "fifth_runner": 871,
            "sex": "M"
        }
    ]
}

您现在应该可以运行以下代码以 JSON 形式从表中获取数据（由于表分页为每页 100 个，并且有 319 个对象，您需要对所有四个执行此操作页）：

import requests
teams1to100 = requests.get('https://c03mmwsf5i.execute-api.us-east-2.amazonaws.com/production/api_ranking/teams/?leagueids=4&nonull=true&page=1').json()
teams101to200 = requests.get('https://c03mmwsf5i.execute-api.us-east-2.amazonaws.com/production/api_ranking/teams/?leagueids=4&nonull=true&page=2').json()
teams201to300 = requests.get('https://c03mmwsf5i.execute-api.us-east-2.amazonaws.com/production/api_ranking/teams/?leagueids=4&nonull=true&page=3').json()
teams301to319 = requests.get('https://c03mmwsf5i.execute-api.us-east-2.amazonaws.com/production/api_ranking/teams/?leagueids=4&nonull=true&page=4').json()

例如，如果您想获取团队列表（在 JSON 的“结果”部分中标题为“名称”），您可以运行以下代码：

teams = []
for i in range(len(teams1to100['results'])):
    teams.append(teams1to100['results'][i]['name'])
for i in range(len(teams101to200['results'])):
    teams.append(teams101to200['results'][i]['name'])
for i in range(len(teams201to300['results'])):
    teams.append(teams201to300['results'][i]['name'])
for i in range(len(teams301to319['results'])):
    teams.append(teams301to319['results'][i]['name'])

编者注：这是用户在回答他们自己的问题！

【讨论】：