使用python抓取ajax xmlhttprequest答案

【问题标题】：Scraping ajax xmlhttprequest using python使用python抓取ajax xmlhttprequest
【发布时间】：2020-03-22 10:53:54
【问题描述】：

我想使用 xmlhttpeequest 从https://www.isc.co.uk/schools/ 这个网站刮取英国的学校名称、地址、电话、电子邮件。它返回错误 500。

import requests
headers = {"Accept": "application/json, text/plain, */*","Accept-Encoding": "gzip, deflate, br",
"Accept-Language": "en-US,en;q=0.9",
"Connection": "keep-alive",
"Content-Type": "application/json;charset=UTF-8",
"Cookie": "_ga=GA1.3.1302518161.1584461820; _hjid=5f23e8d2-23c6-4c87-9cc0-ca216b587ae1; cookie_preference=false; iscFilterStates=%7B%22locationLatitude%22%3Anull%2C%22locationLongitude%22%3Anull%2C%22distanceInMiles%22%3A0%2C%22residencyTypes%22%3A%5B%5D%2C%22genderGroup%22%3Anull%2C%22ageRange%22%3Anull%2C%22religiousAffiliation%22%3Anull%2C%22financialAssistances%22%3A%5B%5D%2C%22examinations%22%3A%5B%5D%2C%22specialNeeds%22%3Afalse%2C%22scholarshipsAndBurseries%22%3Afalse%2C%22latitudeSW%22%3A47.823214345168694%2C%22longitudeSW%22%3A-18.049563984375%2C%22latitudeNE%22%3A59.385618287793505%2C%22longitudeNE%22%3A12.953853984375021%2C%22contactCountyID%22%3A0%2C%22contactCountryID%22%3A0%2C%22londonBoroughID%22%3A0%2C%22filterByBounds%22%3Atrue%2C%22savedBounds%22%3Atrue%2C%22zoom%22%3A5%2C%22center%22%3A%7B%22lat%22%3A54.00366%2C%22lng%22%3A-2.547855%7D%7D; _gid=GA1.3.1000954634.1584850972; _gat=1; __atuvc=11%7C12%2C4%7C13; __atuvs=5e773c3c593ef6aa000; __atssc=google%3B7",
"Host": "www.isc.co.uk",
"Origin": "https://www.isc.co.uk",
"Referer": "https://www.isc.co.uk/schools/",
"Sec-Fetch-Mode": "cors",
"Sec-Fetch-Site": "same-origin",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3927.0 Safari/537.36"}

response = requests.post('https://www.isc.co.uk/Umbraco/Api/FindSchoolApi/FindSchoolListResults?skip=20&take=20', headers = headers)

response.status_code

【问题讨论】：

标签： python-3.x beautifulsoup python-requests

【解决方案1】：

网站加载了JavaScript 事件，该事件在页面加载后动态呈现其数据。

requests 库将无法即时渲染 JavaScript。所以你可以使用selenium 或requests_html。确实有很多模块可以做到这一点。

现在，我们在表格上确实有另一个选项，可以跟踪数据的呈现位置。我能够找到 XHR 请求，该请求用于从 back-end API 检索数据并将其呈现给用户端。

您可以通过打开Developer-Tools 并检查Network 和检查XHR/JS 请求来获取XHR 请求，具体取决于调用类型，例如fetch

import requests
import csv

data = {'locationLatitude': None, 'locationLongitude': None, 'distanceInMiles':
        0, 'residencyTypes': [], 'genderGroup': None, 'ageRange': None, 'religiousAffiliation': None, 'financialAssistances': [], 'examinations': [], 'specialNeeds': False, 'scholarshipsAndBurseries': False, 'latitudeSW': 47.823214345168694, 'longitudeSW': -18.049563984375, 'latitudeNE': 59.385618287793505, 'longitudeNE': 12.953853984375021, 'contactCountyID': 0, 'contactCountryID': 0, 'londonBoroughID': 0, 'filterByBounds': True, 'savedBounds': True, 'zoom': 5, 'center': {'lat': 54.00366, 'lng': -2.547855}}

r = requests.post(
    "https://www.isc.co.uk/Umbraco/Api/FindSchoolApi/FindSchoolListResults?skip=0&take=20", json=data).json()

with open("data.csv", 'w', newline="") as f:
    writer = csv.writer(f)
    writer.writerow(["Name", "Address", "Phone", "Email"])
    for item in r:
        writer.writerow(
            [item["Name"], item["FullAddress"], item["TelephoneNumber"], item["EmailAddress"]])
    print("Done")

输出：View-Online

【讨论】：