无法使用请求从下一页解析不同案例的链接答案

【问题标题】：Unable to parse the links of different cases from next pages using requests无法使用请求从下一页解析不同案例的链接
【发布时间】：2020-07-14 07:58:14
【问题描述】：

我创建了一个脚本来解析在从网页的下拉列表中选择一个选项时显示的不同案例的链接。这是website link，这是选项Probate，应该在点击搜索按钮之前从右上角的Case Type 下拉菜单中选择。所有其他选项都应保持原样。

The script can parse the links of different cases from the first page flawlessly. However, I can't make the script go on to the next pages to collect links from there as well.

这是在底部显示下一页的方式：

和下拉应在选择选项时查看：

到目前为止我已经尝试过：

import requests
from bs4 import BeautifulSoup

link = "http://surrogateweb.co.ocean.nj.us/BluestoneWeb/Default.aspx"

with requests.Session() as s:
    s.headers['User-Agent'] = 'Mozilla/5.0 (Windows NT 6.1; ) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36'
    r = s.get(link)
    soup = BeautifulSoup(r.text,"lxml")
    payload = {i['name']:i.get('value','') for i in soup.select('input[name],select')}
    for k,v in payload.items():
        if k.endswith('ComboBox_case_type'):
            payload[k] = "Probate"
        elif k.endswith('ComboBox_case_type_VI'):
            payload[k] = "WILL"
        elif k.endswith('ComboBox_case_type$DDD$L'):
            payload[k] = "WILL"
        elif k.endswith('ComboBox_town$DDD$L'):
            payload[k] = "%"

    r = s.post(link,data=payload)
    soup = BeautifulSoup(r.text,"lxml")
    for pk_id in soup.select("a.dxeHyperlink_Youthful[href*='Q_PK_ID']"):
        print(pk_id.get("href"))

如何使用请求从下一页收集不同案例的链接？

PS 我不追求任何与硒相关的解决方案。

【问题讨论】：

看看你能不能从中得到一些想法——stackoverflow.com/a/62057796/6490744。第一个代码块是 selenium ，你不需要那个。检查beautifulsoup implementation 的第二个代码块，您也许可以解决。
如果有任何链接连接到下一页，我肯定会使用类似的逻辑。进入下一页的逻辑在这里会有所不同。谢谢@Sowjanya R Bhat。
你的目的是什么？你是想获取所有数据还是只是想抓取它们？为什么不直接将它们全部导出？
@jizhihaoSAMA 他可能想从这个 url http://surrogateweb.co.ocean.nj.us/BluestoneWeb/WebPages/web_case_detail_ocean.aspx?Q_PK_ID= 获取案例 PK id 并拉取 Web 案例详细信息。 csv 仅提供高级数据。 @robots.txt 可以的。
对不起@fcsr，你错了。那个网站的数据对我没用。然而，使用来自下一页的请求来抓取内容对我来说似乎具有挑战性，这就是我追求它的原因。鉴于我只关注链接。

标签： python python-3.x web-scraping beautifulsoup python-requests

【解决方案1】：

首先检查开发工具中的网络请求（在 Chrome 中按 F12）并监控负载。您的请求中缺少一些数据。

缺少表单数据的原因是因为它是由JavaScript添加的（当用户点击页码时）。一旦设置了表单数据，就会有 JavaScript 执行以下操作：

xmlRequest.open("POST", action, true);
xmlRequest.setRequestHeader("Content-Type", "application/x-www-form-urlencoded; charset=utf-8");
xmlRequest.send(postData);

所以您需要做的就是在您的 Python 脚本中模拟它。虽然看起来分页功能只需要两个附加值__CALLBACKID 和__CALLBACKPARAM

在下面的例子中；我已经刮掉了前 4 页（注意：第一个帖子只是登录页面）：

import requests
from bs4 import BeautifulSoup
link = "http://surrogateweb.co.ocean.nj.us/BluestoneWeb/Default.aspx"

with requests.Session() as s:
    s.headers['User-Agent'] = 'Mozilla/5.0 (Windows NT 6.1; ) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36'
    r = s.get(link)
    r.raise_for_status()
    soup = BeautifulSoup(r.text,"lxml")
    payload = {i['name']:i.get('value','') for i in soup.select('input[name],select')}
    for k,v in payload.items():
        if k.endswith('ComboBox_case_type'):
            payload[k] = "Probate"
        elif k.endswith('ComboBox_case_type_VI'):
            payload[k] = "WILL"
        elif k.endswith('ComboBox_case_type$DDD$L'):
            payload[k] = "WILL"
        elif k.endswith('ComboBox_town$DDD$L'):
            payload[k] = "%"

    page_id_list = ['PN0','PN1', 'PN2', 'PN3'] # TODO: This is proof of concept. You need to refactor code. Purhaps scrape the page id from paging html.

    for page_id in page_id_list:
        # Add 2 post items. This is required for ASP.NET Gridview AJAX postback event.          
        payload['__CALLBACKID'] = 'ctl00$ContentPlaceHolder1$ASPxGridView_search',
        # TODO: you might want to examine "__CALLBACKPARAM" acrross multiple pages. However it looks like it works by swapping PageID (e.g PN1, PN2)
        payload['__CALLBACKPARAM'] = 'c0:KV|151;["5798534","5798533","5798532","5798531","5798529","5798519","5798518","5798517","5798515","5798514","5798512","5798503","5798501","5798496","5798495"];CR|2;{};GB|20;12|PAGERONCLICK3|' + page_id + ';'
        
        r = s.post(link, data=payload)
        r.raise_for_status()
        soup = BeautifulSoup(r.text,"lxml")
        for pk_id in soup.select("a.dxeHyperlink_Youthful[href*='Q_PK_ID']"):
            print(pk_id.get("href"))

输出：

WebPages/web_case_detail_ocean.aspx?Q_PK_ID=5798668
WebPages/web_case_detail_ocean.aspx?Q_PK_ID=5798588
WebPages/web_case_detail_ocean.aspx?Q_PK_ID=5798584
WebPages/web_case_detail_ocean.aspx?Q_PK_ID=5798573
WebPages/web_case_detail_ocean.aspx?Q_PK_ID=5798572
WebPages/web_case_detail_ocean.aspx?Q_PK_ID=5798570
WebPages/web_case_detail_ocean.aspx?Q_PK_ID=5798569
WebPages/web_case_detail_ocean.aspx?Q_PK_ID=5798568
WebPages/web_case_detail_ocean.aspx?Q_PK_ID=5798566
WebPages/web_case_detail_ocean.aspx?Q_PK_ID=5798564
WebPages/web_case_detail_ocean.aspx?Q_PK_ID=5798560
WebPages/web_case_detail_ocean.aspx?Q_PK_ID=5798552
WebPages/web_case_detail_ocean.aspx?Q_PK_ID=5798542
WebPages/web_case_detail_ocean.aspx?Q_PK_ID=5798541
WebPages/web_case_detail_ocean.aspx?Q_PK_ID=5798535
WebPages/web_case_detail_ocean.aspx?Q_PK_ID=5798534
WebPages/web_case_detail_ocean.aspx?Q_PK_ID=5798533
WebPages/web_case_detail_ocean.aspx?Q_PK_ID=5798532
WebPages/web_case_detail_ocean.aspx?Q_PK_ID=5798531
WebPages/web_case_detail_ocean.aspx?Q_PK_ID=5798529
WebPages/web_case_detail_ocean.aspx?Q_PK_ID=5798519
WebPages/web_case_detail_ocean.aspx?Q_PK_ID=5798518
WebPages/web_case_detail_ocean.aspx?Q_PK_ID=5798517
WebPages/web_case_detail_ocean.aspx?Q_PK_ID=5798515
WebPages/web_case_detail_ocean.aspx?Q_PK_ID=5798514
WebPages/web_case_detail_ocean.aspx?Q_PK_ID=5798512
WebPages/web_case_detail_ocean.aspx?Q_PK_ID=5798503
WebPages/web_case_detail_ocean.aspx?Q_PK_ID=5798501
WebPages/web_case_detail_ocean.aspx?Q_PK_ID=5798496
WebPages/web_case_detail_ocean.aspx?Q_PK_ID=5798495
WebPages/web_case_detail_ocean.aspx?Q_PK_ID=5798494
WebPages/web_case_detail_ocean.aspx?Q_PK_ID=5798492
WebPages/web_case_detail_ocean.aspx?Q_PK_ID=5798485
WebPages/web_case_detail_ocean.aspx?Q_PK_ID=5798480
WebPages/web_case_detail_ocean.aspx?Q_PK_ID=5798479
WebPages/web_case_detail_ocean.aspx?Q_PK_ID=5798476
WebPages/web_case_detail_ocean.aspx?Q_PK_ID=5798475
WebPages/web_case_detail_ocean.aspx?Q_PK_ID=5798474
WebPages/web_case_detail_ocean.aspx?Q_PK_ID=5798472
WebPages/web_case_detail_ocean.aspx?Q_PK_ID=5798471
WebPages/web_case_detail_ocean.aspx?Q_PK_ID=5798470
WebPages/web_case_detail_ocean.aspx?Q_PK_ID=5798469
WebPages/web_case_detail_ocean.aspx?Q_PK_ID=5798466
WebPages/web_case_detail_ocean.aspx?Q_PK_ID=5798463
WebPages/web_case_detail_ocean.aspx?Q_PK_ID=5798462
WebPages/web_case_detail_ocean.aspx?Q_PK_ID=5798460
WebPages/web_case_detail_ocean.aspx?Q_PK_ID=5798459
WebPages/web_case_detail_ocean.aspx?Q_PK_ID=5798458
WebPages/web_case_detail_ocean.aspx?Q_PK_ID=5798457
WebPages/web_case_detail_ocean.aspx?Q_PK_ID=5798455
WebPages/web_case_detail_ocean.aspx?Q_PK_ID=5798454
WebPages/web_case_detail_ocean.aspx?Q_PK_ID=5798453
WebPages/web_case_detail_ocean.aspx?Q_PK_ID=5798452
WebPages/web_case_detail_ocean.aspx?Q_PK_ID=5798449
WebPages/web_case_detail_ocean.aspx?Q_PK_ID=5798448
WebPages/web_case_detail_ocean.aspx?Q_PK_ID=5798447
WebPages/web_case_detail_ocean.aspx?Q_PK_ID=5798446
WebPages/web_case_detail_ocean.aspx?Q_PK_ID=5798445
WebPages/web_case_detail_ocean.aspx?Q_PK_ID=5798444
WebPages/web_case_detail_ocean.aspx?Q_PK_ID=5798443

虽然可以使用 Requests 来实现解决方案。它可以是喜怒无常的。 Selenium 通常是更好的方法。

【讨论】：

一个小问题：你是怎么发现__CALLBACKPARAM 中的PBN 应该替换为page_id_list 中的ID？谢谢。
如果您检查分页按钮，他们有一个 JavaSript 单击调用，例如：onclick="ASPx.GVPagerOnClick('ContentPlaceHolder1_ASPxGridView_search','PN2');"。之后，我检查了有效负载以查看是否发送了以下内容：ContentPlaceHolder1_ASPxGridView_search 和 PN2
这是关键指针。我总是尝试使用保持不变的右箭头键ASPx.GVPagerOnClick('ContentPlaceHolder1_ASPxGridView_search','PBN');，无论我尝试转到哪个页面。谢谢。

【解决方案2】：

此代码有效，但使用 selenium 而不是请求。

您需要安装selenium python lib 并下载gecko driver。如果您不想在 c:/program 中使用 geckodriver，则必须将 executable_path= 更改为您拥有 geckodriver 的路径。您可能希望缩短睡眠时间，但网站正在加载太慢了（对我来说），所以我必须设置较长的睡眠时间，以便网站在尝试读取之前正确加载。

from selenium import  webdriver
from bs4 import BeautifulSoup
import time

link = "http://surrogateweb.co.ocean.nj.us/BluestoneWeb/Default.aspx"
driver = webdriver.Firefox(executable_path='c:/program/geckodriver.exe')
driver.get(link)
dropdown = driver.find_element_by_css_selector('#ContentPlaceHolder1_ASPxSplitter1_ASPxComboBox_case_type_B-1')
dropdown.click()
time.sleep(0.5)
cases = driver.find_elements_by_css_selector('.dxeListBoxItem_Youthful')
for case in cases:
    if case.text == 'Probate':
        time.sleep(5)
        case.click()
        time.sleep(5)
search = driver.find_element_by_css_selector('#ContentPlaceHolder1_ASPxSplitter1_ASPxButton_search')
search.click()
while True:
    time.sleep(15)
    soup = BeautifulSoup(driver.page_source,"lxml")
    for pk_id in soup.select("a.dxeHyperlink_Youthful[href*='Q_PK_ID']"):
        print(pk_id.get("href"))
    next = driver.find_elements_by_css_selector('.dxWeb_pNext_Youthful')
    if len(next) > 0:
        next[0].click()
    else:
        break

【讨论】：

我可以看到我投了反对票。但据我所知，这是唯一可行的解决方案..
但是 OP 已经宣布他不期待 selenium 的解决方案。不过，我并没有否决您的帖子。
所以很容易处理，你可以直接删除它，或者用requests发布答案。只是我的建议。
但是 requests 库做不到，所以很难做一个可以帮助别人的版本
请不要大肆指责 OP 否决了您的帖子。我知道您必须花费大量时间来提出这个解决方案，所以如果我这样做是不公平的。我添加了PS 部分以确保您或任何其他访问者了解我不追求任何面向硒的解决方案@UWTD TV。谢谢。

【解决方案3】：

以下是使用 PBN 对所有结果进行分页的方法。传递回调状态需要做的关键事情。

import html

import requests
import lxml.html
import demjson
import html


def paginate(url, callback_id):
        
    response = requests.get(url)

    tree = lxml.html.fromstring(response.text)

    yield tree

    # The first page of results is embedded in the full html
    # page. Subsequent pages of results will be extract from
    # partial html returned from an endpoint intended for AJAX

    # Set up the pagination payload with it's constant values
    payload = {}
    payload['__EVENTARGUMENT'] = None
    payload['__EVENTTARGET'] = None
    payload['__VIEWSTATE'], = tree.xpath(
        "//input[@name='__VIEWSTATE']/@value")
    payload['__VIEWSTATEGENERATOR'], = tree.xpath(
        "//input[@name='__VIEWSTATEGENERATOR']/@value")
    payload['__EVENTVALIDATION'], = tree.xpath(
        "//input[@name='__EVENTVALIDATION']/@value")
    payload['__CALLBACKID'] = callback_id

    # To get the next page of results from the AJAX endpoint,
    # it's basically a post request with a 'PBN' argument. But,
    # we also have to pass around the callback state that 
    # the endpoint expects
    event_callback_source, = tree.xpath('''//script[contains(text(), "var dxo = new ASPxClientGridView('{}');")]/text()'''.format(callback_id.replace('$', '_')))
        
    callback_state = demjson.decode(re.search(r'^dxo\.stateObject = \((?P<body>.*)\);$', event_callback_source, re.MULTILINE).group('body'))

    # You may wonder why we are encoding the callback_state back to a string
    # right after we decoded it from a string.
    #
    # The reasons is that the original string uses single quotes and is
    # not html-escaped, and we need to use double quotes and html escape.
    payload[callback_id] = html.escape(demjson.encode(callback_state))

    item_keys = callback_state['keys']
    payload['__CALLBACKPARAM'] = 'c0:KV|61;{};GB|20;12|PAGERONCLICK3|PBN;'.format(demjson.encode(item_keys))

    # We'll break when we attempt to paginate to a next
    # page but we get the same keys
    previous_item_keys = None
        
    while item_keys != previous_item_keys:

        response = requests.post(url, payload)
        previous_item_keys = item_keys

        data_str = re.match(r'.*?/\*DX\*/\((?P<body>.*)\)', response.text)\
                     .group('body')

        data = demjson.decode(data_str)

        table_tree = lxml.html.fromstring(data['result']['html'])

        yield table_tree

        callback_state = data['result']['stateObject']

        payload[callback_id] = html.escape(demjson.encode(callback_state))

        item_keys = callback_state['keys']
        payload['__CALLBACKPARAM'] = 'c0:KV|61;{};GB|20;12|PAGERONCLICK3|PBN;'.format(demjson.encode(item_keys))



if __name__ == '__main__':
    url = "http://surrogateweb.co.ocean.nj.us/BluestoneWeb/Default.aspx"
    callback_id = 'ctl00$ContentPlaceHolder1$ASPxGridView_search'
    results = paginate(url, callback_id)

【讨论】：