Scrapy无法访问子div类

Question

我正在使用 Scrapy 来抓取此网页 https://researchgrant.gov.sg/eservices/advanced-search/?keyword=&source=sharepoint&type=project&status=open&page=2&_pp_projectstatus=&_pp_hiname=ab&_pp_piname=pua&_pp_source=sharepoint&_pp_details=#project 中表格中的 href 链接。我可以访问 div MVCGridTableHolder_advancesearchawardedprojectsp_ 但无法访问它的子元素，即 div 类行和 div 样式，我的尝试如下所示。是不是因为偏观？

html代码：

<div id="MVCGridContainer_advancesearchawardedprojectsp_" data-key="" class="MVCGridContainer">
<!--Partial View!-->
<div class="row"></div>
<div style="overflow-x:auto;">
<table name="MVCGridTable_advancesearchawardedprojectsp" class="table table-striped table-bordered iris-grid">
<thead></thead>
<tbody>
      <tr>
         <td>
         <a class="grid-link" target="_top" href="https://researchgrant.gov.sg/pages/Awarded-Project-Detail.aspx?AXID=MOH-000080&amp;CompanyCode=moh">INVESTIGATING DIVERSIFIED BIFUNCTIONAL MACROCYCLES BY PHAGE DISPLAY AS A NOVEL TECHNOLOGY PLATFORM</a>
         </td>
</div></div>

Scrapy shell 尝试：

In [12]: quote = response.xpath('//div[@id="MVCGridTableHolder_advancesearchawardedprojectsp_"]')

In [13]: quote
Out[13]: [<Selector 
xpath='//div[@id="MVCGridTableHolder_advancesearchawardedprojectsp_"]' data='<div id="MVCGridTableHolder_advancese...'>]

In [14]: quote = response.xpath('//div[@id="MVCGridTableHolder_advancesearchawardedprojectsp_"]/div[@class="row"]')

In [15]: quote
Out[15]: []

Answer 1

如果您在加载此页面时在浏览器中打开浏览器开发人员工具，您会看到发送了一个单独的 XHR 请求来加载该部分视图内容。您可以在代码中模拟该请求。

使用requests的示例：

import requests


with requests.Session() as session:
    session.verify = False

    session.headers = {
        'X-Requested-With': 'XMLHttpRequest'
    }
    response = session.post("https://researchgrant.gov.sg/eservices/mvcgrid", params={
        'keyword': '',
        'source': 'sharepoint',
        'type': 'project',
        'status': 'open',
        'page': '2',
        '_pp_projectstatus': '',
        '_pp_hiname': 'ab',
        '_pp_piname': 'pua',
        '_pp_source': 'sharepoint',
        '_pp_details': ''},
        data={
            'name': 'advancesearchawardedprojectsp'
        })

    print(response.text)

在 Scrapy 中，您可以使用 FormRequest：

• Send Post Request in Scrapy