如何用 python 抓取 aspx 页面？

Question

我正在尝试抓取此网站中的地址 - https://aca.accela.com/ALAMEDA/Cap/CapHome.aspx?module=Building&TabName=Building

这是 HTML 的一部分的外观 -

<tr class="ACA_TabRow_Odd ACA_TabRow_Odd_FontSize">
        <td class="ACA_AlignLeftOrRightTop"><input id="ctl00_PlaceHolderMain_dgvPermitList_gdvPermitList_ctl02_CB_0" type="checkbox" name="ctl00$PlaceHolderMain$dgvPermitList$gdvPermitList$ctl02$CB_0" onclick="Check(this,'ctl00_PlaceHolderMain_dgvPermitList_gdvPermitList_hfSaveSelectedItems');" class="aca_gridview_checkbox" title="To select one record in current page."></td><td class="ACA_AlignLeftOrRightTop" style="width:60px;">
                    <div style="margin-right: 5px;">
                        <span id="ctl00_PlaceHolderMain_dgvPermitList_gdvPermitList_ctl02_lblUpdatedTime" css="ACA_SmLabel ACA_SmLabel_FontSize">07/10/2018</span>
                    </div>
                </td><td class="ACA_AlignLeftOrRightTop" style="width:110px;">
                    <div>
                        <strong>
                            </strong>
                        <a id="ctl00_PlaceHolderMain_dgvPermitList_gdvPermitList_ctl02_hlPermitNumber" href="/ALAMEDA/Cap/CapDetail.aspx?Module=Building&amp;TabName=Building&amp;capID1=18SPV&amp;capID2=00000&amp;capID3=00039&amp;agencyCode=ALAMEDA&amp;IsToShowInspection=">
                            <strong>
                                <span id="ctl00_PlaceHolderMain_dgvPermitList_gdvPermitList_ctl02_lblPermitNumber1">SPV18-0037</span></strong>
                        </a>



                        <input type="hidden" id="RecordId" value="18SPV-00000-00039">
                    </div>
                </td><td class="ACA_AlignLeftOrRightTop" style="width:130px;">
                    <div class="ACA_CapListStyle">
                        <span id="ctl00_PlaceHolderMain_dgvPermitList_gdvPermitList_ctl02_lblType">Solar Photovoltaic System Residential</span>
                    </div>
                </td><td class="ACA_AlignLeftOrRightTop" style="width:130px;">
                    <div class="ACA_CapListStyle">

                        <span id="ctl00_PlaceHolderMain_dgvPermitList_gdvPermitList_ctl02_lblDescription">INSTALL A 5.12 KW SOLAR SYSTEM, ROOFTOP, FLUSH MOUNT, 16 PANELS (BLDG)</span>
                    </div>
                </td><td class="ACA_AlignLeftOrRightTop" style="width:130px;">
                    <div class="ACA_CapListStyle">
                        <span id="ctl00_PlaceHolderMain_dgvPermitList_gdvPermitList_ctl02_lblAddress">1623 CLINTON AVE, ALAMEDA CA 94501</span>
                    </div>
                </td><td class="ACA_AlignLeftOrRightTop" style="width:100px;">
                    <div style="white-space: nowrap;" class="ACA_CapListStyle">
                        <div id="ctl00_PlaceHolderMain_dgvPermitList_gdvPermitList_ctl02_panelStatus">

                            <span id="ctl00_PlaceHolderMain_dgvPermitList_gdvPermitList_ctl02_lblStatus">Plan Review</span>

        </div>
                        <div id="ctl00_PlaceHolderMain_dgvPermitList_gdvPermitList_ctl02_panelbtnRenewalDetail">



        </div>
                    </div>
                </td><td class="ACA_AlignLeftOrRightTop" style="width:100px;">
                    <div style="white-space: nowrap;" class="ACA_CapListStyle">
                        <div id="ctl00_PlaceHolderMain_dgvPermitList_gdvPermitList_ctl02_Panel2">




        </div>
                        <div id="ctl00_PlaceHolderMain_dgvPermitList_gdvPermitList_ctl02_Panel3">



        </div>
                        <div id="ctl00_PlaceHolderMain_dgvPermitList_gdvPermitList_ctl02_Panel4">



        </div>
                        <div id="ctl00_PlaceHolderMain_dgvPermitList_gdvPermitList_ctl02_Panel5">



        </div>


                    </div>
                </td><td class="ACA_AlignLeftOrRightTop" style="width:110px;">
                    <div class="ACA_CapListStyle">
                        <span id="ctl00_PlaceHolderMain_dgvPermitList_gdvPermitList_ctl02_lblShortNote">INSTALL A 5.12 KW SOLAR SYSTEM, ROOFTOP, FLUSH MOU</span>
                    </div>
                </td><td class="ACA_Hide">
                    <div class="ACA_CapListStyle">
                        <span id="ctl00_PlaceHolderMain_dgvPermitList_gdvPermitList_ctl02_lblPermitAddress">1623 CLINTON AVE, ALAMEDA CA 94501</span>
                    </div>
                </td>
    </tr>

我正在尝试从 td 列中获取地址。 这是我在 python 中的代码 -

import bs4
from bs4 import BeautifulSoup
try:
    from urllib.request import urlopen
except ImportError:
    from urllib2 import urlopen

import os

from lxml import html

import requests
import csv
import sys



for page_no in range(2, 50):
    curr_page = str(page_no)
    if page_no < 9 :
        data = {
            'ctl00$ScriptManager1': 'ctl00$PlaceHolderMain$dgvPermitList$updatePanel|ctl00$PlaceHolderMain$dgvPermitList$gdvPermitList$ctl13$ctl0'+curr_page,
            'ctl00$PlaceHolderMain$generalSearchForm$ddlGSPermitType': 'Building/Solar Photovoltaic System/Residential/NA',
            'ctl00$PlaceHolderMain$generalSearchForm$txtGSStartDate': '01/01/2008',
            'ctl00$PlaceHolderMain$generalSearchForm$txtGSEndDate': '07/17/2018'
            }
    else :
        data = {
            'ctl00$ScriptManager1': 'ctl00$PlaceHolderMain$dgvPermitList$updatePanel|ctl00$PlaceHolderMain$dgvPermitList$gdvPermitList$ctl13$ctl1'+curr_page,
            'ctl00$PlaceHolderMain$generalSearchForm$ddlGSPermitType': 'Building/Solar Photovoltaic System/Residential/NA',
            'ctl00$PlaceHolderMain$generalSearchForm$txtGSStartDate': '01/01/2008',
            'ctl00$PlaceHolderMain$generalSearchForm$txtGSEndDate': '07/17/2018'

        }

    page = requests.get('https://aca.accela.com/ALAMEDA/Cap/CapHome.aspx?module=Building&TabName=Building', data = data)

    soup = BeautifulSoup(page.text, 'html.parser')


    print('Page ', page_no)
    #print(soup)

   address = soup.find_all('tbody')
   print(address)

当我尝试打印地址变量时，它只返回一个空字符串。当我尝试打印汤变量时，它返回网站的整个代码，而不是我们单击搜索时找到的数据。有人可以指导我哪里出错了吗？

Answer 1

find_all 的方法签名是find_all(name,attrs,recursive,string,limit,**kwargs)

您将名称作为span 传递，但随后尝试使用字符串作为关键字参数id，这是不正确的。你可能想要的是

id = 'ctl00_PlaceHolderMain_dgvPermitList_gdvPermitList_ctl09_lblAddress'

作为你的第二个论点。