如何通过 Selenium 检索页面源？答案

【问题标题】：How to retrieve the page source through Selenium?如何通过 Selenium 检索页面源？
【发布时间】：2018-07-14 17:13:16
【问题描述】：

我想使用 selenium 从这个网站获取数据：

https://e-civic.coquitlam.ca/tempestprod/webinquiry/frames.cfm

当我使用时：

driver.get(url_main)

driver.page_source

这样返回：

<!-- Vers: 80000 9 --><html xmlns="http://www.w3.org/1999/xhtml"><head>\n\t<title>The City of Coquitlam Property Inquiry System</title>\n</head>\n<frameset rows="100%, *" frameborder="0" framespacing="0">\n\t<frame src="default.cfm?autorefresh=yes" name="frame_map" id="frame_map" frameborder="0" scrolling="auto" noresize="" marginwidth="0" marginheight="0" />\n\t<frame src="nothing.cfm" name="frame_action" id="frame_action" frameborder="0" scrolling="no" marginwidth="0" marginheight="0" />\n</frameset>\n</html>

不是页面的完整html，如何获取本网站的完整页面？

【问题讨论】：

标签： python selenium selenium-webdriver iframe web-crawler

【解决方案1】：

您可以使用此代码：

driver.get("https://e-civic.coquitlam.ca/tempestprod/webinquiry/frames.cfm")  
driver.switch_to.frame("frame_map")
print(driver.page_source)

请注意，您的页面内容位于 frame 中。为了与 iframe/frame/frameset 中的 element/elements 交互，您必须将 Web 驱动程序 的焦点切换到相应的框架。

希望这会有所帮助。

【讨论】：

【解决方案2】：

使用 Selenium 从网站 https://e-civic.coquitlam.ca/tempestprod/webinquiry/frames.cfm 提取数据，您需要诱导 WebDriverWait 以使预期的框架可用并切换到它 你可以使用以下解决方案：

代码块：

from selenium import webdriver
from selenium.webdriver import Chrome
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

options = Options()
options.add_argument("start-maximized")
options.add_argument("disable-infobars")
options.add_argument("--disable-extensions")
driver = webdriver.Chrome(chrome_options=options, executable_path=r'C:\WebDrivers\ChromeDriver\chromedriver_win32\chromedriver.exe')    
driver.get('https://e-civic.coquitlam.ca/tempestprod/webinquiry/frames.cfm')       
WebDriverWait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it((By.ID,"frame_map")))
print(driver.page_source)

控制台输出：

<!-- Vers: 80000 9 --><!-- Vers: 80000 9 --><!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"><html xmlns="http://www.w3.org/1999/xhtml"><head>
    <title>The City of Coquitlam Property Inquiry System</title>
</head>

<body onload="window.location.replace('default.cfm');" topmargin="5" leftmargin="5" rightmargin="5" bottommargin="0" bgcolor="white" text="black" link="#0065A4" vlink="5C5C5C" marginwidth="0" marginheight="0">

<form action="menuredirect.cfm" method="post" name="menuadminform">

<input type="Hidden" name="goto" value="admin" />
<input type="Hidden" name="tempest_wi_sessionid" value="0769A2DA-5056-AE78-7AD8DE5CC457342D" />
<input type="Hidden" name="tempest_wi_userid" value="GUEST" />
<input type="Hidden" name="tempest_wi_layerid" value="WC001-20030620175710" />

</form>

<form action="menuredirect.cfm" method="post" name="menumapform">

<input type="Hidden" name="goto" value="map" />
<input type="Hidden" name="tempest_wi_sessionid" value="0769A2DA-5056-AE78-7AD8DE5CC457342D" />
<input type="Hidden" name="tempest_wi_userid" value="GUEST" />
<input type="Hidden" name="tempest_wi_layerid" value="WC001-20030620175710" />

</form>

<form action="menuredirect.cfm" method="post" name="menuselectionlistform">

<input type="Hidden" name="goto" value="selectionlist" />
<input type="Hidden" name="tempest_wi_sessionid" value="0769A2DA-5056-AE78-7AD8DE5CC457342D" />
<input type="Hidden" name="tempest_wi_userid" value="GUEST" />
<input type="Hidden" name="tempest_wi_layerid" value="WC001-20030620175710" />

</form>

<form action="menuredirect.cfm" method="post" name="menureportsform">

<input type="Hidden" name="goto" value="reports" />
<input type="Hidden" name="tempest_wi_sessionid" value="0769A2DA-5056-AE78-7AD8DE5CC457342D" />
<input type="Hidden" name="tempest_wi_userid" value="GUEST" />
<input type="Hidden" name="tempest_wi_layerid" value="WC001-20030620175710" />

</form>

<form action="menuredirect.cfm" method="post" name="menuhomeform">

<input type="Hidden" name="goto" value="default" />
<input type="Hidden" name="tempest_wi_sessionid" value="0769A2DA-5056-AE78-7AD8DE5CC457342D" />
<input type="Hidden" name="tempest_wi_userid" value="GUEST" />
<input type="Hidden" name="tempest_wi_layerid" value="WC001-20030620175710" />

</form>

<form target="_blank" action="menuredirect.cfm" method="post" name="menudisclaimerform">

<input type="Hidden" name="goto" value="disclaimer" />
<input type="Hidden" name="tempest_wi_sessionid" value="0769A2DA-5056-AE78-7AD8DE5CC457342D" />
<input type="Hidden" name="tempest_wi_userid" value="GUEST" />
<input type="Hidden" name="tempest_wi_layerid" value="WC001-20030620175710" />

</form>

<form action="menuredirect.cfm" method="post" name="menulayerform">

<input type="Hidden" name="goto" value="layers" />
<input type="Hidden" name="tempest_wi_sessionid" value="0769A2DA-5056-AE78-7AD8DE5CC457342D" />
<input type="Hidden" name="tempest_wi_userid" value="GUEST" />
<input type="Hidden" name="tempest_wi_layerid" value="WC001-20030620175710" />

</form>

<form action="menuredirect.cfm" method="post" name="menuloginform" target="tempestwiloginwindow">

<input type="Hidden" name="goto" value="login" />
<input type="Hidden" name="tempest_wi_sessionid" value="0769A2DA-5056-AE78-7AD8DE5CC457342D" />
<input type="Hidden" name="tempest_wi_userid" value="GUEST" />
<input type="Hidden" name="tempest_wi_layerid" value="WC001-20030620175710" />

</form>

<form action="menuredirect.cfm" method="post" name="menulogoutform">

<input type="Hidden" name="goto" value="logout" />
<input type="Hidden" name="tempest_wi_sessionid" value="0769A2DA-5056-AE78-7AD8DE5CC457342D" />
<input type="Hidden" name="tempest_wi_userid" value="GUEST" />
<input type="Hidden" name="tempest_wi_layerid" value="WC001-20030620175710" />

<input type="Hidden" name="logoutmessage" value="You have been logged out." />

</form>



<font face="Arial">
<table cellpadding="0" cellspacing="0" width="100%">

    <tbody><tr><td>

<table width="100%" cellpadding="0" cellspacing="0" border="0" bordercolor="Black">
<tbody><tr>
<td align="Left">
    <table cellpadding="0" cellspacing="0" border="0" bordercolor="Black" width="100%">
        <tbody><tr>



            <td background="/tempestprod/webinquiry/inhouse/coquitlam_header_background.gif" width="100%" align="LEFT">
                <a href="http://www.coquitlam.ca" target="new">
                    <img ismap="" usemap="#map" src="inhouse/coquitlam_logo_white.png" border="0" alt="coquitlam.ca" />
                </a>
            </td>




        </tr>
    </tbody></table>
</td>
</tr>
</tbody></table>
<br /></td></tr>

<tr><td>


<table cellpadding="2" cellspacing="0">
<tbody><tr>
    <td bgcolor="F78E1E" nowrap="" width="320">
        <font style="text-transform: Capitalize; font-size:12" color="404040">

                <b>



                </b>

        </font>
    </td>

    <td bgcolor="F78E1E" nowrap="">
        <font style="font-size:12;color:5C5C5C">


            <font style="font-size:12"> <a href="#" name="loginlink" onclick="window.open('loading.cfm', 'tempestwiloginwindow', 'height=200,width=325,resizable,scrollbars,status'); document.menuloginform.submit();" onmouseover="this.style.color='blue'; this.style.textDecoration='underline';" onmouseout="this.style.color='5C5C5C'; this.style.textDecoration='none';" style="color:5C5C5C;text-decoration:none;font-size:12;"><nobr>Registered User Login</nobr></a><nobr> |</nobr></font><nobr>
        <font style="font-size:12"><a href="inhouse/defaulthelp.cfm" name="helplink" onmouseover="style.color='blue'; style.textDecoration='underline';" onmouseout="style.color='5C5C5C'; style.textDecoration='none';" style="color:5C5C5C;text-decoration:none;font-size:12;" onclick="var helpWin=window.open(this.href,'helpwindow','height=500,width=500,scrollbars');if(window.focus) {helpWin.focus()};return false;">Help</a> |
                <font style="font-size:12"><a href="http://www.coquitlam.ca/privacy-policy.aspx" style="color:5C5C5C;text-decoration:none;font-size:12;" onmouseover="this.style.color='blue'; this.style.textDecoration='underline';" onmouseout="this.style.color='5C5C5C'; this.style.textDecoration='none';" target="_blank">Privacy Policy</a> |</font>

        </font>
    </nobr></font></td>
</tr>
</tbody></table>

</td></tr>
<tr><td>

<table cellspacing="0" cellpadding="2" border="1" bordercolor="Black">
<tbody><tr bgcolor="F78E1E">


    <td bgcolor="80B2D2" nowrap="" valign="top" align="center" onmouseover="this.style.cursor='hand'" onclick="document.menuhomeform.submit();">
        <font style="font-size:12">
            <b><a href="#" name="homelink" id="homelink" onclick="document.menuhomeform.submit();" style="color:black;text-decoration:none;font-size:12;"> Welcome </a></b>
        </font>
    </td>


    <form action="menuredirect.cfm" method="post" name="menusearchform"></form>
    <input type="Hidden" name="goto" value="selectsearch" />
    <input type="Hidden" name="tempest_wi_sessionid" value="0769A2DA-5056-AE78-7AD8DE5CC457342D" />
    <input type="Hidden" name="tempest_wi_userid" value="GUEST" />
    <input type="Hidden" name="tempest_wi_layerid" value="WC001-20030620175710" />

    <td bgcolor="F78E1E" nowrap="" valign="top" align="center" onmouseover="this.style.backgroundColor='snow'; this.style.cursor='hand'; searchlink.style.color='blue'" onmouseout="this.style.backgroundColor='F78E1E'; searchlink.style.color='393939'" onclick="document.menusearchform.submit();">
        <font style="font-size:12">
            <a href="#" name="searchlink" id="searchlink" onclick="document.menusearchform.submit();" style="color:393939;text-decoration:none;font-size:12;" onmouseover="this.style.color='blue';" onmouseout="this.style.color='393939'"> <nobr>Select Properties </nobr></a><nobr>


        </nobr></font><nobr>
    </nobr></td>




    <td bgcolor="F78E1E" nowrap="" valign="top" align="center" onmouseover="this.style.backgroundColor='snow'; this.style.cursor='hand'; selectionlink.style.color='blue'" onmouseout="this.style.backgroundColor='F78E1E'; selectionlink.style.color='393939'" onclick="document.menuselectionlistform.submit();">
        <font style="font-size:12">
            <a href="#" name="selectionlink" id="selectionlink" onclick="document.menuselectionlistform.submit();" style="color:393939;text-decoration:none;font-size:12;" onmouseover="this.style.color='blue';" onmouseout="this.style.color='393939'"> <nobr>View Selected Properties </nobr></a><nobr>
        </nobr></font><nobr>
    </nobr></td>

    <td bgcolor="F78E1E" nowrap="" valign="top" align="center" onmouseover="this.style.backgroundColor='snow'; this.style.cursor='hand'; reportslink.style.color='blue'" onmouseout="this.style.backgroundColor='F78E1E'; reportslink.style.color='393939'" onclick="document.menureportsform.submit();">
        <font style="font-size:12">
            <a href="#" name="reportslink" id="reportslink" onclick="document.menureportsform.submit();" style="color:393939;text-decoration:none;font-size:12;" onmouseover="this.style.color='blue';" onmouseout="this.style.color='393939'"> <nobr>View Property Details </nobr></a><nobr>
        </nobr></font><nobr>
    </nobr></td>


    <td bgcolor="F78E1E" valign="middle" align="center" width="100%" height="100%"><font style="font-size:1pt"> </font></td>
</tr>

</tbody></table>

</td></tr>
</tbody></table>
</font>

<br />

<table width="100%" height="60%">
<tbody><tr>
<td align="Left" valign="top"><font face="Arial" size="2"><font size="2" face="TheSansOffice">
<font size="3" color="#0065A4">Welcome to the City of Coquitlam's Property Inquiry System.</font><br /><br />
<!--
<font face ="TheSansOffice" color=blue> Due to server maintenance, you may experience intermittent disruption on Thursday June 11, 2009 between 10 PM and 11 PM. <br>Thank you for your patience.<br><br></font>
-->
<!--
<font face ="TheSansOffice" color=blue> Please note: This website will be unavailable from Tuesday August 25, 2009 from 8:30 pm to 11:30 pm for server maintenance. Thank you for your patience. <br></font>
-->
You are now able to view information related to properties located in the city. You can
<table>
<tbody><tr><td><li> check property assessments,
</li><li> view legal property descriptions,
</li><li> view property tax and utility levies.</li></td></tr></tbody></table><br />
Select a property using the "Select Properties" tab.<br />
Select the information you want using the "View Property Details" tab. <br /><br />
<invalidtag language="JavaScript">
<!--
function goSetup() {
    var opts = new String("status,resizable,scrollbars");
    var newwin = window.open("../webinquiry/inhouse/accountreq.cfm", "Coquitlam", opts);
    if (document.all){
    newwin.moveTo(0,0)
    newwin.resizeTo(screen.width,screen.height)
    }
    return;
}
//-->

<!--
If you have an account with us, click "Registered User Login" above to obtain a tax certificate or apply a payment to your Web Customer Account.
<br> <br>
If you have any questions on the information provided, please contact us via <a href="mailto:PropertyTaxInquiries@coquitlam.ca">email</a> or call us at 604-927-3050.<br>
-->
<!--
<P>For a Web Customer Account application, please click <a href="#" onclick="goSetup();">Account Request Form</a>. We will contact you as soon as<br> the account has been set up.
<p>
-->
</invalidtag></font></font><p><font face="Arial" size="2"><font size="2" face="TheSansOffice">
<!--If you are already a customer of BC OnLine, log on to their web site at <a target="BC Online" href="https://www.bconline.gov.bc.ca"> www.bconline.gov.bc.ca</a> and select Tax Certificates Online from the main menu. <br>
For information about BC OnLine, or to become a customer, see their website or call them at (250)953-8250.
-->
If you require a <font size="3" color="#0065A4">Property Tax Certificate</font> please visit one of the websites noted below:
<br /><br />
•       Visit <a href="http://www.ltsa.ca">myLTSA Enterprise</a> for more information about myLTSA and becoming a customer. If you already have an account, log in and from the main menu, select Service Providers and Order Tax Certificates.
<br />
•       Visit <a href="http://www.bconline.gov.bc.ca">BC Online</a> for more information about BC Online and becoming a customer. If you are already a customer of BC Online, log in and select Tax Certificates Online from the main menu.
<br />
•       Visit <a href="http://www.apicanada.com">APIC</a> if you prefer to pay for the tax certificate by credit card, open an account and order your Tax Certificate.

</font></font></p></td>
</tr>
</tbody></table>

<br />


<table width="100%">
<tbody><tr>
<td valign="top">

<table cellpadding="4" cellspacing="0" width="100%" bgcolor="F78E1E">
<tbody><tr>
    <td>

            <font face="arial" size="2" color="404040">

                            <a href="http://www.coquitlam.ca/city-services/taxes-utilities/property-taxes.aspx" style="text-decoration:underline" target="_blank">Coquitlam Property Taxes &amp; Assessment</a>

                         | 

                            <a href="http://www.bcassessment.bc.ca/" style="text-decoration:underline" target="_blank">BC Assessment Authority</a>

                         | 

                            <a href="http://www.sbr.gov.bc.ca/individuals/Property_Taxes/Home_Owner_Grant/hog.htm" style="text-decoration:underline" target="_blank">Homeowner Grant</a>

                         | 

                            <a href="http://www.coquitlam.ca/city-services/taxes-utilities/property-taxes/how-to-pay-your-property-tax-bill.aspx" style="text-decoration:underline" target="_blank">Homeowner Grant</a>


            </font>

    </td>
</tr>
</tbody></table>

</td>
</tr>
</tbody></table>
<hr />
<table width="100%" cellpadding="0" cellspacing="0" border="0" bordercolor="Black">
<tbody><tr>
<td align="Left">
    <table cellpadding="0" cellspacing="2" border="0" bordercolor="Black">
        <tbody><tr>
            <td><font size="1" face="TheSansOffice">Copyright © 2001-2018 <a href="http:\\www.tempestdg.com" target="_blank">Tempest Development Inc.</a></font></td>

            <td width="10"> </td>

            <td><font size="1" face="TheSansOffice">Copyright © 2001-2018 <a href="http:\\www.coquitlam.ca" target="_blank">City of Coquitlam</a></font></td>


        </tr>
    </tbody></table>
</td>
</tr>
</tbody></table>


</body></html>

【讨论】：