Selenium Python Loop 只获得第一个项目答案

【问题标题】：Selenium Python Loop only gets fist itemSelenium Python Loop 只获得第一个项目
【发布时间】：2017-06-28 18:36:46
【问题描述】：

我无法获取完整的汽车列表。我只得到每一页的第一辆车。我喜欢字典的组织方式，但我愿意接受任何建议。下一个功能将简单地获取有效负载并将其发送到数据库。我想使用 executemany(sqlCode, payload)，但我可能会使事情复杂化我希望能够

这是循环迭代的相关部分：

Form = driver.find_elements_by_xpath('//*[@id="compareForm"]')
cars = driver.find_elements_by_xpath('*//div[@class="description"]')
time.sleep(10)
for car in Form:

其余代码：

import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from selenium.common.exceptions import ElementNotVisibleException
from selenium.common.exceptions import NoSuchElementException


def init_driver():
    options = webdriver.ChromeOptions()
    options.binary_location = '/usr/bin/google-chrome-stable'
    #options.add_argument('headless')
    options.add_argument('window-size=1200x600')
    driver = webdriver.Chrome(chrome_options=options)
    driver.wait = WebDriverWait(driver, 5)
    return driver


def scrape(driver):



    Form = driver.find_elements_by_xpath('//*[@id="compareForm"]')
    cars = driver.find_elements_by_xpath('*//div[@class="description"]')
    time.sleep(10)
    for car in Form:
        payload = [{
        "Year" : car.find_element_by_xpath("//*[starts-with(@class, 'item')]").get_attribute("data-year"),
        "Make" :  car.find_element_by_xpath("//*[starts-with(@class, 'item')]").get_attribute("data-make"),
        "Model" : car.find_element_by_xpath("//*[starts-with(@class, 'item')]").get_attribute("data-model"),
        "Trim" : car.find_element_by_xpath("//*[starts-with(@class, 'item')]").get_attribute("data-trim"),
        "Engine" : car.find_element_by_xpath("//dt[contains(text(),'Engine:')]/following-sibling::dd").text,
        "Transmission" : car.find_element_by_xpath("//dt[contains(text(),'Transmission:')]/following-sibling::dd").text,
        "Mileage" : car.find_element_by_xpath("//dt[contains(text(),'Mileage:')]/following-sibling::dd").text,
        "MPG_Range" : car.find_element_by_xpath("//dt[contains(text(),'MPG Range:')]/following-sibling::dd").text,
        "Exterior_Color" : car.find_element_by_xpath("//dt[contains(text(),'Exterior Color:')]/following-sibling::dd").text,
        "Interior_Color" : car.find_element_by_xpath("//dt[contains(text(),'Interior Color:')]/following-sibling::dd").text,
        "Stock_No" : car.find_element_by_xpath("//dt[contains(text(),'Stock #:')]/following-sibling::dd").text,
        "VIN" : car.find_element_by_xpath("//*[starts-with(@class, 'item')]").get_attribute("data-vin"),
        "Type" : car.find_element_by_xpath("//*[starts-with(@class, 'item')]").get_attribute("data-type"),
        "classification" : car.find_element_by_xpath("//*[starts-with(@class, 'hproduct')]").get_attribute("data-classification"),
        "Price" : car.find_element_by_xpath("//*[starts-with(@class, 'internetPrice')]/span[2]").text
        }]

        Final.append(payload)
        print Final;
    time.sleep(15)

新：

def scrape(driver):


    # for
    #get data from first pagebs
    cars = driver.find_elements_by_xpath('//*[@id="compareForm"]/div/div[2]/ul')
    print cars;
    time.sleep(10)
    for car in cars: #car.find_element_by_xpat('//li[contains(text(),"low to high")]h()')
        car_element = car.find_element_by_xpath(".//*[starts-with(@class, 'item')]")
        payload = [{
        "Year" : car_element.get_attribute("data-year"),
        "Make" :  car_element.get_attribute("data-make"),
        "Model" : car_element.get_attribute("data-model"),
        "Trim" : car_element.get_attribute("data-trim")
        "Engine" : car.get_attribute("data-engine"),
        "Transmission" : car.get_attribute("data-transmission"),
        "Exterior_Color" : car.get_attribute("data-exteriorcolor"),
        "Interior_Color" : car.get_attribute("data-interiorcolor"),
        "VIN" : car.get_attribute("data-vin"),
        "Type" : car.get_attribute("data-type"),
        # "Stock_No" : driver.find_element_by_xpath(".//dt[contains(text(),'Stock #:')]/following-sibling::dd").text,
        # "Mileage" : driver.find_element_by_xpath(".//dt[contains(text(),'Mileage:')]/following-sibling::dd").text,
        # "MPG_Range" : driver.find_element_by_xpath(".//dt[contains(text(),'MPG Range:')]/following-sibling::dd").text,
        # "Price" : driver.find_element_by_xpath(".//*[starts-with(@class, 'internetPrice')]/span[2]").text,
        # "classification" : driver.find_element_by_xpath(".//*[starts-with(@class, 'hproduct')]").get_attribute("data-classification")
        }]

        Final.append(payload)
        print Final;
    time.sleep(15)

谢谢，我已经做了调整，@Andersson 我在传递元素之前打印了它们，这是成功的：

selenium.webdriver.remote.webelement.WebElement (session="d5b2d93b4e8ad78be19a080cc34d548b", element="0.271760196680197-12")>,

元素="0.271760196680197-13")>,

元素="0.271760196680197-14")>,

元素="0.271760196680197-15")>,

元素="0.271760196680197-16")>,

element="0.271760196680197-17")>]

ETC

由于某种原因，在第二次执行循环时找不到该项目。

这是 HTML，我确保使用 (.) 以便从当前节点查看。

我使用了该元素，我希望希望能够遍历

以 item 开头的元素

cars = driver.find_elements_by_xpath('//*[@id="compareForm"]/div/div[2]/ul')
print cars;
time.sleep(10)
for car in cars: #car.find_element_by_xpat('//li[contains(text(),"low to high")]h()')
    car_element = car.find_element_by_xpath(".//*[starts-with(@class, 'item')]")
    payload = [{
    "Year" : car_element.get_attribute("data-year"),

这里是 HTML

<div class="hproduct auto volkswagen clearfix" data-classification="primary" data-type="certified" data-internetprice="MTI5MDMuMA==" data-msrp="MC4w" data-exteriorcolor="Reflex Silver Metallic" data-bodystyle="Sedan" data-trim="Auto SEL Pzev" data-year="2014" data-model="Jetta" data-vin="3VWL17AJ8EM394570" data-make="Volkswagen" data-index-position="7">
<div>
<div class="media">
<a href="/certified/Volkswagen/2014-Volkswagen-Jetta-afc6c4970a0e0adf279214712962aff7.htm">
<img class="photo thumb" src="https://pictures.dealer.com/ddc/resize/320x/quality/70/sharpen/1/ddc/c/carolinavolkswagenvw/0554/c9f0a2adc8cdb8a719d211c045fa1e6bx.jpg" data-src="https://pictures.dealer.com/ddc/resize/320x/quality/70/sharpen/1/ddc/c/carolinavolkswagenvw/0554/c9f0a2adc8cdb8a719d211c045fa1e6bx.jpg" alt="2014 Volkswagen Jetta Auto SEL Pzev Sedan" title="2014 Volkswagen Jetta Auto SEL Pzev Sedan">
</a>
<a class="dialog video ddc-btn ddc-btn-link ddc-btn-small" href="#" title="2014 Volkswagen Jetta Auto SEL Pzev Sedan" data-title=" 2014 Volkswagen Jetta Auto SEL Pzev Sedan" data-href="/carflix-popup.htm?hideVcard=false&amp;videoUrl=//videos2.dealer.com/clients/c/carolinavolkswagenvw/carflix2/afc6c4970a0e0adf279214712962aff7/dfbd71dd0a0a00f7116d789e7f2f80ae.xml&amp;videoHeight=400&amp;videoWidth=680&amp;autoPlay=true&amp;itemId=afc6c4970a0e0adf279214712962aff7&amp;category=AUTO" data-min-width="730" data-min-height="500" data-draggable="true" data-modal="false" data-destroy-on-close="true">
<i class="ddc-icon ddc-icon-video"></i>
Watch Video
</a>

</div>


<h3 class="fn  "><a class="url" href="/certified/Volkswagen/2014-Volkswagen-Jetta-afc6c4970a0e0adf279214712962aff7.htm"> 2014 Volkswagen Jetta Auto SEL Pzev Sedan</a></h3>
<div class="pricing-area has-buttons">
    <ul class="pricing  multiple-prices has-eprice list-unstyled">
        <li style="margin-bottom: 10px;"><span class="retailValue"><span class="label">Average Market Price<span class="separator">:</span></span><span class="value">$17,950</span></span></li>

        <li style="margin-bottom: 10px;"><span class="abSub"><span class="label">Savings<span class="separator">:</span></span><span class="value">$5,047</span></span></li>

        <li style="margin-bottom: 10px;"><span class="internetPrice final-price"><span class="label">Carolina VW Value Price*<span class="separator">:</span></span><span class="value">$12,903</span></span></li>

        <li style="margin-bottom: 10px;"><a data-href="/eprice-form.htm?itemId=afc6c4970a0e0adf279214712962aff7&amp;vehicleId=afc6c4970a0e0adf279214712962aff7&amp;category=AUTO&amp;parentPageAlias=INVENTORY_LISTING_DEFAULT_AUTO_ALL" class="dialog pointer epriceLink" data-eprice="true" data-title="Get Your ePrice"><img class="customEpriceImage" src="https://pictures.dealer.com/c/carolinavolkswagenvw/0001/25b1b5172b1c8523a5b83e0ec8b14cebx.jpg"></a></li>

        <li style="margin-bottom: 10px;">
        <a class="dialog pointer ddc-btn-block ui-button ui-widget ui-state-default ui-corner-all ui-button-text-only ddc-btn ddc-btn-primary sms-button custom3pcTextUsBtn" href="sms:+17042766637" data-make="Volkswagen" data-model="Jetta" data-vin="3VWL17AJ8EM394570" data-year="2014" data-status="Used" style="visibility: visible;" data-sdk-attached="true"><span class="ui-button-text">Text Us</span></a>
        </li>
    </ul>
</div>

<div class="description">
<dl> <dt>Engine:</dt> <dd>1.8L I-4 cyl<span class="separator">,</span></dd> <dt>Transmission:</dt> <dd>Automatic<span class="separator">,</span></dd> <dt>Mileage:</dt> <dd>45,084 miles<span class="separator">,</span></dd> <dt>MPG Range:</dt> <dd>25/36<span class="separator">,</span></dd></dl><dl class="last"> <dt>Exterior Color:</dt> <dd>Reflex Silver Metallic<span class="separator">,</span></dd> <dt>Interior Color:</dt> <dd>Titan Black<span class="separator">,</span></dd> <dt>Stock #:</dt> <dd>P3237</dd></dl> <span class="ddc-more">More<span class="hellip">…</span></span> 
<div class="calloutDetails">
<ul class="list-unstyled">
<li class="certified" style="margin-bottom: 10px;"><div class="badge "><img class="align-center" src="https://static.dealer.com/v8/global/images/franchise/white/en_US/logo-certified-volkswagen.gif?r=1356028132000" alt="Certified"></div></li><li class="carfax" style="margin-bottom: 10px;"><a href="http://www.carfax.com/cfm/ccc_displayhistoryrpt.cfm?partner=DLR_3&amp;vin=3VWL17AJ8EM394570" class="badge carfax-one-owner pointer" target="_blank"><img class="align-center" src="https://static.dealer.com/v8/global/images/franchise/white/logo-certified-carfax-one-owner-lrg.png?r=1405027620000" alt="Carfax One Owner"></a></li>
</ul>
</div>
<div class="hproductDynamicArea"></div>
</div>
</div>
<div class="mycars-vlp ddc-btn-mycars">
<a class="ddc-btn ddc-btn-default mycars-btn mycars-save-btn ddc-btn-mycars" href="#" data-id="afc6c4970a0e0adf279214712962aff7" data-mycars-save="">
Save This Car!
</a>
<a class="ddc-btn ddc-btn-default mycars-btn mycars-saved-btn mycars-action-set ddc-btn-mycars   hide" href="#" data-id="afc6c4970a0e0adf279214712962aff7" label-remove="Remove Car" label-set="Saved!" data-mycars-remove="">
Saved!
</a>
<a class="ddc-btn ddc-btn-default mycars-btn mycars-add-alert-btn ddc-btn-mycars" href="#" data-id="afc6c4970a0e0adf279214712962aff7" data-mycars-get-alerts="">
Get Price Alerts
</a>
<a class="ddc-btn ddc-btn-default mycars-btn mycars-alerts-isset mycars-action-set ddc-btn-mycars  hide" href="#" data-id="afc6c4970a0e0adf279214712962aff7" label-remove="Remove Alerts" label-set="Alert Set!">
Alert Set!
</a>
</div></div>

【问题讨论】：

嗯，只有一个 id 为 compareForm 的元素。所以Form 将永远是一个单元素列表。
for car in Form: 正确吗？还是你的意思是写for car in cars:？
它的效率也可以大大提高。 find_element_by... 是一些昂贵的调用。如果你可以只做一次car_element=find_element_by_xpath(...) 或其他什么，那就做你的"Year": car_element.get_attribute("data-year")

标签： python loops selenium

【解决方案1】：

第一个问题似乎在你的循环中——你可能需要使用for car in cars而不是for car in Form

第二个问题- 你需要在 XPath 中指定对当前节点 (.) 的引用。

简单把这个

car.find_element_by_xpath("//*[starts-with(@class, 'item')]")

表示查找class属性值以"item"开头的元素的第一个出现从根html节点开始

此时

car.find_element_by_xpath(".//*[starts-with(@class, 'item')]")

表示查找class属性值以"item"开头的元素的第一个出现从当前car节点开始

【讨论】：

我已经调整了我的代码。如果你有机会看，我仍然有麻烦。
所以你在第一次迭代时得到了想要的输出，但在第二次你的car 元素找不到，对吧？你能澄清一下吗？