用 cssselct 刮擦答案

【问题标题】：scrapy with cssselct用 cssselct 刮擦
【发布时间】：2020-01-09 08:54:21
【问题描述】：

我正在尝试使用cssselect 进行练习，不幸的是，当我阅读<h1> 时，我将向您展示示例：

我在本练习中使用的 HTML 代码如下：

<h1 class="lbl_titulo">3 Bedroom House<span class="subtit"> - Well located house in  the heart of Lapa (Lisbon) </span></h1>

    <div class="bloco-imovel-dados">

                        <div class="bloco-imovel-resumo-dados">
                            <div id="Cpl_modulodadosresumidos_module_holder"     class="modulo-dados-resumidos">

    <h2 class="lbl_descricao_dados">Property Information</h2>

    <ul class="bloco-dados">

        <li>
            <b>Condition:</b> <span>Renewed</span></li>
        <li>
            <b>Living Area:</b><span> 80 m<sup>2</sup></span></li>
        <li>
            <b>Total Area:</b><span> 0 m<sup>2</sup></span></li>
        <li>
            <b>Bathrooms:</b><span> 1 </span></li>
        <li>
            <b>Bedrooms:</b><span> 2 </span></li>
        <li>
            <b>Energy Rating:</b><span> C</span></li>
        <li>
            <b>Construction year:</b><span> 1997</span></li>
        <li>
            <b>ID Property:</b><span> CAS.10.13286</span></li>

    </ul>

</div>

    <div class="row pref-search-results"><script>setTimeout(function(){     $(".pref-search-results").addClass("pref-search-result").removeClass("pref-    search-results") }, 1000)</script>
        <div class="col-lg-4 col-md-4 col-sm-6 col-xs-12"><a class="pref-   property-container" href="http://xxxxxxxxx.com/link-to-image" style="height:    420px;">
            <div class="pref-teaser-shadow"></div>
            <div class="pref-teaser-image">
                <div class="pref-teaser-icons-container bottom">
                    <div class="pref-watchlist-teaser-icon-container">
                        <div class="pref-teaser-icon pref-watchlist-icon active initial-hide"></div>
                        <div class="pref-teaser-icon pref-watchlist-icon inactive "></div>
                    </div>
                </div>
                <div class="pref-teaser-shadow"></div>
                <img src="https://upload.wikimedia.org/wikipedia/commons/thumb/c/c9/Ranch_style_home_in_Salinas%2C_California.JPG/220px-Ranch_style_home_in_Salinas%2C_California.JPG" title="Link to Image" alt="Link Image">
            </div>
        </a></div>

                    </div>
                    <div class="bloco-imovel-caracteristicas-holder">
                        <div id="Cpl_modulocaracteristicas_module_holder" class="modulo-caracteristicas">
    <a class="pref-property-container" href="http://xxxxxxxxxx.com/link-to-listing" style="height: 420px;">Link 1</a>
    <a class="pref-property-container" href="http://xxxxxxxxxx.com/link-to-listing" style="height: 420px;">Link 2</a>
    <h3 class="lbl_titulo_caracteristicas">Features</h3>
    <div id="Cpl_modulocaracteristicas_div_caracteristicas_gerais" class="modulo-caracteristicas-item  open">
        <ul class="modulo-caracteristicas-conteudo js-caracteristicas-holder">
            <li>
                <span id="features">Garden</span>
            </li>
            <li>
                <span id="features">Gas Heating</span>
            </li>
            <li>
                <span id="features">2 garages</span>
            </li>
            <li>
                <span id="features">Large pool</span>
            </li>
        </ul>
    </div>


</div>

                    </div>

                    <div class="bloco-imovel-texto">
                        <h3 class="lbl_description">
                            Description </h3>
                        <p>At vero eos et accusamus et iusto odio dignissimos ducimus qui blanditiis praesentium voluptatum deleniti atque corrupti quos dolores et quas molestias excepturi sint occaecati cupiditate non provident, similique sunt in culpa qui officia deserunt mollitia animi, id est laborum et dolorum fugaEt harum quidem rerum facilis est et expedita distinctio.Nam libero tempore, cum soluta nobis est eligendi optio cumque nihil impedit quo minus id quod maxime placeat facere possimus, omnis voluptas assumenda est, omnis dolor repellendus.</p>
                    </div>

                     <div class="bloco-imovel-content">
            <!-- Galeria -->
            <div class="bloco-imovel-galeria">
                <div id="Cpl_modulogaleriavertical_module_holder" class="modulo-galeria-vertical">
            <span id="Cpl_modulogaleriavertical_lbl_galeria" class="lbl_galeria"><b>Relevant Information</b></span>

</div>

                    <div id="Cpl_pnl_mapa" class="pnl_mapa">

                        <span id="Cpl_lbl_mapa" class="lbl_mapa"><b>Location:</b></span>
                        <span id="Cpl_lbl_morada" class="lbl_morada">Portugal, Lisboa, Estrela, Lapa</span>
                        <div id="Cpl_modulomapa_module_holder" class="modulo-mapa">

    <div class="bloco-mapa">
        <div id="Cpl_modulomapa_mapa" class="mapa"></div>
        <div id="map-canvas" data-coorgps="36.5194999,-4.7743365"></div> 
    </div>

    </div>


</div>
                </div>
            </div>
            <div class="bloco-imovel-sidebar">
                <div id="Cpl_moduloinformacaolateral_module_holder" class="modulo-informacao-lateral">

    <div class="informacao">
        <div class="info1">
            <div class="lbl_titulo">Agent John Doe</div>
            <div class="lbl_subtitulo">
            Contact Information
            </div>
            <div class="lbl_resumo">
            Phone Number: 0800-1111<br><b><a href="mailto:casa@casa.pt">casa@casa.pt</a></b><br><b></b>
            </div>
        </div>
    </div>

</div>

<a class="pref-property-container" href="http://xxxxxxxxxx.com/link-to-listing" style="height: 420px;">Link 3</a>

</div></div></div>

所以我当然会做安装：

!pip install lxml
!pip install cssselect
from lxml import html,etree

我打开并阅读文件

with open(r'listing.html', "r") as f:
    page = f.read()
tree = html.fromstring(page)

然后我想抓取以下信息：

Name of the property
Number of bathrooms
Number of bedrooms
Living Area
Energy Rating
Description
Agent Name
Extract the location of the property

所以我尝试做的是：

Selector_Example = "h1.lbl_titulo"
print('Example -> Property type: {}'.format(tree.cssselect(Selector_Example)[0].text))

我得到了答案：

Example -> Property type: 3 Bedroom House

当我尝试使用其他示例时，我总是遇到错误：

#Number of bathrooms
Selector_1 = "li:nth-child(1)"
print('Bathrooms: {}'.format(tree.cssselect(Selector_1)[0].text))
print('')
# this returns just the word Bathrooms: but I tried to use `li.b` and does not work as well.

#Number of bedrooms
Selector_2 = "li:nth-child(2)"
print('Bedrooms: {}'.format(tree.cssselect(Selector_2)[0].text))
print('')
# this returns just the word Bedrooms: but I tried to use `li.b` and does not work as well.

#Living Area
Selector_3 = "li:nth-child(3)"
print('Total area: {}'.format(tree.cssselect(Selector_3)[0].text))
print('')
this returns just the words Living Area: but I tried to use `li.b` and does not work as well.

#Energy Rating
Selector_4 = "li:nth-child(4)"

#Description
Selector_5 = "h3.lbl_description"
print('Description: {}'.format(tree.cssselect(Selector_5)[0].text))
print('')
# This returns the word Description: Description but not the description.



#Agent Name
Selector_6 = "div.lbl_titulo"
print('Agent name: {}'.format(tree.cssselect(Selector_6)[0].text))
print('')
# This gives my in fact the agent name: Agent John Doe

#Extract the location of the property
Selector_7 = "div.Cpl_lbl_morada.lbl_morada"
print('Location: {}'.format(tree.cssselect(Selector_7)[0].text))
# I got direct : IndexError: list index out of range

有人知道我做错了什么以及如何实际修改此代码吗？提前感谢您的建议！

【问题讨论】：

标签： python css regex css-selectors

【解决方案1】：

你的例子：

#Number of bedrooms
Selector_2 = "li:nth-child(2)"
print('Bedrooms: {}'.format(tree.cssselect(Selector_2)[0].text))

据我所知，在这种情况下"li:nth-child(2)" 返回'\n ' 第二个li 元素是什么，"li:nth-child(3)" 返回第三个li。如果你想得到Bedrooms你需要使用

#Number of bedrooms
Selector_2 = "li:nth-child(5) span"
print('Bedrooms: {}'.format(tree.cssselect(Selector_2)[0].text))

li:nth-child(5) 因为Bedrooms 是第5 个li 元素和span，因为您需要来自li 中的span 元素的信息

编辑。您分配给Selector_ 变量的字符串是CSS，因此您可以使用与CSS 文件中相同的符号。所以li.b 表示<li class='b'>

【讨论】：

非常感谢，它适用于除 description 之外的所有人...如果我这样做 #Description Selector_5 = "h3.lbl_description" print('Description: {}'.format(tree.cssselect (Selector_5)[0].text)) print('') # 这将返回单词描述：描述但不是描述。当我尝试输入scan 或<p> 时，我总是遇到错误……知道为什么吗？
@fachc 以及为什么你不能这样做Selector_5 = "div.bloco-imovel-texto p"
非常感谢，我尝试了所有可能的方法，但使用 h3，它完全符合您的要求！
很抱歉再次打扰您，但我又被 css 选择器项目卡住了......我正在尝试在链接上刮擦（xxxxxxxxx.com/link-to-image），我尝试了所有可能的方法，但我得到了错误或者只是一个空字符串：Selector_8 = "a.pref-property-container "print('Link: {}'.format(tree.cssselect(Selector_8)[0].text)) 这个只给了我这个词 [link:] 没有别的。我也试过了：'span[class=pref-property-container] a::attr(href)').extract()[0] 但结果也是一个错误，你知道吗有什么问题？
最后一个问题我需要抓取所有功能，所以我这样做了：Selector_9 = "div.bloco-imovel-caracteristicas-holder span" print('features: {}'.format(tree.cssselect(Selector_9)[0].text)) 但我只得到了第一个而不是全部？有没有办法抓住所有 4 个功能？