如何使用scrapy spider python在<ol> <li>下获取值

Question

我是网络爬虫新手，刚刚关注了这篇文章。这很容易理解。

https://www.digitalocean.com/community/tutorials/how-to-crawl-a-web-page-with-scrapy-and-python-3

我有 1 个网站目标来做这件事。目的是获取 ais-Hits-list 类下的产品价格和名称列表。

例如 --> 价格（259）和名称（XT7 女士越野跑鞋深蓝色和粉色）

<header id="header">
    <div id="search-suggestions-algolia" style="display:none">
        <div>
            <div class="ais-Hits">
                <ol class="ais-Hits-list">
                    <li class="ais-Hits-item"></li>
                    <li class="ais-Hits-item"></li>
                    <li class="ais-Hits-item"></li>
                    <li class="ais-Hits-item"></li>
                </ol>
            </div>
        </div>
    </div>
</header>
<section id="wrapper">
    <div id="#content" class="site-content shop-grid">
        <div id="">
            <div id="js-product-list">
                <div class="products product_content" id="hits">
                    <div>
                        <div class="ais-Hits">
                            <ol class="ais-Hits-list">
                                <li class="ais-Hits-item">...</li>
                                <li class="ais-Hits-item">...</li>
                                <li class="ais-Hits-item">
                                    <div class="price-button">
                                        <a class="algolia_link" href="/p/8552163_xt7-women-s-trail-running-shoes-dark-blue-and-pink.html">
                                            <div class="product_price">
                                                <button class="is-skewed" itemprop="price">259</button>
                                            </div>
                                            <div class="product_name">
                                                <h4 class="title-single" title="XT7 women's trail running shoes dark blue and pink"></h4>
                                            </div>
                                        </a>
                                        <a class="product-flags"></a>
                                    </div>
                                </li>
                                <li class="ais-Hits-item">...</li>
                            </ol>
                        </div>
                    </div>
                </div>
            </div>
        </div>
    </div>
</section>

我的代码是

# scrapy runspider scraper.py
import scrapy
import json

class xxxx(scrapy.Spider):
    name = 'xxxx_spider'
    start_urls = ['https://www.xxxx.co.id/8548777-running-shoes']
    allowed_domains = ['xxxx.co.id']


    def parse(self, response):

        PRODUCT_SELECTOR = '#js-product-list .product_content .ais-Hits .ais-Hits-list .ais-Hits-item'

        for item in response.css(PRODUCT_SELECTOR):
            NAME_SELECTOR = 'h4 ::attr(title)'
            PRICE_SELECTOR = 'button ::text'

            yield {
                'name': item.css(NAME_SELECTOR).extract_first(),
                'price': item.css(PRICE_SELECTOR).extract_first()
            }

但它总是什么也不返回。有没有我遗漏的部分？

2019-12-11 18:16:39 [scrapy.core.engine] INFO: Spider opened
2019-12-11 18:16:39 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2019-12-11 18:16:39 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2019-12-11 18:16:42 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.xxxx.co.id/8548777-running-shoes> (referer: None)
[]
2019-12-11 18:16:42 [scrapy.core.engine] INFO: Closing spider (finished)
2019-12-11 18:16:42 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 236,
 'downloader/request_count': 1,
 'downloader/request_method_count/GET': 1,
 'downloader/response_bytes': 29492,
 'downloader/response_count': 1,
 'downloader/response_status_count/200': 1,
 'elapsed_time_seconds': 2.876863,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2019, 12, 11, 10, 16, 42, 734103),
 'log_count/DEBUG': 1,
 'log_count/INFO': 10,
 'response_received_count': 1,
 'scheduler/dequeued': 1,
 'scheduler/dequeued/memory': 1,
 'scheduler/enqueued': 1,
 'scheduler/enqueued/memory': 1,
 'start_time': datetime.datetime(2019, 12, 11, 10, 16, 39, 857240)}
2019-12-11 18:16:42 [scrapy.core.engine] INFO: Spider closed (finished)

该网站有 2 个用于 ais-Hits-item 的类（标题和部分）。

之所以没有放PRODUCT_SELECTOR='.ais-Hits-item'，是因为它会直接指向 ais-Hits-item 标题，而不是第一节。

Answer 1

问题是这个页面是由 Javascript 呈现的。 您可以在浏览器的网络选项卡中看到它正在发出发布请求以检索数据。

您可以通过 Scrapy 发送 post 请求以检索 json 响应，然后对其进行解析。

另一个选项，因为页面中还有一个带有产品数据的 javascript 变量： 将页面上存在的var thispageproductjavascript变量加载到python字典中，然后解析它。

class xxxx(scrapy.Spider):
    name = 'xxxx_spider'
    start_urls = ['https://www.decathlon.my/8548777-running-shoes']
    allowed_domains = ['xxxx.co.id']

    def parse(self, response):

        page_data = json.loads((response.xpath('//script').re_first('var thispageproduct = (.*?);')))

        for product_id in page_data.keys():
            product_data = page_data.get(product_id)

            yield {
                'name': product_data.get('name'),
                'price': product_data.get('price')
            }