我如何抓取这种动态生成的网站数据？答案

【问题标题】：how do i scrape this kind of dynamic generated website data?我如何抓取这种动态生成的网站数据？
【发布时间】：2019-07-07 08:46:49
【问题描述】：

我正在尝试抓取电子商务网站，示例链接：https://www.lazada.sg/products/esogoal-2-in-1-selfie-stick-tripod-bluetooth-selfie-stand-with-remote-shutter-foldable-tripod-monopod-i279432816-s436738661.html?mp=1

数据是通过 React 呈现的，当我对几个链接执行 scraping 时，大部分数据都以null 的形式返回，当我查看页面源代码时，我实际上找不到 HTML通过检查元素可用，只是 Javascript 标记中的一个 json。我测试了几次在以前没有找到的相同链接和数据上运行 scrapy scraper，实际上返回内容，所以它以某种方式随机。我不知道我应该如何抓取这种网站。我也在使用用户代理池和请求之间的中断。

 script = '''
        function main(splash, args)
            assert(splash:go(args.url))
            assert(splash:wait(1.5))
            return splash:html()
        end
    '''

    def start_requests(self):
        url= [
            'https://www.lazada.sg/products/esogoal-tactical-sling-bag-outdoor-chest-pack-shoulder-backpack-military-sport-bag-for-trekking-camping-hiking-rover-sling-daypack-for-men-women-i204814494-s353896924.html?mp=1',
            'https://www.lazada.sg/products/esogoal-2-in-1-selfie-stick-tripod-bluetooth-selfie-stand-with-remote-shutter-foldable-tripod-monopod-i279432816-s436738661.html?mp=1',
            'https://www.lazada.sg/products/esogoal-selfie-stick-tripod-extendable-selfie-stick-monopod-with-integrated-tripod-and-bluetooth-remote-shutter-wireless-selfie-stick-tripod-for-cellphonecameras-i205279097-s309050125.html?mp=1',
            'https://www.lazada.sg/products/esogoal-mini-umbrella-travel-umbrella-sun-rain-umbrella8-ribs-98cm-big-surface-lightweight-compact-parasol-uv-protection-for-men-women-i204815487-s308312226.html?mp=1',
            'https://www.lazada.sg/products/esogoal-2-in-1-selfie-stick-tripod-bluetooth-selfie-stand-with-remote-shutter-foldable-tripod-monopod-i279432816-s436738661.html?mp=1'
        ]

        for link in url:
            yield SplashRequest(url=link, callback=self.parse, endpoint='render.html', args={'wait' : 0.5, 'lua_source' : self.script}, dont_filter=True)

 def parse(self, response):
        yield {
            'title' : response.xpath("//span[@class='pdp-mod-product-badge-title']/text()").extract_first(),
            'price' : response.xpath("//span[contains(@class, 'pdp-price')]/text()").extract_first(),
            'description' : response.xpath("//div[@id='module_product_detail']").extract_first()
        }

【问题讨论】：

你有没有考虑过，而不是使用 Splash，换一种方式尝试以与 React 代码相同的方式获取底层数据？见stackoverflow.com/q/8550114/939364 和docs.scrapy.org/en/master/topics/dynamic-content.html
也许他们正在阻止基于 IP 的请求。您是否考虑过使用代理？对我来说，当我使用在本地运行的 Splash 实例时，它似乎可以工作，但是来自在 Google Cloud 中运行的 Splash 实例的请求无法正常工作。

标签： python scrapy scrapy-splash

【解决方案1】：

我试试这个：

传递 'execute' 作为 splash 方法的参数而不是 'render html'

from scrapy_splash import SplashRequest

class DynamicSpider(scrapy.Spider):
name = 'products'
url = [
    'https://www.lazada.sg/products/esogoal-tactical-sling-bag-outdoor-chest-pack-shoulder-backpack-military-sport-bag-for-trekking-camping-hiking-rover-sling-daypack-for-men-women-i204814494-s353896924.html?mp=1',
    'https://www.lazada.sg/products/esogoal-2-in-1-selfie-stick-tripod-bluetooth-selfie-stand-with-remote-shutter-foldable-tripod-monopod-i279432816-s436738661.html?mp=1',
    'https://www.lazada.sg/products/esogoal-selfie-stick-tripod-extendable-selfie-stick-monopod-with-integrated-tripod-and-bluetooth-remote-shutter-wireless-selfie-stick-tripod-for-cellphonecameras-i205279097-s309050125.html?mp=1',
    'https://www.lazada.sg/products/esogoal-mini-umbrella-travel-umbrella-sun-rain-umbrella8-ribs-98cm-big-surface-lightweight-compact-parasol-uv-protection-for-men-women-i204815487-s308312226.html?mp=1',
    'https://www.lazada.sg/products/esogoal-2-in-1-selfie-stick-tripod-bluetooth-selfie-stand-with-remote-shutter-foldable-tripod-monopod-i279432816-s436738661.html?mp=1',
]

script = """
    function main(splash, args)
      assert(splash:go(args.url))
      assert(splash:wait(1.5))
      return {
        html = splash:html()
      }
    end
"""

def start_requests(self):
    for link in self.url:
        yield SplashRequest(
            url=link,
            callback=self.parse,
            endpoint='execute',
            args={'wait': 0.5, 'lua_source': self.script},
            dont_filter=True,
        )

def parse(self, response):
    yield {
        'title': response.xpath("//span[@class='pdp-mod-product-badge-title']/text()").extract_first(),
        'price': response.xpath("//span[contains(@class, 'pdp-price')]/text()").extract_first(),
        'description': response.xpath("//div[@id='module_product_detail']/h2/text()").extract_first()
    }

这是结果

【讨论】：