scrapy splash渲染js页面的问题答案

【问题标题】：Issue with scrapy splash rendering js pagescrapy splash渲染js页面的问题
【发布时间】：2018-05-30 10:57:20
【问题描述】：

我在抓取包含动态加载内容的网页时遇到问题。我用以下方式启动了splash docker镜像：

docker run -p 8050:8050 scrapinghub/splash --disable-private-mode

我的 scrapy-splash 蜘蛛使用 LUA 脚本，它应该滚动并返回整个页面的 HTML：

import scrapy
from scrapy_splash import SplashRequest

class MySplashSpider(scrapy.Spider):
    # requires the scrapy-splash docker image running
    name = "psplash" 

    def __init__(self):
        self.domain = 'http://www.phillips.com'
        self.user_agent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:10.0) Gecko/20100101 Firefox/10.0"
        self.script = """
                        function main(splash)
                            local num_scrolls = 3
                            local scroll_delay = 1.0
                            splash:set_viewport_full()
                            splash:wait(5.0)
                            return splash:html()
                        end
                      """ 
        self.splash_args = {'lua_source': self.script,
                            'ua': self.user_agent
                            }

    def start_requests(self):
        base_url = "https://www.phillips.com/auctions/past/filter/Department=20TH%20CENTURY%20%26%20CONTEMPORARY%20ART!Editions!Latin%20America!Photographs"
        yield SplashRequest(base_url,
                            callback = self.parse_pagination,
                            endpoint = 'execute', 
                            args = self.splash_args
                            )      

    def parse_pagination(self, response):
        print('xxxxxxxxxx', response.xpath("//footer/ul/li[last()-1]/a/text()").extract())
        print('xxxxxxxxxx', response.xpath("//h2/a/@href").extract())

使用 chrome 开发工具检查时，我得到了 29 for //footer/ul/li[last()-1]/a/text() 为什么response.xpath 没有结果：

[
{"response_text": "hello world", "response_xpath_value": []}
]

控制台输出显示没有错误：

2017-12-16 13:05:16 [scrapy.core.engine] INFO: Spider opened
2017-12-16 13:05:16 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
xxxxxxxxxx []
xxxxxxxxxx []
2017-12-16 13:05:21 [scrapy.core.engine] INFO: Closing spider (finished)
2017-12-16 13:05:21 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 986,
 'downloader/request_count': 1,
 'downloader/request_method_count/POST': 1,
 'downloader/response_bytes': 163,
 'downloader/response_count': 1,
 'downloader/response_status_count/200': 1,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2017, 12, 16, 12, 5, 21, 707451),
 'log_count/INFO': 7,
 'log_count/WARNING': 1,
 'response_received_count': 1,
 'scheduler/dequeued': 2,
 'scheduler/dequeued/memory': 2,
 'scheduler/enqueued': 2,
 'scheduler/enqueued/memory': 2,
 'splash/execute/request_count': 1,
 'splash/execute/response_count/200': 1,
 'start_time': datetime.datetime(2017, 12, 16, 12, 5, 16, 816927)}
2017-12-16 13:05:21 [scrapy.core.engine] INFO: Spider closed (finished)

我错过了什么？

【问题讨论】：

标签： python python-3.x lua scrapy scrapy-splash

【解决方案1】：

将响应正文保存在 HTML 文件中，然后检查您是否下载了整个页面，尤其是根据您的要求。如果是，请尝试使用selectors

【讨论】：