刮痧。第一反应需要 Selenium

Question

我正在抓取一个严重依赖 Javascript 的网站。我需要从中提取要解析的 url 的主页取决于 Javascript，因此我必须修改 start_requests。 我正在寻找一种方法来连接 start_requests、linkextractor 和 process_match

class MatchSpider(CrawlSpider):
    name = "match"
    allowed_domains = ["whoscored"]
    rules = (
    Rule(LinkExtractor(restrict_xpaths='//*[contains(@class, "match-report")]//@href'), callback='parse_item'),
)

    def start_requests(self):
        url = 'https://www.whoscored.com/Regions/252/Tournaments/2/Seasons/6335/Stages/13796/Fixtures/England-Premier-League-2016-2017'
        browser = Browser(browser='Chrome')
        browser.get(url)
        # should return a request with the html body from Selenium driver so that LinkExtractor rule can be applied


    def process_match(self, response):
        match_item = MatchItem()
        regex = re.compile("matchCentreData = \{.*?\};", re.S)
        match = re.search(regex, response.text).group()
        match = match.replace('matchCentreData =', '').replace(';', '')
        match_item['match'] = json.loads(match)
        match_item['url'] = response.url
        match_item['project'] = self.settings.get('BOT_NAME')
        match_item['spider'] = self.name
        match_item['server'] = socket.gethostname()
        match_item['date'] = datetime.datetime.now()
        yield match_item

我在 Selenium 周围使用的包装器：

class Browser:
    """
    selenium on steroids. allows you to create different types of browsers plus
    adds methods for safer calls
    """
    def __init__(self, browser='Firefox'):
        """
        type: silent or not
        browser: chrome of firefox
        """
        self.browser = browser
        self._start()

    def _start(self):
        '''
        starts browser
        '''
        if self.browser == 'Chrome':
            chrome_options = webdriver.ChromeOptions()
            prefs = {"profile.managed_default_content_settings.images": 2}
            chrome_options.add_extension('./libcommon/adblockpluschrome-1.10.0.1526.crx')
            chrome_options.add_experimental_option("prefs", prefs)
            chrome_options.add_argument("user-agent={0}".format(random.choice(USER_AGENTS)))
            self.driver_ = webdriver.Chrome(executable_path='./libcommon/chromedriver', chrome_options=chrome_options)
        elif self.browser == 'Firefox':
            profile = webdriver.FirefoxProfile()
            profile.set_preference("general.useragent.override", random.choice(USER_AGENTS))
            profile.add_extension('./libcommon/adblock_plus-2.7.1-sm+tb+an+fx.xpi')
            profile.set_preference('permissions.default.image', 2)
            profile.set_preference('dom.ipc.plugins.enabled.libflashplayer.so', 'false')
            profile.set_preference("webdriver.load.strategy", "unstable")
            self.driver_ = webdriver.Firefox(profile)
        elif self.browser == 'PhantomJS':
            self.driver_ = webdriver.PhantomJS()
            self.driver_.set_window_size(1120, 550)

    def close(self):
        self.driver_.close()

    def return_when(self, condition, locator):
        """
        returns browser execution when condition is met
        """
        for _ in range(5):
            with suppress(Exception):
                wait = WebDriverWait(self.driver_, timeout=100, poll_frequency=0.1)
                wait.until(condition(locator))
                self.driver_.execute_script("return window.stop")
                return True
        return False

    def __getattr__(self, name):
        """
        ruby-like method missing: derive methods not implemented to attribute that
        holds selenium browser
        """
        def _missing(*args, **kwargs):
            return getattr(self.driver_, name)(*args, **kwargs)
        return _missing

Answer 1

我在调查后发现了两个问题。请原谅我的无知，因为距离我上次进入 Python/Scrapy 世界已经有一段时间了。

---

首先： 我们如何从 Selenium 获取 HTML？

根据Selenium docs，驱动程序应该有一个包含页面内容的page_source属性。

browser = Browser(browser='Chrome')
browser.get(url)
html = browser.driver_.page_source
browser.close()

您可能希望在浏览器类中将其设为函数以避免从MatchSpider 访问browser.driver_。

# class Browser
def page_source(self):
    return self.driver_.page_source 
# end class

browser.get(url)
html = browser.page_source()

---

第二： 我们如何覆盖 Scrapy 的内部网络请求？

看起来 Scrapy 试图将幕后 Web 请求与您编写的每个蜘蛛的我正在尝试解析的功能分离。 start_requests() 应该“返回一个带有第一个爬取请求的可迭代对象” 并且make_requests_from_url(url)（如果您不覆盖 start_requests()，则会调用它）接受 “一个 URL 并返回一个Request 对象”。在内部处理Spider 时，Scrapy 开始创建大量的Request 对象，这些对象将被异步执行，随后的Response 将被发送到parse(response)...Spider 实际上从不执行来自@ 的处理987654338@ 回复。

长话短说，这意味着您需要create middleware for the Scrapy Downloader 才能使用 Selenium。然后，您可以删除覆盖的 start_requests() 方法并添加 start_urls 属性。具体来说，您的 SeleniumDownloaderMiddleware 应该覆盖 process_request(request, spider) 方法以使用上述 Selenium 代码。