Scrapy - 蜘蛛在产生请求后卡住并且回调函数不起作用

Question

过去 2 到 3 周我一直在使用 Scrapy，具有以下依赖项：

1. Python 2.7
2. 扭曲的 17.9.0
3. Lxml 4.1.1
4. 密码学 2.1.4

现在，我已将依赖项升级到

1. Python 3.5.2
2. 扭曲的 17.9.0
3. Lxml 4.2.1
4. 密码学 2.2.2

更新我的依赖项后，我的蜘蛛卡住了。 产生请求后卡住了，回调函数也没有启动，蜘蛛进入无限等待状态。

ScraperRequestCode：

def start_requests(self):
    driver = webdriver.Chrome()
    driver.get(**)
    urls = driver.find_elements_by_xpath("//div[contains(@class, 'media__item')]//a")
    time_list = driver.find_elements_by_xpath("//span[contains(@class, 'timestamp--time')]")
    index = 0
    for url in urls:
        url_href = str(url.get_attribute('href')).replace('http:', 'https:')
        media_url = url.find_element_by_xpath("./*").get_attribute('src')
        text = url.find_element_by_xpath("./*").get_attribute('alt')
        try:
            time = time_list[index].get_attribute('title')
            time = str(datetime.datetime.strptime(str(time).lstrip(), '%b %d, %Y %I:%M%p'))
            index = index + 1
        except IndexError:
            time = ''
            pass
        if url_href != "None":
            count = News.objects.filter(url__icontains=url_href).count()
            if count == 0:
                headers = {'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64) '
                                         'AppleWebKit/537.36 (KHTML, like Gecko)'
                                         ' Chrome/60.0.3112.101 Safari/537.36'}

                request = Request(url_href, callback=self.parse, headers=headers, dont_filter=True)
                request.meta['url'] = url_href
                request.meta['text'] = text
                request.meta['media_url'] = media_url
                request.meta['time'] = time
                yield request
    sleep(5)
    driver.close()

解析函数代码：

def parse(self, response):
    print(response, "RESPONSE===========================")
    summary_list = []
    story_url = response.meta['url']
    story_title = response.meta['text']
    media_url = response.meta['media_url']
    time = response.meta['time']
    try:
        dummay_news = News.objects.get(url=story_url)
        return
    except News.DoesNotExist:
        dummay_news = News()
        story_p_tags = response.xpath("//div[contains(@class,'story__content')]/p/text()")
        for p in story_p_tags:
            summary_list.append(p.extract())

        dummay_news.source = 'xyz'
        dummay_news.time = time
        dummay_news.title = story_title
        dummay_news.url = story_url
        dummay_news.set_summary(summary_list)
        if media_url:
            dummay_news.media_url = media_url
        dummay_news.save()

    yield dummay_news

Scraper 在此请求后卡住：

2018-05-02 13:47:18 [selenium.webdriver.remote.remote_connection] DEBUG: Finished Request
2018-05-02 13:47:18 [selenium.webdriver.remote.remote_connection] DEBUG: GET http://127.0.0.1:35091/session/657a553b268a671e8117dba401ceeee8/element/0.8764352324704219-426/attribute/alt {"id": "0.8764352324704219-426", "name": "alt", "sessionId": "657a553b268a671e8117dba401ceeee8"}
2018-05-02 13:47:18 [selenium.webdriver.remote.remote_connection] DEBUG: Finished Request
2018-05-02 13:47:18 [selenium.webdriver.remote.remote_connection] DEBUG: GET http://127.0.0.1:35091/session/657a553b268a671e8117dba401ceeee8/element/0.8764352324704219-226/attribute/title {"id": "0.8764352324704219-226", "name": "title", "sessionId": "657a553b268a671e8117dba401ceeee8"}
2018-05-02 13:47:18 [selenium.webdriver.remote.remote_connection] DEBUG: Finished Request

进一步没有回调函数被启动，并且在此错误发生后它永远卡住了。
任何帮助都非常感谢，在此先感谢。

Answer 1

您在 start_requests() 方法中执行driver.close()。但是，我认为您不应该过早关闭 selenium 驱动程序。删除此行并尝试。

Answer 2

不要在start_requests 中关闭您的驱动程序。 你可以像这样关闭并退出它

def __del__(self):
    self.driver.close()
    self.driver.quit()

你的yield dummay_news 应该在except

try:
    dummay_news = News.objects.get(url=story_url)
    return
except News.DoesNotExist:
    dummay_news = News()
    story_p_tags = response.xpath("//div[contains(@class,'story__content')]/p/text()")
    for p in story_p_tags:
        summary_list.append(p.extract())

    dummay_news.source = 'xyz'
    dummay_news.time = time
    dummay_news.title = story_title
    dummay_news.url = story_url
    dummay_news.set_summary(summary_list)
    if media_url:
        dummay_news.media_url = media_url
    dummay_news.save()

    yield dummay_news