为什么我的 spider_idle / on-demand / URL-feeding 就像逐渐关闭一样？答案

【问题标题】：Why is my spider_idle / on-demand / URL-feeding like gradually shutting down?为什么我的 spider_idle / on-demand / URL-feeding 就像逐渐关闭一样？
【发布时间】：2019-01-31 21:21:09
【问题描述】：

我设置了一个spider_idle 信号来向蜘蛛提供另一批网址。然而，这似乎在一开始工作得很好，但随后Crawled (200)... 消息出现越来越少，最终停止出现。我有 115 个要分发的测试 URL，正如 Scrapy 所说的那样，Crawled 38 pages... 是其中的一部分。下面是蜘蛛和scrapy日志的代码。

一般来说，我正在实现两阶段抓取，第一遍仅将网址下载到urls.jl 文件，第二遍是对这些网址执行抓取。我现在正在编写第二个蜘蛛的编码。

import json
import scrapy
import logging
from scrapy import signals
from scrapy.http.request import Request
from scrapy.exceptions import DontCloseSpider


class A2ndexample_comSpider(scrapy.Spider):
    name = '2nd_example_com'
    allowed_domains = ['www.example.com']

    def parse(self, response):
        pass

    @classmethod
    def from_crawler(cls, crawler, *args, **kwargs):
        spider = cls(crawler, *args, **kwargs)
        crawler.signals.connect(spider.idle_consume, signals.spider_idle)
        return spider

    def __init__(self, crawler):
        self.crawler = crawler
        # read from file
        self.urls = []

        with open('urls.jl', 'r') as f:
            for line in f:
                self.urls.append(json.loads(line))
        # How many urls to return from start_requests()
        self.batch_size = 5

    def start_requests(self):
        for i in range(self.batch_size):
            if 0 == len(self.urls):
                return
            url = self.urls.pop(0)
            yield Request(url["URL"])

    def idle_consume(self):
        # Everytime spider is about to close check our urls 
        # buffer if we have something left to crawl
        reqs = self.start_requests()
        if not reqs:
            return
        logging.info('Consuming batch... [left: %d])' % len(self.urls))
        for req in reqs:
            self.crawler.engine.schedule(req, self)
        raise DontCloseSpider

日志：

INFO: Spider opened
INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
DEBUG: Telnet console listening on 127.0.0.1:6023
DEBUG: Crawled (200) <GET https://www.example.com/robots.txt> (referer: None)
DEBUG: Crawled (200) <GET https://www.example.com/oferta/mieszkanie-140-m-wroclaw-ID3EMF6.html> (referer: None)
DEBUG: Crawled (200) <GET https://www.example.com/wynajem/mieszkanie/dolnoslaskie/?nrAdsPerPage=72&search[order]=filter_float_price%3Adesc> (referer: None)
DEBUG: Crawled (200) <GET https://www.example.com/oferta/wynajem-obiekt-5-mieszkan-dla-firmy-legnica-ID3Khvk.html> (referer: None)
DEBUG: Crawled (200) <GET https://www.example.com/oferta/komfortowy-apartament-sky-tower-41-pietro-ID3ytn1.html> (referer: None)
DEBUG: Crawled (200) <GET https://www.example.com/oferta/apartament-z-przepieknym-widokiem-z-45-pietra-ID3PWvI.html> (referer: None)
INFO: Consuming batch... [left: 110])
DEBUG: Crawled (200) <GET https://www.example.com/oferta/mieszkanie-139-04-m-wroclaw-ID3A6dp.html> (referer: None)
DEBUG: Crawled (200) <GET https://www.example.com/oferta/centrum-willowy-lokal-dostepny-dla-firmy-ID3TgV4.html> (referer: None)
DEBUG: Crawled (200) <GET https://www.example.com/oferta/wynajem-pietro-na-16-osob-legnica-ID3KcPe.html> (referer: None)
DEBUG: Crawled (200) <GET https://www.example.com/wynajem/mieszkanie/dolnoslaskie/?nrAdsPerPage=72&search%5Border%5D=filter_float_price%3Adesc&page=2> (referer: None)
DEBUG: Crawled (200) <GET https://www.example.com/oferta/apartament-trzypokojowy-na-44-pietrze-sky-tower-ID3qXA8.html> (referer: None)
INFO: Consuming batch... [left: 105])
DEBUG: Filtered duplicate request: <GET https://www.example.com/wynajem/mieszkanie/dolnoslaskie/?nrAdsPerPage=72&search[order]=filter_float_price%3Adesc> - no more duplicates will be shown (see DUPEFILTER_DEBUG to show all duplicates)
DEBUG: Crawled (200) <GET https://www.example.com/oferta/mieszkanie-3-pokoje-ul-zatorska-wysoki-standard-ID3GBfa.html> (referer: None)
DEBUG: Crawled (200) <GET https://www.example.com/oferta/nowe-mieszkanie-2-pokoje-wroclaw-ul-gornicza-ID2NeJT.html> (referer: None)
DEBUG: Crawled (200) <GET https://www.example.com/oferta/sprzedam-mieszkanie-bezczynszowe-gromadka-ID3S1sA.html> (referer: None)
DEBUG: Crawled (200) <GET https://www.example.com/oferta/mieszkanie-ID3ALrp.html> (referer: None)
INFO: Consuming batch... [left: 100])
DEBUG: Crawled (200) <GET https://www.example.com/oferta/2-pok-balkonosobna-kuchniawindado-urzadzenia-ID3Scza.html> (referer: None)
DEBUG: Crawled (200) <GET https://www.example.com/oferta/mieszkanie-47-m-wroclaw-ID3RTOY.html> (referer: None)
INFO: Consuming batch... [left: 95])
INFO: Consuming batch... [left: 90])
DEBUG: Crawled (200) <GET https://www.example.com/oferta/luksusowy-apartament-101m2-centrum-obok-renomy-ID3O1yI.html> (referer: None)
DEBUG: Crawled (200) <GET https://www.example.com/oferta/mieszkanie-70-m-wroclaw-ID3SS4A.html> (referer: None)
INFO: Consuming batch... [left: 85])
INFO: Consuming batch... [left: 80])
INFO: Consuming batch... [left: 75])
DEBUG: Crawled (200) <GET https://www.example.com/oferta/mieszkanie-103-m-wroclaw-ID2ZhbS.html> (referer: None)
DEBUG: Crawled (200) <GET https://www.example.com/wynajem/mieszkanie/dolnoslaskie/?nrAdsPerPage=72&search%5Border%5D=filter_float_price%3Adesc&page=3> (referer: None)
DEBUG: Crawled (200) <GET https://www.example.com/oferta/luksusowe-przestronne-dwa-garaze-ID3LwIs.html> (referer: None)
INFO: Consuming batch... [left: 70])
DEBUG: Crawled (200) <GET https://www.example.com/oferta/mieszkanie-118-74-m-wroclaw-ID2W9Fd.html> (referer: None)
DEBUG: Crawled (200) <GET https://www.example.com/oferta/ekskluzywny-apartament-z-dostepem-do-silowni-i-spa-ID3pGmQ.html> (referer: None)
DEBUG: Crawled (200) <GET https://www.example.com/oferta/mieszkanie-170-m-wroclaw-ID3MBI0.html> (referer: None)
INFO: Consuming batch... [left: 65])
INFO: Crawled 25 pages (at 25 pages/min), scraped 0 items (at 0 items/min)
DEBUG: Crawled (200) <GET https://www.example.com/oferta/loftowe-mieszkanie-krzyki-100-m2-ID3Tfc0.html> (referer: None)
DEBUG: Crawled (200) <GET https://www.example.com/oferta/nieruchompsc-dla-pracownikow-od-zaraz-ID3TrcA.html> (referer: None)
DEBUG: Crawled (200) <GET https://www.example.com/oferta/stare-miasto-3-pok-69m2-luxurious-apartment-ID3Qn4o.html> (referer: None)
DEBUG: Crawled (200) <GET https://www.example.com/oferta/loftowe-100-metrowe-mieszkanie-idealne-na-biuro-ID3Txu4.html> (referer: None)
INFO: Consuming batch... [left: 60])
DEBUG: Crawled (200) <GET https://www.example.com/oferta/lesnica-ul-niepierzynska-123-m2-6-pokoi-ogrod-ID3OoI8.html> (referer: None)
DEBUG: Crawled (200) <GET https://www.example.com/oferta/mieszkanie-63-m-wroclaw-ID3Tbne.html> (referer: None)
DEBUG: Crawled (200) <GET https://www.example.com/wynajem/mieszkanie/dolnoslaskie/?nrAdsPerPage=72&search%5Border%5D=filter_float_price%3Adesc&page=4> (referer: None)
DEBUG: Crawled (200) <GET https://www.example.com/oferta/komfortow-apartament-do-wynajecia-3km-od-wroclawia-ID3SA0M.html> (referer: None)
INFO: Consuming batch... [left: 55])
DEBUG: Crawled (200) <GET https://www.example.com/oferta/zamienie-mieszanie-2-pokoje-40m2-bielawa-na-wieksz-ID3yyFN.html> (referer: None)
DEBUG: Crawled (200) <GET https://www.example.com/oferta/apartament-sky-tower-z-wanna-przy-oknie-i-sauna-ID2Z7EA.html> (referer: None)
INFO: Consuming batch... [left: 50])
INFO: Consuming batch... [left: 45])
DEBUG: Crawled (200) <GET https://www.example.com/oferta/ul-ksiecia-witolda-3pok-75m2-wysoki-standard-3700-ID3PK2g.html> (referer: None)
DEBUG: Crawled (200) <GET https://www.example.com/oferta/komfortowe-mieszkanie-do-wynajecia-ID3Lcvk.html> (referer: None)
INFO: Consuming batch... [left: 40])
INFO: Consuming batch... [left: 35])
INFO: Consuming batch... [left: 30])
DEBUG: Crawled (200) <GET https://www.example.com/oferta/hit-klimatyczne-w-sercu-wroclawia-2-pok-ID3SkJ2.html> (referer: None)
INFO: Consuming batch... [left: 25])
INFO: Consuming batch... [left: 20])
INFO: Consuming batch... [left: 15])
INFO: Consuming batch... [left: 10])
INFO: Crawled 38 pages (at 13 pages/min), scraped 0 items (at 0 items/min)
INFO: Consuming batch... [left: 5])
INFO: Consuming batch... [left: 0])
INFO: Consuming batch... [left: 0])
INFO: Consuming batch... [left: 0])
INFO: Consuming batch... [left: 0])
(...)
INFO: Consuming batch... [left: 0])
INFO: Consuming batch... [left: 0])
INFO: Consuming batch... [left: 0])
INFO: Crawled 38 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
INFO: Consuming batch... [left: 0])
INFO: Consuming batch... [left: 0])
INFO: Consuming batch... [left: 0])
...

我预计蜘蛛会抓取所有 115 个 URL，而不仅仅是 38 个。另外，如果它不想再抓取，并且 singal-handler 函数不会引发 DontCloseSpider，那么不应该至少它会关闭吗？

【问题讨论】：

标签： python scrapy signals

【解决方案1】：

丢失的请求没有失败，否则您也会在日志中看到有关它的信息。它们根本没有被发送。

如果你仔细观察你的日志，你会注意到这条消息：

DEBUG: Filtered duplicate request: <GET https://www.example.com/wynajem/mieszkanie/dolnoslaskie/?nrAdsPerPage=72&search[order]=filter_float_price%3Adesc> - no more duplicates will be shown (see DUPEFILTER_DEBUG to show all duplicates)

缺少的请求被跳过，因为它们被认为是重复的。有关详细信息，请参阅DUPEFILTER_CLASS 设置的文档。

【讨论】：

谢谢，就是这样！ TIL：uniq -d 首先需要 sort |。