如何在使用scrapy框架进行抓取时排除已经抓取的网址

Question

我正在抓取一个新闻网站，它提取新闻数据并将其转储到 MongoDB。

我的蜘蛛是用以下规则定义的：

rules = [Rule(
            LinkExtractor(
                allow=["foo.tv/en/*",
                       "https://fooports.tv/*"]  # only such urls

我目前所做的是它从数据库中获取已经抓取的 url，如果在数据库中找到这些 url，则不处理这些 url，例如：

    urls_visited = get_visited_urls() # Fetches from MongoDB
    if response.url not in urls_visited:
        # do scraping here

我正在寻找的是有没有办法让蜘蛛跳过那些已经被抓取的网址。我想通过不查看那​​些已经处理过的网址来尝试减少抓取时间。我知道规则中有一个拒绝功能，但不确定在这种情况下如何使用它。

我已经包含了下载器中间件自定义类来过滤掉已经被抓取的请求：

class NewsCrawlerDownloaderMiddleware:
    # Not all methods need to be defined. If a method is not defined,
    # scrapy acts as if the downloader middleware does not modify the
    # passed objects.

    def __init__(self):
        self.urls_visited = get_visited_urls() # from database

    @classmethod
    def from_crawler(cls, crawler):
        # This method is used by Scrapy to create your spiders.
        s = cls()
        crawler.signals.connect(s.spider_opened, signal=signals.spider_opened)
        return s

    def process_request(self, request, spider):
        # Called for each request that goes through the downloader
        # middleware.

        # Must either:
        # - return None: continue processing this request
        # - or return a Response object
        # - or return a Request object
        # - or raise IgnoreRequest: process_exception() methods of
        #   installed downloader middleware will be called
        # Here we check if url has already been scraped,
        # if not process the requests
        if request.url in self.urls_visited:
            logging.info('ignoring url %s', request.url)
            raise IgnoreRequest()
        else:
            return request

我在settings.py的中间件订单

['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'news_crawler.middlewares.NewsCrawlerDownloaderMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']

但是，当它尝试抓取第一个 url 时，它给了我以下错误：

ERROR: Error downloading <GET https://arynews.tv/robots.txt>: maximum recursion depth exceeded while calling a Python object

任何想法如何正确使用我的自定义下载器中间件来过滤掉 url。

Answer 1

您可以创建一个下载器中间件，该中间件将根据您的数据库查询过滤请求。查看documentation。

在这种情况下，您需要使用 process_request(request, spider) 方法定义类并在您的设置中启用此中间件（取决于您如何启动蜘蛛 - 通过 cli 或在 python 脚本中）。

您也可以定义自己的重复过滤器，查看dupefilters.py。但这可能是一种更复杂的方法，因为您需要对scrapy有一些了解和经验。