Scrapy 自定义链接提取器以限制跟随的链接数量答案

【问题标题】：Scrapy custom link extractor to limit number of links followedScrapy 自定义链接提取器以限制跟随的链接数量
【发布时间】：2019-08-30 10:50:21
【问题描述】：

我正在尝试基于 Scrapy 的 LxmlLinkExtractor 编写自定义链接提取器。这个想法是包含一个maxpages 参数，以在达到限制后停止跟踪该域的链接（并继续下一个）。但是，我无法让我的自定义链接提取器工作：

from scrapy.linkextractors.lxmlhtml import *

class LimitedLinkExtractor(FilteringLinkExtractor):

    def __init__(self, allow=(), deny=(), allow_domains=(), deny_domains=(), restrict_xpaths=(),
                 tags=('a', 'area'), attrs=('href',), canonicalize=False,
                 unique=True, process_value=None, deny_extensions=None, restrict_css=(),
                 strip=True, maxpages=10): #added maxpages 

        tags, attrs = set(arg_to_iter(tags)), set(arg_to_iter(attrs))
        tag_func = lambda x: x in tags
        attr_func = lambda x: x in attrs

        lx = LxmlParserLinkExtractor(
            tag=tag_func,
            attr=attr_func,
            unique=unique,
            process=process_value,
            strip=strip,
            canonicalized=canonicalize,
        )

        super(FilteringLinkExtractor, self).__init__(lx, allow=allow, deny=deny,
            allow_domains=allow_domains, deny_domains=deny_domains,
            restrict_xpaths=restrict_xpaths, restrict_css=restrict_css,
            canonicalize=canonicalize, deny_extensions=deny_extensions,maxpages=maxpages) #added maxpages 

    def extract_links(self, response):
        base_url = get_base_url(response)
        if self.restrict_xpaths:
            docs = [subdoc
                    for x in self.restrict_xpaths
                    for subdoc in response.xpath(x)]
        else:
            docs = [response.selector]

        all_links = []
        for doc in docs:
            links = self._extract_links(doc, response.url, response.encoding, base_url)
            links = links[0:self.max_pages] #added maxpages 
            all_links.extend(self._process_links(links))
        return unique_list(all_links)

除了我评论的#added maxpages 之外的所有内容都与Scrapy 在lxmlhtml.py 中默认提供的LxmlLinkExtractor 相同。我得到的错误是：

"TypeError: object.__init__() takes exactly one argument (the instance to initialize)"

【问题讨论】：

标签： python web-scraping scrapy web-crawler

【解决方案1】：

super(FilteringLinkExtractor, self) → super(LimitedLinkExtractor, self)

【讨论】：