编写scrapy规则答案

【问题标题】：Writing scrapy rules编写scrapy规则
【发布时间】：2014-01-13 00:25:26
【问题描述】：

我不明白 scrapy 规则是如何工作的。假设我想抓取一个网站，我希望它通过包含“类别”的链接。我想打开包含“产品”的网址，然后将其传递给回调。这个怎么写？

这有什么问题？

rules = (  
           Rule(SgmlLinkExtractor(allow=r'.*?categoryId.*'), follow=True),
           Rule(SgmlLinkExtractor(allow=r'.*?productId.*'), callback='parse_item'),
        )

我收到以下错误：

    Traceback (most recent call last):
      File "/home/scraper/.fakeroot/lib/python2.7/site-packages/twisted/internet/base.py", line 824, in runUntilCurrent
        call.func(*call.args, **call.kw)
      File "/home/scraper/.fakeroot/lib/python2.7/site-packages/twisted/internet/task.py", line 638, in _tick
        taskObj._oneWorkUnit()
      File "/home/scraper/.fakeroot/lib/python2.7/site-packages/twisted/internet/task.py", line 484, in _oneWorkUnit
        result = next(self._iterator)
      File "/home/scraper/.fakeroot/lib/python2.7/site-packages/scrapy/utils/defer.py", line 57, in <genexpr>
        work = (callable(elem, *args, **named) for elem in iterable)
    --- <exception caught here> ---
      File "/home/scraper/.fakeroot/lib/python2.7/site-packages/scrapy/utils/defer.py", line 96, in iter_errback
        yield next(it)
      File "/home/scraper/.fakeroot/lib/python2.7/site-packages/scrapy/contrib/spidermiddleware/offsite.py", line 23, in process_spider_output
        for x in result:
      File "/home/scraper/.fakeroot/lib/python2.7/site-packages/scrapy/contrib/spidermiddleware/referer.py", line 22, in <genexpr>
        return (_set_referer(r) for r in result or ())
      File "/home/scraper/.fakeroot/lib/python2.7/site-packages/scrapy/contrib/spidermiddleware/urllength.py", line 33, in <genexpr>
        return (r for r in result or () if _filter(r))
      File "/home/scraper/.fakeroot/lib/python2.7/site-packages/scrapy/contrib/spidermiddleware/depth.py", line 50, in <genexpr>
        return (r for r in result or () if _filter(r))
      File "/home/scraper/.fakeroot/lib/python2.7/site-packages/scrapy/contrib/spiders/crawl.py", line 73, in _parse_response
        for request_or_item in self._requests_to_follow(response):
      File "/home/scraper/.fakeroot/lib/python2.7/site-packages/scrapy/contrib/spiders/crawl.py", line 52, in _requests_to_follow
        links = [l for l in rule.link_extractor.extract_links(response) if l not in seen]
      File "/home/scraper/.fakeroot/lib/python2.7/site-packages/scrapy/contrib/linkextractors/sgml.py", line 128, in extract_links
        links = self._extract_links(body, response.url, response.encoding, base_url)
      File "/home/scraper/.fakeroot/lib/python2.7/site-packages/scrapy/contrib/linkextractors/sgml.py", line 29, in _extract_links
        self.feed(response_text)
      File "/home/scraper/.fakeroot/lib/python2.7/sgmllib.py", line 104, in feed
        self.goahead(0)
      File "/home/scraper/.fakeroot/lib/python2.7/sgmllib.py", line 174, in goahead
        k = self.parse_declaration(i)
      File "/home/scraper/.fakeroot/lib/python2.7/markupbase.py", line 140, in parse_declaration
        "unexpected %r char in declaration" % rawdata[j])
      File "/home/scraper/.fakeroot/lib/python2.7/sgmllib.py", line 111, in error
        raise SGMLParseError(message)
    sgmllib.SGMLParseError: unexpected '=' char in declaration

【问题讨论】：

看起来不错。除了第二个follow=True 之外，只有当您想关注“产品”页面中的“类别”链接时才需要。您可以取出第一个 follow=True，因为默认行为是在未定义回调时跟随链接。您是否遇到过此规则的任何问题？
sgmllib.SGMLParseError：声明中出现意外的'='字符
更新了完整的回溯。
我认为这与 HTML 响应有关，而不是您声明抓取规则的方式。它们看起来不错。
我该如何解决这个问题？当我在另一个网站上尝试这些规则时，它确实有效。所以，这个网站发生了一些奇怪的事情。

标签： python scrapy

【解决方案1】：

试试这个：

rules = (
       Rule(SgmlLinkExtractor(allow=(r'.*?categoryId.*',)), follow=True),
       Rule(SgmlLinkExtractor(allow=(r'.*?productId.*',)), callback='parse_item'),
    )

【讨论】：