使用 scapy Request 尝试/探测 url

Question

我想知道是否可以在实际处理页面之前使用 scrapy Request 检查 url 的有效性（url 事先不知道，但可以测试它们出现的不同模式）。 失败的示例代码如下。 （为简单起见，使用了 reties 变量，测试条件也可以是 s.th like if response.code != 200 )

代码失败，因为在第二个回调 (parse_page_2) 结束时，即使发出新请求，控制也不会返回到第一个回调 (parse_page_1)，回调为 parse_page_1。 为什么会这样？ 我知道urllib2 - 基于here 的解决方案，只是检查这是否可以在scrapy 上下文中严格完成。

import scrapy

from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors import LinkExtractor
from scrapy.http import Request

class MySpider(CrawlSpider):
    name = 'alexa'
    allowed_domains = ['alexa.com']
    start_urls = ['http://www.alexa.com']
    retries = 0
    rules = (
        # Extract links matching 'category.php' (but not matching 'subsection.php')
        # and follow links from them (since no callback means follow=True by default).
        # Rule(LinkExtractor(allow=('topsites', ))),

        # Extract links matching 'item.php' and parse them with the spider's method parse_item
        Rule(LinkExtractor(allow=('topsites', )), callback='parse_page1'),
    )

    def parse_page1(self, response):
        if self.retries < 5:
            self.retries += 1
            print 'Retries in 1: ', self.retries
            return scrapy.Request("http://www.alexa.com/siteieekeknfo/google.com",
                                 meta={'dont_merge_cookies': True,
                                'dont_redirect': False,
                                "handle_httpstatus_list": [301, 302, 303, 404]},
                               callback=self.parse_page2)
        else:
            print "Finished in 1"

    def parse_page2(self, response):
        if self.retries < 5:
            self.retries += 1
            print 'Retries in 2: ', self.retries
            return scrapy.Request("http://www.alexa.com/siteieekeknfo/google.com",
                                 meta={'dont_merge_cookies': True,
                                'dont_redirect': False,
                                "handle_httpstatus_list": [301, 302, 303, 404]},
                               callback=self.parse_page1)
        else:
            print "Finished in 2"

抓取结果粘贴here。

Answer 1

递归回调似乎有效：

import random


from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors import LinkExtractor
from scrapy.http import Request

class MySpider(CrawlSpider):
    name = 'alexa'
    allowed_domains = ['alexa.com']
    start_urls = ['http://www.alexa.com']    
    rules = (
        Rule(LinkExtractor(allow=('topsites', )), callback='parse_page1'),
    )

    _retries = 0

    _random_urls = [
        'http://www.alexa.com/probablydoesnotexist',
        'http://www.alexa.com/neitherdoesthis',
        'http://www.alexa.com/siteinfo/google.com'
    ]

    def parse_page1(self, response):
        print "Got status: ", response.status
        if self._retries == 0 or response.status != 200:
            self._retries += 1
            print 'Retries in 1: ', self._retries
            return Request(random.choice(self._random_urls),
                                 meta={"handle_httpstatus_list": [301, 302, 303, 404]},
                               callback=self.parse_page1)
        else:
            print "Exiting"