Scrapy忽略robots.txt而不解析页面

Question

我正在尝试以与我在one of the answers 中找到的类似的方式抓取ACRIS（查看更新答案）。我稍微修改了代码以删除不推荐使用的内容。

起初我遇到robots.txt 限制我的问题，所以我发现我可以在settings.py 中设置ROBOTSTXT_OBEY=False，实际上它似乎忽略了它，但由于某种原因蜘蛛没有到达parse方法了。

这是我的蜘蛛

from scrapy.http import Request, FormRequest
from scrapy.item import Item, Field
from scrapy.spiders import Spider
import logging

class AcrisItem(Item):
    borough = Field()
    block = Field()


class AcrisSpider(Spider):
    name = "acris"
    allowed_domains = ["a836-acris.nyc.gov"]
    start_urls = ['https://a836-acris.nyc.gov/DS/DocumentSearch/PartyName']

    def start_requests(self):
        return [(Request(url, meta={'dont_redirect': True}, callback=self.parse)) for url in self.start_urls]

    def parse(self, response):
        form_token = response.selector.xpath('//input[@name="__RequestVerificationToken"]/@value').extract_first()

        logging.debug('THE FORM TOKEN IS: %s\n\n' % form_token)

        formdata = {
            "__RequestVerificationToken": form_token,
            "hid_last": "SMITH",
            "hid_first": "JOHN",
            "hid_ml": "",
            "hid_suffix": "",
            "hid_business": "",
            "hid_selectdate": "To Current Date",
            "hid_datefromm": "",
            "hid_datefromd": "",
            "hid_datefromy": "",
            "hid_datetom": "",
            "hid_datetod": "",
            "hid_datetoy": "",
            "hid_partype": "",
            "hid_borough": "All Boroughs/Counties",
            "hid_doctype": "All Document Classes",
            "hid_max_rows": "10",
            "hid_page": "1",
            "hid_partype_name": "All Parties",
            "hid_doctype_name": "All Document Classes",
            "hid_borough_name": "All Boroughs/Counties",
            "hid_ReqID": "",
            "hid_SearchType": "PARTYNAME",
            "hid_ISIntranet": "N",
            "hid_sort": ""
        }

        if form_token:
            yield FormRequest(url="https://a836-acris.nyc.gov/DS/DocumentSearch/PartyNameResult",
                              method="POST",
                              formdata=formdata,
                              meta={'dont_redirect': True},
                              callback=self.parse_page)

    def parse_page(self, response):
        rows = response.selector.xpath('//form[@name="DATA"]/table/tbody/tr[2]/td/table/tbody/tr')

        for row in rows:
            item = AcrisItem()

            borough = row.xpath('.//td[3]/div/font/text()').extract_first()
            block = row.xpath('.//td[4]/div/font/text()').extract_first()

            if borough and block:
                item['borough'] = borough
                item['block'] = block

                yield item

这是输出（减去初始化消息）

2017-01-04 17:06:12 [scrapy.core.engine] INFO: Spider opened
2017-01-04 17:06:12 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2017-01-04 17:06:12 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2017-01-04 17:06:41 [scrapy.core.engine] DEBUG: Crawled (301) <GET https://a836-acris.nyc.gov/DS/DocumentSearch/PartyName> (referer: None)
2017-01-04 17:06:41 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <301 https://a836-acris.nyc.gov/DS/DocumentSearch/PartyName>: HTTP status code is not handled or not allowed
2017-01-04 17:06:41 [scrapy.core.engine] INFO: Closing spider (finished)
2017-01-04 17:06:41 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 243,
 'downloader/request_count': 1,
 'downloader/request_method_count/GET': 1,
 'downloader/response_bytes': 414,
 'downloader/response_count': 1,
 'downloader/response_status_count/301': 1,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2017, 1, 4, 15, 6, 41, 791647),
 'log_count/DEBUG': 2,
 'log_count/INFO': 8,
 'response_received_count': 1,
 'scheduler/dequeued': 1,
 'scheduler/dequeued/memory': 1,
 'scheduler/enqueued': 1,
 'scheduler/enqueued/memory': 1,
 'start_time': datetime.datetime(2017, 1, 4, 15, 6, 12, 467659)}
2017-01-04 17:06:41 [scrapy.core.engine] INFO: Spider closed (finished)

Answer 1

您禁止在start_requests 上直接重定向。所以301 应该到达parse 方法，不幸的是你没有让这个响应状态码通过。

使用属性允许它：

class AcrisSpider(Spider):
    ...
    handle_httpstatus_list = [301, 302]
    ...

或在请求中传递handle_httpstatus_all=True 元参数，如：

return [(Request(url, meta={'dont_redirect': True, 'handle_httpstatus_all':True}, callback=self.parse)) for url in self.start_urls]