scrapy 重定向到 127.0.0.1答案

【问题标题】：scrapy redirecting to 127.0.0.1scrapy 重定向到 127.0.0.1
【发布时间】：2016-01-23 19:25:40
【问题描述】：

新手在这里scrapy。我正在尝试从桥接网站上抓取一些基本数据，但由于某种原因，我不断被重定向回 localhost。

大多数其他网站不会发生这种情况（例如教程中的 dmoz 示例）。我的预感是我还没有设置一些东西来处理有问题的域。我的蜘蛛（几乎与教程中的蜘蛛完全相同，除了 URL 已更改）：

import scrapy

class BboSpider(scrapy.Spider):
    name = "bbo"
    allowed_domains = ["bridgebase.com"]
    start_urls = [
            "http://www.bridgebase.com/vugraph/schedule.php"
            ]

    # rules for parsing main response
    def parse(self, response):
        filename = 'test.html' 
        with open(filename, 'wb') as f:
            f.write(response.body)

我得到的错误是（相关部分）：

2016-01-23 14:21:50 [scrapy] INFO: Scrapy 1.0.4 started (bot: bbo)
2016-01-23 14:21:50 [scrapy] INFO: Optional features available: ssl, http11
2016-01-23 14:21:50 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'bbo.spiders', 'SPIDER_MODULES': ['bbo.spiders'], 'BOT_NAME': 'bbo'}
2016-01-23 14:21:50 [scrapy] INFO: Enabled extensions: CloseSpider, TelnetConsole, LogStats, CoreStats, SpiderState
2016-01-23 14:21:50 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats
2016-01-23 14:21:50 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2016-01-23 14:21:50 [scrapy] INFO: Enabled item pipelines: 
2016-01-23 14:21:50 [scrapy] INFO: Spider opened
2016-01-23 14:21:50 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2016-01-23 14:21:50 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2016-01-23 14:21:54 [scrapy] DEBUG: Redirecting (302) to <GET http://127.0.0.1> from <GET http://www.bridgebase.com/vugraph/schedule.php>
2016-01-23 14:21:54 [scrapy] DEBUG: Retrying <GET http://127.0.0.1> (failed 1 times): Connection was refused by other side: 111: Connection refused.

这可能是一个非常基本的问题，但我什至不知道从哪里开始。有没有人对从哪里开始有任何预感？

【问题讨论】：

标签： python scrapy

【解决方案1】：

您必须提供User-Agent 标头才能伪装成真正的浏览器。

您可以通过提供headers 字典，同时从start_requests() 返回scrapy.Request，直接在蜘蛛中执行此操作：

import scrapy

class BboSpider(scrapy.Spider):
    name = "bbo"
    allowed_domains = ["bridgebase.com"]

    def start_requests(self):
        yield scrapy.Request("http://www.bridgebase.com/vugraph/schedule.php", headers={
            "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.111 Safari/537.36"
        })

    # rules for parsing main response
    def parse(self, response):
        filename = 'test.html'
        with open(filename, 'wb') as f:
            f.write(response.body)

或者，您可以只设置USER_AGENT project setting。

【讨论】：