scrapy 无法关注完整链接答案

【问题标题】：scrapy can not follow full linkscrapy 无法关注完整链接
【发布时间】：2019-02-19 19:12:00
【问题描述】：

scrapy shell ""https://www.winemag.com/wine-ratings/2/"
response

不管怎样

2019-02-19 14:16:35 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023 2019-02-19 14:16:35 [scrapy.core.engine] INFO: Spider opened 2019-02-19 14:16:35 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.winemag.com/robots.txt> (referer: None) 2019-02-19 14:16:35 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET http://www.winemag.com/wine-ratings> from <GET https://www.winemag.com/wine-ratings/2/> 2019-02-19 14:16:35 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET https://www.winemag.com/wine-ratings> from <GET http://www.winemag.com/wine-ratings> 2019-02-19 14:16:35 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET https://www.winemag.com/wine-ratings/> from <GET https://www.winemag.com/wine-ratings> 2019-02-19 14:16:35 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.winemag.com/wine-ratings/> (referer: None)

<200 https://www.winemag.com/wine-ratings/>

我不知道为什么它没有获得完整的链接，请有人给我一个建议。

【问题讨论】：

标签： python web-scraping scrapy scrapy-shell

【解决方案1】：

似乎winemag 将爬虫重定向到其主页：

⇾ curl -I 'https://www.winemag.com/wine-ratings/2/'
HTTP/2 301
[...]
location: http://www.winemag.com/wine-ratings
[...]

所以这似乎是scrapy 的预期行为，它遵循您正在访问的网站返回给它的重定向？

【讨论】：

谢谢，这是正确的，网站正在重定向我，因为我错过了通过设置 USER_AGENT 来识别自己。

【解决方案2】：

我找到了答案。我必须在设置文件中指定 USER_AGENT。

【讨论】：