Scrapy - 无法点击 URL 进行网页抓取答案

【问题标题】：Scrapy - unable to hit the URL for webscrapingScrapy - 无法点击 URL 进行网页抓取
【发布时间】：2018-09-25 22:52:48
【问题描述】：

我试图复制以下 scrapy 教程 - http://blog.florian-hopf.de/2014/07/scrapy-and-elasticsearch.html。

我在运行蜘蛛时收到以下错误跟踪-

[scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min),     scraped 0 items (at 0 items/min)
2018-04-16 14:00:41 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2018-04-16 14:00:41 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying  <GET http://www.meetup.com/robots.txt> (failed 1 times):   [<twisted.python.failure.Failure twisted.internet.error.ConnectionDone: Connection was closed cleanly.>]
2018-04-16 14:00:41 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET http://www.meetup.com/robots.txt> (failed 2 times): [<twisted.python.failure.Failure twisted.internet.error.ConnectionDone: Connection was closed cleanly.>]
2018-04-16 14:00:41 [scrapy.downloadermiddlewares.retry] DEBUG: Gave up retrying <GET http://www.meetup.com/robots.txt> (failed 3 times): [<twisted.python.failure.Failure twisted.internet.error.ConnectionDone: Connection was closed cleanly.>]
2018-04-16 14:00:41 [scrapy.downloadermiddlewares.robotstxt] ERROR: Error downloading <GET http://www.meetup.com/robots.txt>: [<twisted.python.failure.Failure twisted.internet.error.ConnectionDone: Connection was closed cleanly.>]
Traceback (most recent call last):
  File "D:\anaconda\lib\site-packages\twisted\internet\defer.py", line 1384, in _inlineCallbacks
  result = result.throwExceptionIntoGenerator(g)
  File "D:\anaconda\lib\site-packages\twisted\python\failure.py", line 393, in throwExceptionIntoGenerator
  return g.throw(self.type, self.value, self.tb)
  File "D:\anaconda\lib\site-packages\scrapy\core\downloader\middleware.py", line 43, in process_request
defer.returnValue((yield download_func(request=request,spider=spider)))

谁能帮我解决/理解这个问题。

【问题讨论】：

您的系统中有代理吗

标签： python url web-scraping scrapy

【解决方案1】：

您似乎无法到达目标地址，wget http://www.meetup.com/robots.txt 的输出是什么？

当你在 meetup 升级到 https 时尝试访问 http 端点，尝试将 start_urls 更改为 https，例如：

start_urls = [
    "https://www.meetup.com/Search-Meetup-Karlsruhe/"
]

【讨论】：