Scrapy KeyError - 找不到蜘蛛

Question

我有一个无法从命令行使用scrapy crawl <spider-name> 执行的scrapy 项目。

我刚刚转移到 ubuntu 16.04 上的新开发环境，所以我想再次检查问题是否与我的设置有关。为此，我使用python 2.7.12 使用virtualenvwrapper 创建了一个干净的虚拟环境，并遵循Scrapy 文档中的tutorial instructions。 （v1.1 模仿我的其他项目）。

尽管使用了新环境，但我仍然看到 scrapy 的奇怪行为，但它不会：

• 列出蜘蛛scrapy list
• 列出设置scrapy settings
• 开始爬网scrapy crawl quotes

scrapy crawl quotes 引发以下错误：

2017-01-06 14:20:50 [scrapy] INFO: Scrapy 1.1.1 started (bot: scrapybot)
2017-01-06 14:20:50 [scrapy] INFO: Overridden settings: {}
Traceback (most recent call last):
  File "/home/alan/QueryClick/.virtualenvs/test/bin/scrapy", line 11, in <module>
    sys.exit(execute())
  File "/home/alan/QueryClick/.virtualenvs/test/local/lib/python2.7/site-packages/scrapy/cmdline.py", line 142, in execute
    _run_print_help(parser, _run_command, cmd, args, opts)
  File "/home/alan/QueryClick/.virtualenvs/test/local/lib/python2.7/site-packages/scrapy/cmdline.py", line 88, in _run_print_help
    func(*a, **kw)
  File "/home/alan/QueryClick/.virtualenvs/test/local/lib/python2.7/site-packages/scrapy/cmdline.py", line 149, in _run_command
    cmd.run(args, opts)
  File "/home/alan/QueryClick/.virtualenvs/test/local/lib/python2.7/site-packages/scrapy/commands/crawl.py", line 57, in run
    self.crawler_process.crawl(spname, **opts.spargs)
  File "/home/alan/QueryClick/.virtualenvs/test/local/lib/python2.7/site-packages/scrapy/crawler.py", line 162, in crawl
    crawler = self.create_crawler(crawler_or_spidercls)
  File "/home/alan/QueryClick/.virtualenvs/test/local/lib/python2.7/site-packages/scrapy/crawler.py", line 190, in create_crawler
    return self._create_crawler(crawler_or_spidercls)
  File "/home/alan/QueryClick/.virtualenvs/test/local/lib/python2.7/site-packages/scrapy/crawler.py", line 194, in _create_crawler
    spidercls = self.spider_loader.load(spidercls)
  File "/home/alan/QueryClick/.virtualenvs/test/local/lib/python2.7/site-packages/scrapy/spiderloader.py", line 43, in load
    raise KeyError("Spider not found: {}".format(spider_name))
KeyError: 'Spider not found: quotes'

我的目录结构是：

└── tutorial
    ├── scrapy.cfg
    └── tutorial
        ├── __init__.py
        ├── items.py
        ├── pipelines.py
        ├── settings.py
        └── spiders
            ├── __init__.py
            └── quote_spider.py

我还仔细检查了 Scrapy 文档中指定的所有系统要求。这个问题在我的团队中被复制，在 Ubuntu 14.04 上，遵循与 virtualenv 相同的步骤和与我相同的设置。

如果有人能对此有所了解，我将永远感激不尽。

---

编辑：添加settings.py

settings.py 中唯一活跃的是：

BOT_NAME = 'tutorial'
SPIDER_MODULES = ['tutorial.spiders']
NEWSPIDER_MODULE = 'tutorial.spiders'
ROBOTSTXT_OBEY = True

---

编辑：分享scrapy.cfg

# Automatically created by: scrapy startproject
#
# For more information about the [deploy] section see:
# https://scrapyd.readthedocs.org/en/latest/deploy.html

[settings]
default = tutorial.settings

[deploy]
#url = http://localhost:6800/
project = tutorial

Answer 1

这与为调用蜘蛛的 django 项目设置的某些环境变量发生冲突 - 它们的命名空间为 SCRAPY_ 并且一定存在冲突。

编辑：供参考：GitHub issue on undocumented environment variable(s)

Answer 2

确保您的蜘蛛是 .py，例如，如果从 jupyter 创建，它可能是 .ipynb，它不会被拾取。