twisted.internet.error.ReactorAlreadyInstalledError：反应器已安装

Question

当我多次运行爬网过程时出现此错误。 我正在使用scrapy 2.6 这是我的代码：

from scrapy.crawler import CrawlerProcess
from football.spiders.laliga import LaligaSpider
from scrapy.utils.project import get_project_settings

process = CrawlerProcess(settings=get_project_settings())
for i in range(1, 29):
    process.crawl(LaligaSpider, **{'week': i})
process.start()

Answer 1

对我来说这很有效，我把它放在 CrawlerProcess 之前 import sys if "twisted.internet.reactor" in sys.modules: del sys.modules["twisted.internet.reactor"]

Answer 2

此解决方案避免使用文档中所述的 CrawlerProcess。 https://docs.scrapy.org/en/latest/topics/practices.html#run-scrapy-from-a-script

还有另一个 Scrapy 实用程序可以更好地控制抓取过程：scrapy.crawler.CrawlerRunner。这个类是一个瘦包装器，它封装了一些简单的帮助器来运行多个爬虫，但它不会以任何方式启动或干扰现有的反应器。

如果您的应用程序已经在使用 Twisted 并且您想在同一个反应器中运行 Scrapy，建议您使用 CrawlerRunner 而不是 CrawlerProcess。

from twisted.internet import reactor
from scrapy.crawler import CrawlerRunner
from scrapy.utils.project import get_project_settings
from scrapy.utils.log import configure_logging

from football.spiders.laliga import LaligaSpider

# Enable logging for CrawlerRunner
configure_logging()

runner = CrawlerRunner(settings=get_project_settings())
for i in range(1, 29):
    runner.crawl(LaligaSpider, **{'week': i})

deferred = runner.join()
deferred.addBoth(lambda _: reactor.stop())

reactor.run()  # the script will block here until all crawling jobs are finished

Answer 3

我也遇到了这个问题。 https://docs.scrapy.org/en/latest/topics/practices.html 的文档似乎不正确地指出 CrawlerProcess 可用于运行由蜘蛛构建的多个爬虫，因为如果你给它一个蜘蛛，每个新的爬虫都会尝试加载一个新的反应器实例。我可以通过使用 CrawlerRunner 来让我的代码正常工作，在同一页面上也有详细说明。

import scrapy
from twisted.internet import reactor
from scrapy.crawler import CrawlerRunner
from scrapy.utils.log import configure_logging
from scrapy.utils.project import get_project_settings

class MySpider1(scrapy.Spider):
    # Your first spider definition
    ...

class MySpider2(scrapy.Spider):
    # Your second spider definition
    ...

configure_logging()
settings = get_project_settings() # settings not required if running
runner = CrawlerRunner(settings)  # from script, defaults provided
runner.crawl(MySpider1) # your loop would go here
runner.crawl(MySpider2)
d = runner.join()
d.addBoth(lambda _: reactor.stop())
reactor.run() # the script will block here until all crawling jobs are finished