Scrapy：无法安排

Question

我想每隔几分钟运行一次蜘蛛。我将以下脚本放入我想要为此目的调用的项目中。

import schedule, os
from scrapy.crawler import CrawlerProcess
from scrapy.utils.project import get_project_settings

def job():
      process = CrawlerProcess(get_project_settings())
      process.crawl('amazon_spider')
      process.start() # error: twisted.internet.error.ReactorNotRestartable 
      #process.start(stop_after_crawl=False)  #process get stuck

while True:
     schedule.run_pending()
     schedule.every().minutes.do(job)

使用此方法，该过程会出现以下错误：

twisted.internet.error.ReactorNotRestartable 或者如果我把 process.start(stop_after_crawl=False) 卡住了>

从之前发布的 stackoverflow 中，我也尝试了这个：

from twisted.internet import reactor
from amazon.spiders.amazon_spider import AmazonSpider
from scrapy.crawler import CrawlerRunner

def run_crawl():

    runner = CrawlerRunner({
    'USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)',
    })
     deferred = runner.crawl(AmazonSpider)
     deferred.addCallback(reactor.callLater, 10, run_crawl)

     return deferred

     run_crawl()
     reactor.run()   

进程再次卡在解析函数的中间。我真的不知道下一步该尝试什么。如果您有任何想法，请告诉我。提前谢谢你....（顺便说一句，它不是重复的，因为同一主题的帖子没有解决我的问题。

Answer 1

我用的是调度器

pip install apscheduler

然后

# -*- coding: utf-8 -*-
from scrapy.crawler import CrawlerProcess
from scrapy.utils.project import get_project_settings
from apscheduler.schedulers.twisted import TwistedScheduler

from Demo.spiders.baidu import YourSpider

process = CrawlerProcess(get_project_settings())
scheduler = TwistedScheduler()
scheduler.add_job(process.crawl, 'interval', args=[YourSpider], seconds=10)
scheduler.start()
process.start(False)