使用scrapy-Djangoitem时芹菜无法启动工作进程答案

【问题标题】：Celery could not start worker processes when using scrapy-Djangoitem使用scrapy-Djangoitem时芹菜无法启动工作进程
【发布时间】：2017-04-14 06:41:27
【问题描述】：

问题详情：https://github.com/celery/celery/issues/3598

我想用 celery 运行一个 scrapy 蜘蛛，其中包含 Djangoitems。

这是我的芹菜任务：

# coding_task.py
import sys

from celery import Celery
from collector.collector.crawl_agent import crawl

app = Celery('coding.net', backend='redis', broker='redis://localhost:6379/0')
app.config_from_object('celery_config')


@app.task
def period_task():
    crawl()

collector.collector.crawl_agent.crawl 包含一个使用 djangoitem 作为项目的爬虫。像这样的项目：

import django

os.environ['DJANGO_SETTINGS_MODULE'] = 'RaPo3.settings'
django.setup()

from scrapy_djangoitem import DjangoItem
from xxx.models import Collection


class CodingItem(DjangoItem):
    django_model = Collection
    amount = scrapy.Field(default=0)
    role = scrapy.Field()
    type = scrapy.Field()
    duration = scrapy.Field()
    detail = scrapy.Field()
    extra = scrapy.Field()

运行时：celery -A coding_task worker --loglevel=info --concurrency=1，会出现以下错误：

[2016-11-16 17:33:41,934: ERROR/Worker-1] Process Worker-1
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/billiard/process.py", line 292, in _bootstrap
    self.run()
  File "/usr/local/lib/python2.7/site-packages/billiard/pool.py", line 292, in run
    self.after_fork()
  File "/usr/local/lib/python2.7/site-packages/billiard/pool.py", line 395, in after_fork
    self.initializer(*self.initargs)
  File "/usr/local/lib/python2.7/site-packages/celery/concurrency/prefork.py", line 80, in process_initializer
    signals.worker_process_init.send(sender=None)
  File "/usr/local/lib/python2.7/site-packages/celery/utils/dispatch/signal.py", line 151, in send
    response = receiver(signal=self, sender=sender, **named)
  File "/usr/local/lib/python2.7/site-packages/celery/fixups/django.py", line 152, in on_worker_process_init
    self._close_database()
  File "/usr/local/lib/python2.7/site-packages/celery/fixups/django.py", line 181, in _close_database
    funs = [self._db.close_connection]  # pre multidb
AttributeError: 'module' object has no attribute 'close_connection'
[2016-11-16 17:33:41,942: INFO/MainProcess] Connected to redis://localhost:6379/0
[2016-11-16 17:33:41,957: INFO/MainProcess] mingle: searching for neighbors
[2016-11-16 17:33:42,962: INFO/MainProcess] mingle: all alone
/usr/local/lib/python2.7/site-packages/celery/fixups/django.py:199: UserWarning: Using settings.DEBUG leads to a memory leak, never use this setting in production environments!
  warnings.warn('Using settings.DEBUG leads to a memory leak, never '

[2016-11-16 17:33:42,968: WARNING/MainProcess] /usr/local/lib/python2.7/site-packages/celery/fixups/django.py:199: UserWarning: Using settings.DEBUG leads to a memory leak, never use this setting in production environments!
  warnings.warn('Using settings.DEBUG leads to a memory leak, never '

[2016-11-16 17:33:42,968: WARNING/MainProcess] celery@MacBook-Pro.local ready.
[2016-11-16 17:33:42,969: ERROR/MainProcess] Process 'Worker-1' pid:2777 exited with 'exitcode 1'
[2016-11-16 17:33:42,991: ERROR/MainProcess] Unrecoverable error: WorkerLostError('Could not start worker processes',)
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/celery/worker/__init__.py", line 208, in start
    self.blueprint.start(self)
  File "/usr/local/lib/python2.7/site-packages/celery/bootsteps.py", line 127, in start
    step.start(parent)
  File "/usr/local/lib/python2.7/site-packages/celery/bootsteps.py", line 378, in start
    return self.obj.start()
  File "/usr/local/lib/python2.7/site-packages/celery/worker/consumer.py", line 271, in start
    blueprint.start(self)
  File "/usr/local/lib/python2.7/site-packages/celery/bootsteps.py", line 127, in start
    step.start(parent)
  File "/usr/local/lib/python2.7/site-packages/celery/worker/consumer.py", line 766, in start
    c.loop(*c.loop_args())
  File "/usr/local/lib/python2.7/site-packages/celery/worker/loops.py", line 50, in asynloop
    raise WorkerLostError('Could not start worker processes')
WorkerLostError: Could not start worker processes

如果我删除项目中的 djangoitem：

from scrapy.item import Item
class CodingItem(item):
    amount = scrapy.Field(default=0)
    role = scrapy.Field()
    type = scrapy.Field()
    duration = scrapy.Field()
    detail = scrapy.Field()
    extra = scrapy.Field()

该任务将运行良好并且没有任何错误。如果我想在这个 celery-scrapy 任务中使用 djangoitem 该怎么办？

谢谢！

【问题讨论】：

标签： python django scrapy rabbitmq celery

【解决方案1】：

升级 Celery 到 4.0 即可解决问题。

【讨论】：

【解决方案2】：

您应该检查内存使用情况。芹菜可能没有足够的内存

【讨论】：