芹菜连接错误

Question

我正在将一个带有 Celery 的 Django 应用程序部署到 Heroku，并且遇到了连接错误。我的 AMQP 提供者声称他能够访问自己的资源，并且我有足够的连接。我认为我的设置是在我的特定工作进程中找到 celery 应用程序，但未能在 shell 中获取正确的设置。有没有办法告诉正在调用任务的连接 URL 是什么？我的 celery_app 检查是否应该正确处理事情？

问题调用：

$ heroku run bash
~$ python <app>/manage.py shell
>>> from <app>.management.tasks.tasks import <task>
>>> t = <task>()
>>> dt = '20140101'
>>> t.delay(dt=dt)

堆栈跟踪：

Traceback (most recent call last):
  File "<input>", line 4, in <module>
  File "/app/.heroku/python/lib/python2.7/site-packages/celery/app/task.py", line 453, in delay
    return self.apply_async(args, kwargs)
  File "/app/.heroku/python/lib/python2.7/site-packages/celery/app/task.py", line 555, in apply_async
    **dict(self._get_exec_options(), **options)
  File "/app/.heroku/python/lib/python2.7/site-packages/celery/app/base.py", line 353, in send_task
    reply_to=reply_to or self.oid, **options
  File "/app/.heroku/python/lib/python2.7/site-packages/celery/app/amqp.py", line 305, in publish_task
    **kwargs
  File "/app/.heroku/python/lib/python2.7/site-packages/kombu/messaging.py", line 168, in publish
    routing_key, mandatory, immediate, exchange, declare)
  File "/app/.heroku/python/lib/python2.7/site-packages/kombu/connection.py", line 457, in _ensured
    interval_max)
  File "/app/.heroku/python/lib/python2.7/site-packages/kombu/connection.py", line 369, in ensure_connection
    interval_start, interval_step, interval_max, callback)
  File "/app/.heroku/python/lib/python2.7/site-packages/kombu/utils/__init__.py", line 243, in retry_over_time
    return fun(*args, **kwargs)
  File "/app/.heroku/python/lib/python2.7/site-packages/kombu/connection.py", line 237, in connect
    return self.connection
  File "/app/.heroku/python/lib/python2.7/site-packages/kombu/connection.py", line 741, in connection
    self._connection = self._establish_connection()
  File "/app/.heroku/python/lib/python2.7/site-packages/kombu/connection.py", line 696, in _establish_connection
    conn = self.transport.establish_connection()
  File "/app/.heroku/python/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 112, in establish_connection
    conn = self.Connection(**opts)
  File "/app/.heroku/python/lib/python2.7/site-packages/amqp/connection.py", line 165, in __init__
    self.transport = create_transport(host, connect_timeout, ssl)
  File "/app/.heroku/python/lib/python2.7/site-packages/amqp/transport.py", line 294, in create_transport
    return TCPTransport(host, connect_timeout)
  File "/app/.heroku/python/lib/python2.7/site-packages/amqp/transport.py", line 95, in __init__
    raise socket.error(last_err)
error: [Errno 111] Connection refused

在 shell 中检查配置：

~$ python <app>/manage.py shell
>>> from celery_app import Config
>>> Config.BROKER_URL
'<correct amqp resource>'

环境变量：

CELERYD_CONCURRENCY=1
CELERY_IGNORE_RESULT=True
CELERYD_TASK_TIME_LIMIT=60
CLOUDAMQP_URL=<amqp url>
RESULT_EXPIRY_RATE=600
BROKER_CONNECTION_TIMEOUT=10
PWD=/app
DJANGO_SETTINGS_MODULE=settings.production
DJANGO_PROJECT_DIR=/app/<app>
BROKER_POOL_LIMIT=1
HOME=/app
PYTHONPATH=/app:/app/<app>:/app/<app>/<app>

Procfile 调用（有效）

web: gunicorn <app>.<app>.wsgi -w 1 --log-file -
worker: celery worker --app=<app>.<app> -E -Q <app>,celery --loglevel=INFO  -c 1  --workdir=<app>

我的芹菜应用：

from __future__ import absolute_import
from os import getenv
from kombu import Exchange, Queue
from django.conf import settings
from celery import Celery


app = Celery('<app>')

class Config(object):

    # List of modules to import when celery starts.
    CELERY_IMPORTS = ("<imports>",)


    BROKER_CONNECTION_RETRY = True
    API_RATE_LIMIT = getenv('API_RATE_LIMIT')
    BROKER_POOL_LIMIT = int(getenv('BROKER_POOL_LIMIT', 1))
    BROKER_URL = getenv('CLOUDAMQP_URL')
    BROKER_CONNECTION_TIMEOUT = int(getenv('BROKER_CONNECTION_TIMEOUT'))
    CELERYD_CONCURRENCY = int(getenv('CELERYD_CONCURRENCY'))


app.config_from_object(Config)

if __name__ == '__main__':
    app.start()

在同一目录中的 init.py 中导入应用程序：

from __future__ import absolute_import

# This will make sure the app is always imported when
# Django starts so that shared_task will use this app.
from .celery_app import app as celery_app  # nolint

Answer 1

核心问题是 django、celery 和 gunicorn 的 DJANGO_SETTINGS_MODULE 预期格式不同。我通过将 DJANGO_SETTINGS_MODULE=settings.production 更改为 DJANGO_SETTINGS_MODULE=<app>.settings.production 来解决此问题，这修复了 shell 代理连接但破坏了我的 web 和工作进程。一个有效的 Procfile 规范是

web: cd <app> && gunicorn <app>.wsgi -w 1 --log-file -
worker: celery worker --app=<app> -E -Q <app>,celery --loglevel=INFO  -c 1  --workdir=<app>

web 之前的 cd 很傻，但是可以。

Answer 2

我在将 Django 项目部署到 Tornado 服务器时遇到了这个异常。 这是部署代码：

import os
import tornado.httpserver
import tornado.ioloop
import tornado.wsgi
from django.core.wsgi import get_wsgi_application

# add this line when use celery.
# import app

def main():
    os.environ['DJANGO_SETTINGS_MODULE'] = 'myproject.settings' # path to your settings module
    application = get_wsgi_application()
    container = tornado.wsgi.WSGIContainer(application)
    http_server = tornado.httpserver.HTTPServer(container)
    http_server.listen(8888)
    tornado.ioloop.IOLoop.instance().start()

if __name__ == "__main__":
    main()

当我添加 import app（包含 celery app 初始化文件的应用程序）时，一切正常。