【发布时间】：2016-02-02 15:15:28
【问题描述】：
我正在尝试使用 django + celery + djcelery + rabbitmq + memcached 将同步任务卸载到其他服务器。
根据 celery log 和 celery flower 都将 async 任务发送到 worker 并执行成功。但是结果似乎永远不会返回。
网页服务器上运行的代码是：
def wait_task(task, *args):
async_result = task.delay(*args)
return async_result.get()
@shared_task(serializer="pickle")
def render_pdf(pdf_id):
pdf = do_stuff()
return pdf
pdf = wait_task(render_pdf, pdf_id)
我的BROKER_URL 设置为librabbitmq://guest:guest@172.17.0.5:5672/，这是由芹菜原木和芹菜花产量证明的。
芹菜花中显示的截断结果让人想起成功渲染的 PDF。
/etc/rabbitmq/rabbitmq-env.conf 是：
NODENAME=rabbit@localhost
NODE_IP_ADDRESS=0.0.0.0
不太确定下一步该往哪里看。假设 celery 没有返回/回调到 rabbitmq，或者 rabbitmq 没有返回/回调到原始的 wait_task 调用。
如果我在与代理相同的 VM 上运行 Web 服务器，那么一切正常，即使使用相同的 BROKER_URL。在配置中的某个地方，我想它正在假设回调到哪里，例如 localhost/127.0.0.1 或其他什么？
更新 1：
Celery 从启动到第一次共享任务完成的日志输出：
[2016-02-02 23:54:07,786: DEBUG/MainProcess] | Worker: Preparing bootsteps.
[2016-02-02 23:54:07,792: DEBUG/MainProcess] | Worker: Building graph...
[2016-02-02 23:54:07,793: DEBUG/MainProcess] | Worker: New boot order: {StateDB, Timer, Hub, Queues (intra), Pool, Autoreloader, Beat, Autoscaler, Consumer}
[2016-02-02 23:54:07,796: DEBUG/MainProcess] | Consumer: Preparing bootsteps.
[2016-02-02 23:54:07,797: DEBUG/MainProcess] | Consumer: Building graph...
[2016-02-02 23:54:07,799: DEBUG/MainProcess] | Consumer: New boot order: {Connection, Events, Mingle, Gossip, Tasks, Control, Agent, Heart, event loop}
[2016-02-02 23:54:07,806: DEBUG/MainProcess] | Worker: Starting Hub
[2016-02-02 23:54:07,807: DEBUG/MainProcess] ^-- substep ok
[2016-02-02 23:54:07,807: DEBUG/MainProcess] | Worker: Starting Pool
[2016-02-02 23:54:07,811: DEBUG/MainProcess] ^-- substep ok
[2016-02-02 23:54:07,813: DEBUG/MainProcess] | Worker: Starting Autoscaler
[2016-02-02 23:54:07,813: DEBUG/MainProcess] ^-- substep ok
[2016-02-02 23:54:07,814: DEBUG/MainProcess] | Worker: Starting Consumer
[2016-02-02 23:54:07,815: DEBUG/MainProcess] | Consumer: Starting Connection
[2016-02-02 23:54:07,824: INFO/MainProcess] Connected to amqp://guest:**@172.17.0.5:5672//
[2016-02-02 23:54:07,825: DEBUG/MainProcess] ^-- substep ok
[2016-02-02 23:54:07,826: DEBUG/MainProcess] | Consumer: Starting Events
[2016-02-02 23:54:07,836: DEBUG/MainProcess] ^-- substep ok
[2016-02-02 23:54:07,837: DEBUG/MainProcess] | Consumer: Starting Mingle
[2016-02-02 23:54:07,838: INFO/MainProcess] mingle: searching for neighbors
[2016-02-02 23:54:08,848: INFO/MainProcess] mingle: all alone
[2016-02-02 23:54:08,850: DEBUG/MainProcess] ^-- substep ok
[2016-02-02 23:54:08,851: DEBUG/MainProcess] | Consumer: Starting Gossip
[2016-02-02 23:54:08,855: DEBUG/MainProcess] ^-- substep ok
[2016-02-02 23:54:08,856: DEBUG/MainProcess] | Consumer: Starting Tasks
[2016-02-02 23:54:08,862: DEBUG/MainProcess] ^-- substep ok
[2016-02-02 23:54:08,863: DEBUG/MainProcess] | Consumer: Starting Control
[2016-02-02 23:54:08,867: DEBUG/MainProcess] ^-- substep ok
[2016-02-02 23:54:08,868: DEBUG/MainProcess] | Consumer: Starting Heart
[2016-02-02 23:54:08,870: DEBUG/MainProcess] ^-- substep ok
[2016-02-02 23:54:08,871: DEBUG/MainProcess] | Consumer: Starting event loop
[2016-02-02 23:54:08,873: DEBUG/MainProcess] /virtualenv/processing/local/lib/python2.7/site-packages/celery/fixups/django.py:265: UserWarning: Using settings.DEBUG leads to a memory leak, never use this setting in production environments!
warnings.warn('Using settings.DEBUG leads to a memory leak, never '
[2016-02-02 23:54:08,875: DEBUG/MainProcess] worker-1@172.17.0.5 ready.
[2016-02-02 23:54:08,876: DEBUG/MainProcess] | Worker: Hub.register Autoscaler...
[2016-02-02 23:54:08,877: DEBUG/MainProcess] | Worker: Hub.register Pool...
[2016-02-02 23:54:08,878: DEBUG/MainProcess] basic.qos: prefetch_count->1
[2016-02-02 23:54:13,263: INFO/MainProcess] Received task: tasks.render_pdf[149cc075-1d09-4547-86ee-57608c1df8aa]
[2016-02-02 23:54:13,265: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f08fccd7488> (args:('tasks.render_pdf', '149cc075-1d09-4547-86ee-57608c1df8aa', (u'en', 8583L), {}, {'utc': True, u'is_eager': False, 'chord': None, u'group': None, 'args': (u'en', 8583L), 'retries': 0, u'delivery_info': {u'priority': None, u'redelivered': False, u'routing_key': 'default', u'exchange': 'default'}, 'expires': None, u'hostname': 'worker-1@172.17.0.5', 'task': 'tasks.render_pdf', 'callbacks': None, u'correlation_id': '149cc075-1d09-4547-86ee-57608c1df8aa', 'errbacks': None, 'timelimit': (None, None), 'taskset': None, 'kwargs': {}, 'eta': None, u'reply_to': '8502acac-0475-3f84-b542-03fdf4ea4d3d', 'id': '149cc075-1d09-4547-86ee-57608c1df8aa', u'headers': {}}) kwargs:{})
[2016-02-02 23:54:13,267: DEBUG/MainProcess] Task accepted: tasks.render_pdf[149cc075-1d09-4547-86ee-57608c1df8aa] pid:5033
[2016-02-02 23:54:14,193: INFO/MainProcess] Task tasks.render_pdf[149cc075-1d09-4547-86ee-57608c1df8aa] succeeded in 0.926254872s: '%PDF-1.4\n1 0 obj\n<<\n/Title (\xfe\xff\x00T\x00i\x00c\x00k\x00e\x00t\x00f\x00l\x00a\x00p\x00...
更新 2：
缓存设置：
CELERY_RESULT_BACKEND = 'djcelery.backends.cache:CacheBackend'
MEMCACHED_HOST = '172.17.0.5'
MEMCACHED_PORT = 11211
CACHES = {
'default': {
'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',
'LOCATION': '%s:%s' % (MEMCACHED_HOST, str(MEMCACHED_PORT)),
'TIMEOUT': 18000 # 5 hours
},
}
更新 3：
/var/logs/rabbitmq/rabbitmq@localhost.log 中的输出：
如果我在不同的实例上运行 celery 和 web 服务器，但如上所示：
=INFO REPORT==== 2-Feb-2016::16:12:33 ===
accepting AMQP connection <0.469.0> (172.17.0.6:36786 -> 172.17.0.5:5672)
如果我在同一个实例上运行 celery 和 web 服务器，效果很好：
=INFO REPORT==== 2-Feb-2016::16:13:09 ===
accepting AMQP connection <0.482.0> (172.17.0.5:33863 -> 172.17.0.5:5672)
【问题讨论】：
-
你的 celery 结果后端设置是什么？ docs.celeryproject.org/en/latest/…
-
Celery 本身是否会向控制台打印任何内容？尝试将日志记录设为 DEBUG 来启动它。
-
@AlexMorozov Celery 正在将调试输出到日志。看不到任何错误或任何相关的东西，不过我会用输出更新 OP。
-
@dm03514 忘了我也在使用 djcelery 作为后端，用缓存设置更新了 OP。
-
您的结果后端似乎配置错误。