您如何处理 celery（不是您的代码）引发的异常？答案

【问题标题】：How do you deal with an exception raised by celery (not your code)?您如何处理 celery（不是您的代码）引发的异常？
【发布时间】：2017-07-10 03:45:22
【问题描述】：

所以现在在我的烧瓶应用程序中，我正在使用 Celery 在远程机器上部署服务器。现在，我有一个枚举状态，它表示我的部署过程的生命周期：

@celery.task(bind=True)
def deploy_server(self, server_id):
  server = Server.query.get(server_id)
  if not server.can_launch():
    return

  try:
    server.status = RemoteStatus.LAUNCHING
    db.session.commit()

    verify_DNS(server)

    host = server.server.ssh_user + '@' + server.server.ip
    execute(fabric_deploy_server, self, server, hosts=host)

    server.status = RemoteStatus.LAUNCHED
    db.session.commit()
  except Exception as e:
    server.status = RemoteStatus.ERROR
    db.session.commit()
    traceback.print_exc()
    raise e

如您所见，在部署服务器时，其状态更改为“正在启动”。如果有异常，会改为ERROR。

我发现了一个完全绕过这组代码的异常：当我的 celery 服务器因太多请求而超载时，我得到了这个异常：

[2017-07-09 18:00:03,127: WARNING/PoolWorker-3] /app/.heroku/python/lib/python2.7/site-packages/celery/app/trace.py:542: RuntimeWarning: Exception raised outside body: ConnectionError('max number of clients reached',): 
Traceback (most recent call last): 
  File "/app/.heroku/python/lib/python2.7/site-packages/celery/app/trace.py", line 427, in trace_task 
    uuid, retval, task_request, publish_result, 
  File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/base.py", line 152, in mark_as_done 
    self.store_result(task_id, result, state, request=request) 
  File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/base.py", line 309, in store_result 
    request=request, **kwargs) 
  File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/base.py", line 652, in _store_result 
    self.set(self.get_key_for_task(task_id), self.encode(meta)) 
  File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/redis.py", line 204, in set 
    return self.ensure(self._set, (key, value), **retry_policy) 
  File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/redis.py", line 194, in ensure 
    **retry_policy) 
  File "/app/.heroku/python/lib/python2.7/site-packages/kombu/utils/functional.py", line 333, in retry_over_time 
    return fun(*args, **kwargs) 
  File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/redis.py", line 213, in _set 
    pipe.execute() 
  File "/app/.heroku/python/lib/python2.7/site-packages/redis/client.py", line 2641, in execute 
    return execute(conn, stack, raise_on_error) 
  File "/app/.heroku/python/lib/python2.7/site-packages/redis/client.py", line 2495, in _execute_transaction 
    connection.send_packed_command(all_cmds) 
  File "/app/.heroku/python/lib/python2.7/site-packages/redis/connection.py", line 538, in send_packed_command 
    self.connect() 
  File "/app/.heroku/python/lib/python2.7/site-packages/redis/connection.py", line 446, in connect 
    self.on_connect() 
  File "/app/.heroku/python/lib/python2.7/site-packages/redis/connection.py", line 514, in on_connect 
    if nativestr(self.read_response()) != 'OK': 
  File "/app/.heroku/python/lib/python2.7/site-packages/redis/connection.py", line 577, in read_response 
    response = self._parser.read_response() 
  File "/app/.heroku/python/lib/python2.7/site-packages/redis/connection.py", line 255, in read_response 
    raise error 
ConnectionError: max number of clients reached 
  exc, exc_info.traceback)))

我最大的问题是这个错误是在我的 Try/Catch 块之外的某个地方引发的。因此，当这个异常发生时，我所有的服务器都保持在“Launching”模式，而不是“Error”。

如何捕捉这个异常并修改 Server.status？

【问题讨论】：

max number of clients reached by redis/connection.py... 你确定这完全是 Celery 的问题吗？

标签： python celery

【解决方案1】：

在 redis 2.4 中，最大连接数有一个硬编码限制，即 10,000。在 redis 2.6+ 中，您可以在 redis.conf 中指定最大客户端数。这也不是你的经纪人redis拒绝接受连接的问题。

使用 redis CLI 设置 redis 可以同时处理的最大客户端数。查看redis clients

【讨论】：