同一个 docker-compose 中的 Celery 和 Flask

Question

我正在尝试使用 docker-compose 来生成我的 Flask/Celery/Redis 服务。

这是我的docker-compose.yml：

flask:
    build: .
    command: "python3 app.py"
    ports:
        - '5000:5000'
    links:
        - redis
    volumes:
        - .:/usr/src/app:ro

celery:
    build: .
    command: "celery -A app.celery worker --loglevel=info"
    volumes:
        - .:/usr/src/app:ro

redis:
    image: redis
    ports:
        - '6379:6379'

当我运行这个docker-compose 时，Flask 和 Redis 都可以正常启动并按预期运行。关于 Celery，Docker 报告：flaskcelery_celery_1 exited with code 1，没有其他信息。

如果我在没有 Docker 的情况下运行我的三个服务，并使用 celery -A app.celery worker --loglevel=info 启动 Celery，我的应用程序运行正常。

如有必要，请提供更多信息：

Dockerfile：（这个镜像在构建时也安装了requirements.txt）

FROM python:3.5-onbuild
EXPOSE 5000

requirements.txt：

flask==0.11.1
celery==3.1.23

docker-compose up 输出：

Starting flaskcelery_celery_1
Starting flaskcelery_redis_1
Starting flaskcelery_flask_1
Attaching to flaskcelery_celery_1, flaskcelery_redis_1, flaskcelery_flask_1
redis_1  |                 _._                                                  
redis_1  |            _.-``__ ''-._                                             
redis_1  |       _.-``    `.  `_.  ''-._           Redis 3.2.3 (00000000/0) 64 bit
redis_1  |   .-`` .-```.  ```\/    _.,_ ''-._                                   
redis_1  |  (    '      ,       .-`  | `,    )     Running in standalone mode
redis_1  |  |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379
redis_1  |  |    `-._   `._    /     _.-'    |     PID: 1
redis_1  |   `-._    `-._  `-./  _.-'    _.-'                                   
redis_1  |  |`-._`-._    `-.__.-'    _.-'_.-'|                                  
redis_1  |  |    `-._`-._        _.-'_.-'    |           http://redis.io        
redis_1  |   `-._    `-._`-.__.-'_.-'    _.-'                                   
redis_1  |  |`-._`-._    `-.__.-'    _.-'_.-'|                                  
redis_1  |  |    `-._`-._        _.-'_.-'    |                                  
redis_1  |   `-._    `-._`-.__.-'_.-'    _.-'                                   
redis_1  |       `-._    `-.__.-'    _.-'                                       
redis_1  |           `-._        _.-'                                           
redis_1  |               `-.__.-'                                               
redis_1  | 
redis_1  | 1:M 23 Aug 10:23:08.409 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
redis_1  | 1:M 23 Aug 10:23:08.409 # Server started, Redis version 3.2.3
redis_1  | 1:M 23 Aug 10:23:08.409 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
redis_1  | 1:M 23 Aug 10:23:08.409 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
redis_1  | 1:M 23 Aug 10:23:08.409 * The server is now ready to accept connections on port 6379
flaskcelery_celery_1 exited with code 1
flask_1  |  * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
flask_1  |  * Restarting with stat
flask_1  |  * Debugger is active!
flask_1  |  * Debugger pin code: 196-119-737

Answer 1

解决了我的问题。我最终发现我可以在 Docker 映像上获得命令提示符：

docker build -t <image name> .
docker run -it <image name> /bin/bash

然后尝试在容器内运行celery 发现问题：

root@4a6edc5d7372:/usr/src/app# celery -A app.celery worker --loglevel=info
Running a worker with superuser privileges when the
worker accepts messages serialized with pickle is a very bad idea!

If you really want to continue then you have to set the C_FORCE_ROOT
environment variable (but please think about this before you do).

User information: uid=0 euid=0 gid=0 egid=0

Docker 通常以 root 身份运行，而 Celery 出于安全原因不喜欢以 root 身份运行（我相信您可以通过 pickle 反序列化获得代码执行）。更安全的解决方案是将celery 容器设置为nobody。工作docker-compose.yml：

flask:
    build: .
    command: "python3 app.py"
    ports:
        - '5000:5000'
    links:
        - redis
        - celery
    volumes:
        - .:/usr/src/app:ro

celery:
    build: .
    command: "celery -A app.celery worker --loglevel=info"
    user: nobody
    links:
        - redis
    volumes:
        - .:/usr/src/app:ro

redis:
    image: redis
    ports:
        - '6379:6379'

Answer 2

如果您不想处理 docker-compose 而只想使用 docker，那么这可能对您有用。首先，您需要创建一个“supervisord”文件，如下所示。

[supervisord]
nodaemon=true

[program:python]
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0
command=python flask_api.py

[program:celeryworker]
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0
command=celery -A flask_api worker -c 2

那么你的 docker 文件应该是这样的。显然这只是一个模板，您可以根据需要更改它。

FROM ubuntu:16.04
RUN apt-get update -y
RUN apt-get install -y python-pip python-dev build-essential libpq-dev git postgresql-9.5 postgresql-contrib
RUN apt-get install -y supervisor
RUN pip install --upgrade pip
COPY . /app
WORKDIR /app
RUN pip install -r requirements.txt
WORKDIR /app
CMD ["supervisord"]

通过这种方式，您可以在同一个 docker 容器中运行烧瓶应用程序和 celery worker。