Celery + Docker + Django 无法运行异步作业

Question

我是 celery 和 docker 的新手，对此我有疑问。我有以下结构的 django 项目：

generic generic celery.py ... web tasks.py

generic/celery.py：

from __future__ import absolute_import, unicode_literals
import os
from celery import Celery
import logging
logger = logging.getLogger("Celery")

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'generic.settings')

app = Celery('generic')

app.config_from_object('django.conf:settings', namespace='CELERY')

app.autodiscover_tasks()

web/tasks.py：

from __future__ import absolute_import, unicode_literals
from django.conf import settings  
from generic.celery import app

@app.task
def foo():
    ...

docker-compose.yml:

version: '3.0'
services:
  db:
    image: postgres
  web:
    build: .
    command: python3 manage.py runserver 0.0.0.0:8000
    volumes:
      - .:/src
    ports:
      - "8000:8000"
    depends_on:
      - db
 redis:
    image: redis:latest
    container_name: rd01
    ports:
     - '6379:6379'
 celery:
    build: .
     container_name: cl01
     command: celery worker --app=app.tasks
     volumes:
       - ..:/src
     links:
       - db
      - redis

当我运行我的应用程序 ModuleNotFound 时引发

ModuleNotFoundError: No module named 'app.tasks'

我也试过用 docker-compose.yml 写：

command: celery worker --app=generic.tasks

我收到了

AttributeError: module 'generic.tasks' has no attribute 'celery'

我也试过了：

command: celery worker --app=web.tasks

和：

ModuleNotFoundError: No module named 'web'

Dockerfile

FROM python:3
ENV PYTHONUNBUFFERED 1

ENV APP_USER user
ENV APP_ROOT /src

RUN groupadd -r ${APP_USER} \
    && useradd -r -m \
    --home-dir ${APP_ROOT} \
    -s /usr/sbin/nologin \
    -g ${APP_USER} ${APP_USER}

WORKDIR ${APP_ROOT}

RUN apt-get update && apt-get -y install postgresql
ADD config/requirements.txt .
RUN pip install -r requirements.txt
ADD . /src/

USER ${APP_USER}
ADD . ${APP_ROOT}

我也尝试过将模块作为包含参数传递给 Celery 对象

谁能告诉我这有什么问题？我发现了很多类似的帖子，但没有答案没有帮助

Answer 1

你可以试试这个（通用 - django 项目目录）：

结构：

generic
  generic
    __init__.py
    celery.py
    settings.py
    ...
  web
    tasks.py
  docker-compose.yml
  Dockerfile
  entrypoint.sh
  django-stack.env 

generic/generic/celery.py：

import os
from celery import Celery


os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'generic.settings')
app = Celery('generic')
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()

generic/web/tasks.py：

from celery import shared_task

@shared_task
def foo():
    ...

通用/通用/__init.__py：

from .celery import app as celery_app


__all__ = ("celery_app",)

通用/docker-compose.yml：

version: '3.7'

services:
  web:
    build: .
    command: python manage.py runserver 0.0.0.0:8000
    volumes:
      - ./:/usr/src/app/
    ports:
      - "8000:8000"
    env_file: ./django-stack.env
    container_name: web-django-stack
    restart: always
    depends_on:
      - db
      - redis

  celery:
    build: .
    container_name: cl01
    command: celery worker --app=generic --loglevel=info --logfile=logs/celery.log
    volumes:
      - ./:/usr/src/app/
    restart: always
    env_file: ./django-stack.env
    links:
        - db
        - redis
    depends_on:
      - web
      - redis

  flower:
    build: .
    command: flower -A generic --port=5555 --broker=redis://redis:6379/0
    volumes:
      - ./:/usr/src/app/
    ports:
      - 5555:5555
    env_file: ./django-stack.env
    container_name: flower-django-stack
    restart: always
    depends_on:
      - web
      - redis


  db:
    image: postgres:12.3-alpine
    restart: always
    env_file: ./django-stack.env
    
 redis:
    image: redis:5.0.3-alpine
    container_name: rd01
    restart: always
    ports:
     - 6379:6379

通用/Dockerfile：

# pull official base image
FROM python:3.8.5

# set environment variables
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1

# copy requirements
COPY ./requirements.txt /tmp/

# install all requirements (be carefull in production with it)
RUN apt-get update && python -m pip install --upgrade pip && python -m pip install -r /tmp/requirements.txt

# set work directory
WORKDIR /usr/src/app

# copy entrypoint.sh
COPY ./entrypoint.sh /usr/src/app/entrypoint.sh

# copy project
COPY . /usr/src/app/

# run entrypoint.sh
ENTRYPOINT ["/usr/src/app/entrypoint.sh"]

generic/entrypoint.sh：

#!/bin/sh

# uncomment that you need
# python manage.py flush --no-input
python manage.py migrate
# python manage.py collectstatic --no-input --clear
# echo "from django.contrib.auth.models import User; User.objects.filter(email='$DJANGO_ADMIN_EMAIL').delete(); User.objects.create_superuser('$DJANGO_ADMIN_NAME', '$DJANGO_ADMIN_EMAIL', '$DJANGO_ADMIN_PASSWORD')"  | python manage.py shell

# Where: $DJANGO_ADMIN_NAME, $DJANGO_ADMIN_EMAIL, $DJANGO_ADMIN_PASSWORD - env var from generic/django-stack.env

exec "$@"

一些环境变量，例如你可以添加到generic/django-stack.env的Django SECRET_KEY：

PROJECT35_DJANGO_SECRET_VALUE=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
DJANGO_SETTINGS_MODULE=generic.settings

# Database Settings
POSTGRES_USER=admin 
POSTGRES_PASSWORD=admin
POSTGRES_DB=admin  
DJANGO_DB_HOST=postgres
DJANGO_DB_PORT=5432

# Django admin (like createsuperuser), if this user exists - it was deleted before creation (uncomment line in entrypoint.sh)
DJANGO_ADMIN_NAME=some_admin_name
DJANGO_ADMIN_EMAIL=admin_mail@mail.com
DJANGO_ADMIN_PASSWORD=admin_password

# Also, add two lines below in settings.py
CELERY_BROKER=redis://redis:6379/0
CELERY_BACKEND=redis://redis:6379/0   

设置你的环境。变量。

我不是专家，但它对我有用。也许，这会对你有所帮助..