如何禁用上游缓冲 Nginx + Docker + Gunicorn？

Question

我正在尝试使用 Nginx 作为客户端和 Gunicorn 之间的代理服务器。 Nginx、Gunicorn (Django) 是 Docker 的容器。问题是当我将大文件从客户端发送到 Django 应用程序时，我无法禁用上游缓冲。 TTFB 时间非常短，因此我的进度条（使用 xhr.upload.progress 事件）变得非常快（不到一秒）100%。然后我必须等待 30 秒，而文件将被上传到服务器。这是我的设置。请帮忙。我尝试了许多配置，试图将缓冲区大小设置为零等，检查 StackOverflow 上的许多答案，但没有任何帮助。

docker-compose.yaml

...
    services:
  db:
    image: postgres:12.4
    volumes:
      - postgres_data:/var/lib/postgresql/data/
    restart: always
    ports:
      - ${DB_PORT}:${DB_PORT}
    env_file:
      - ./.env

  backend:
    build: ./src/backend
    volumes:
      - RUDZASV0021:/code/storage/RUDZASV0021
      - logs:/code/logs
    restart: always
    depends_on:
      - db
    env_file:
      - ./.env

  nginx:
    build:
      context: .
      dockerfile: ./src/frontend/Dockerfile
    volumes:
      - ./docker-settings/default.conf:/etc/nginx/conf.d/default.conf:ro
    restart: always
    ports:
      - 80:80
      - 443:443
    depends_on:
      - backend

后端 Dockerfile

FROM python:3.8.7-slim

WORKDIR /code
COPY . .
RUN pip install -r /code/requirements.txt
RUN apt-get update && apt-get install -y mc
CMD gunicorn entrypoints.wsgi:application --workers=4 --worker-class=gevent  --timeout=90 --graceful-timeout=10 --bind 0.0.0.0:8000

Nginx Dockerfile

FROM nginx:1.20.0

WORKDIR /frontend
COPY ./src/frontend/dist .
WORKDIR /cert
COPY ./cert/device.key .
COPY ./cert/device.crt .

Nginx 默认配置文件

upstream hello_django {
    server backend:8000 fail_timeout=0;
}

server {
    listen 80;
    return 301 https://$host$request_uri;
}

server {

    listen 443;

    ssl on;
    ssl_certificate /cert/device.crt;
    ssl_certificate_key /cert/device.key;

    client_max_body_size 2G;
    keepalive_timeout 5;

    access_log /frontend/nginx-access.log;
    error_log /frontend/nginx-error.log;

    location / {
        root /frontend;
        try_files $uri /index.html;
    }

    location /api/ {
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Host $http_host;
        proxy_buffering off;
        proxy_request_buffering off;
        proxy_redirect off;
        proxy_pass http://hello_django;
    }

}

Answer 1

经过长时间的调查，我使用了以下决定。

1. 我不得不用 uWSGI 更改 Gunicorn，因为我发现“uwsgi_request_buffering off;”与 Nginx + uWSGI 一起正常工作
2. 在 Django 中创建自己的上传处理程序，它同时为任何文件 (guid) 写入数据进度并将临时文件写入驱动器
3. 开始上传后，将额外的请求从前端推送到后端（每 1 秒），询问特定文件的进度。我使用了两个附加标头来指定文件夹 id 和文件 id，因为在上传处理程序中您无法访问 request.GET 并且获取路径参数并不容易。

Nginx 默认配置文件

...
location /api/ {
        uwsgi_pass hello_django;
        include uwsgi_params;
        uwsgi_request_buffering off;
    }

后端 Dockerfile

FROM python:3.8.7-slim

WORKDIR /code
COPY . .
RUN apt-get update && apt-get install -y mc build-essential
RUN pip install -r /code/requirements.txt
CMD uwsgi --ini /etc/uwsgi.ini

uswgi.ini

[uwsgi]
module=entrypoints.wsgi:application 
master=True 
pidfile=/tmp/project-master.pid 
processes=5 
harakiri=300 
harakiri-verbose=True
max-requests=5000 
vacuum=True
single-interpreter=True
enable-threads=True
socket=0.0.0.0:8000
req-logger = file:/tmp/uwsgi_req.log
logger = file:/tmp/uwsgi_err.log

hadlers.py

from django.core.files.uploadhandler import TemporaryFileUploadHandler
from logic.domain import commands
from .views import bus
from django.conf import settings

class ProgressBarUploadHandler(TemporaryFileUploadHandler):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self._previous_start = 0

    @property 
    def _folder_id(self):
        return self.request.META['HTTP_X_FOLDER_ID']

    @property 
    def _id(self):
        return self.request.META['HTTP_X_FILE_ID']

    def receive_data_chunk(self, raw_data, start):
        # Update upload Progress 
        if start >= self._previous_start + settings.UPLOAD_PROGRESS_INCREMENT_SIZE:
            cmd = commands.SetUploadProgress(self._folder_id, self._id, start + len(raw_data))
            bus.handle(cmd)
            self._previous_start = start

        super().receive_data_chunk(raw_data, start)

    def file_complete(self, file_size):
        # Update upload Progress 
        cmd = commands.SetUploadProgress(self._folder_id, self._id, file_size)
        bus.handle(cmd)
        # Delete old upload Progress 
        cmd = commands.DeleteOldUploadProgress()
        bus.handle(cmd)

        return super().file_complete(file_size)