Python Django 截断响应答案

【问题标题】：Python Django truncates the responsePython Django 截断响应
【发布时间】：2016-11-29 16:14:02
【问题描述】：

我创建了一个小 Django 应用程序并将其部署在 Ubuntu 上，视图.py

from django.shortcuts import render
import json
import psycopg2
import csv
from django.http import HttpResponse
from ratelimit.decorators import ratelimit
def create_connection():
    return psycopg2.connect(database="data",user="ag",host="localhost",port=5432,password='pass')

def execute_query(query, parameters):
    with create_connection() as connection:
        result = []
        cursor = connection.cursor()
        cursor.execute(query, parameters)
        for row in cursor:
            result.append(row)
        for i in range(len(result)):
            result[i] = list(result[i])
            for j in range(len(result[i])):
                result[i][j] = result[i][j] if result[i][j] else ""
        return result

@ratelimit(key='ip', rate='10/m')
def to_csv(request):
    j_body = json.loads(request.GET.get('data', ''))
    query = j_body['query']
    print(query)
    print("****************************************************************")
    parameters = j_body['params']
    headers = j_body['headers']
    rows = execute_query(query, parameters)
    response = HttpResponse(content_type='text_csv')
    response['Content-Disposition'] = 'attachment; filename="result.csv"'
    writer = csv.writer(response)
    writer.writerow(headers)
    writer.writerows(rows)
    return response

这个应用程序的主要目标 - 获取 sql 查询并将 csv 响应作为文件返回，其大小可以超过 200 mb。我有一个问题 - 我的响应总是被截断为 696 kb（不管 cursor.execute 结果中的行数）。有例子。您可以看到最后截断的行。

服务器由“python3 manage.py runserver 0.0.0.0:8000”命令运行。请帮我解决我的问题。

更新我用 Nginx 和 gunicorn 重新部署了它，并尝试了这里建议的解决方案 ngnix + gunicorn throws truncated response body 尽管如此，我仍然遇到同样的错误

我按照这篇文章修复了我的代码 - https://docs.djangoproject.com/en/1.10/howto/outputting-csv/#streaming-large-csv-files

#Also changed execute_query method, so it doesn't create redurant copy of response from database
def execute_query(query, parameters):
    with create_connection() as connection:
        result = []
        cursor = connection.cursor()
        cursor.execute(query, parameters)
        return cursor

class Echo(object):
    """An object that implements just the write method of the file-like
    interface.
    """
    def write(self, value):
        """Write the value by returning it, instead of storing in a buffer."""
        return value

@ratelimit(key='ip', rate='10/m')
def to_csv(request):
    try:
        j_body = json.loads(request.GET.get('data', ''))
        query = j_body['query']
        print("****************************************************************")
        parameters = j_body['params']
        headers = j_body['headers']
        cursor = execute_query(query, parameters)
        pseudo_buffer = Echo()
        writer = csv.writer(pseudo_buffer)
        writer.writerow(headers)
        print('Writing began')
        response = StreamingHttpResponse((writer.writerow(row) for row in cursor),
                                     content_type="text/csv")
        response['Content-Disposition'] = 'attachment; filename="result.csv"'
        print('Sending began')
        return response
    except:
        return HttpResponse('Error')

现在一切正常。

【问题讨论】：

标签： python django wsgi

【解决方案1】：

听起来您可能需要StreamingHttpResponse。你试过实现这个吗？您似乎遇到了缓冲/分块响应的某种问题。可以使用 gunicorn 或 nginx 之类的东西来增加此缓冲区大小，但这不是真正的解决方案。

【讨论】：