【发布时间】：2015-06-12 17:13:54
【问题描述】：
当我上传一个 4.8mb 的文件时，我的系统内存使用量从 30mb 跃升至 300mb+。
基本上，用户上传（例如）4.8mb jpeg，然后上传到 webfaction，然后存储在 Amazon S3 存储桶中。
我认为让我感到沮丧的一件事是我正在使用 easy_thumbnails 生成 3 个文件，这些文件也存储在 S3 存储桶中。
更新 2： 在这一点上，我认为我的主要问题是内存峰值但从未释放。在 Photo.objects.create() 完成后，我将开始研究运行 gc.collect()。看起来 sorl-thumbnail 可能是一个更好的选择，而且听起来它更适合使用远程存储。
更新 1： 我正在使用 django-debug-toolbar 并让它拦截重定向。我现在得到了一些有用的数据。它告诉我我已经执行了 20 个 db 查询（eek），但比这更糟糕（我相信）是 boto（我用于文件存储的东西）正在记录 124 条消息。看起来它正在分别发送每个文件。也许这是正常的，也许不是？无论哪种方式，它似乎都很高。
文件上传后，除非我重置 apache，否则内存永远不会恢复。
这是我的模型：
from django.db import models
from easy_thumbnails.fields import ThumbnailerImageField
from django.conf import settings
import datetime
import os
import string
from django.core.urlresolvers import reverse
...
class Photo(models.Model):
"""
A photo belongs to a user. A photo has a preview size and the original which
is referenced when printing or generating the PDF for print.
"""
def original_resolution(instance, filename):
"""
Returns a path to upload the image to. The path is created with the
website's slug and the current year. Using Amazon S3 for CDN storage
"""
today = datetime.datetime.now()
return 'uploads/{0}/{1}/{2}/{8}-{3}-{4}-{5}-{6}-{7}'.format(
instance.owner.pk,
today.year,
today.month,
today.day,
today.hour,
today.minute,
today.second,
clean_filename(filename),
'original')
def thumbnail_resolution(instance, filename):
"""
Returns a path to upload the image to. The path is created with the
website's slug and the current year. Using Amazon S3 for CDN storage
"""
today = datetime.datetime.now()
return 'uploads/{0}/{1}/{2}/{8}-{3}-{4}-{5}-{6}-{7}'.format(
instance.owner.pk,
today.year,
today.month,
today.day,
today.hour,
today.minute,
today.second,
clean_filename(filename),
'thumbnail')
def editor_resolution(instance, filename):
"""
Returns a path to upload the image to. The path is created with the
website's slug and the current year. Using Amazon S3 for CDN storage
"""
today = datetime.datetime.now()
return 'uploads/{0}/{1}/{2}/{8}-{3}-{4}-{5}-{6}-{7}'.format(
instance.owner.pk,
today.year,
today.month,
today.day,
today.hour,
today.minute,
today.second,
clean_filename(filename),
'editor')
height = models.PositiveIntegerField(blank=True)
width = models.PositiveIntegerField(blank=True)
owner = models.ForeignKey(settings.AUTH_USER_MODEL)
original = ThumbnailerImageField(
upload_to=original_resolution,
resize_source=dict(size=(0, 3100), crop="scale", quality=99),
height_field='height',
width_field='width',
verbose_name=u'Choose Photo')
thumbnail = ThumbnailerImageField(
upload_to=thumbnail_resolution,
resize_source=dict(size=(0, 100), crop="scale"),
blank=True,
null=True)
editor = ThumbnailerImageField(
upload_to=editor_resolution,
resize_source=dict(size=(0, 1000), crop="scale"),
blank=True,
null=True)
def get_absolute_url(self):
return reverse('photos:proxy_editor_image', kwargs={
'pk': self.pk})
def __unicode__(self):
return "Photo #{}".format(self.pk)
...
这是我处理上传的视图：
@login_required
def upload_photo(request):
"""
Creates and saves a new photo.
"""
if request.is_ajax():
response = {}
form = UploadPhotoForm(data=request.POST, files=request.FILES)
if form.is_valid():
new_photo = Photo.objects.create(
original=form.cleaned_data['original'],
thumbnail=form.cleaned_data['original'],
editor=form.cleaned_data['original'],
owner=form.cleaned_data['owner']
)
response['result'] = 'success'
response['message'] = 'Photo successfully uploaded!'
response['new_photo_pk'] = new_photo.pk
response['thumbnail_path'] = new_photo.thumbnail.url
response['editor_path'] = new_photo.editor.url
response['original_path'] = new_photo.original.url
response['editor_path_proxy'] = new_photo.get_absolute_url()
else:
response['result'] = 'fail'
response['message'] = 'The photo failed to upload.'
response['new_photo_pk'] = False
return HttpResponse(
json.dumps(response),
content_type='application/json'
)
if request.method == 'POST':
form = UploadPhotoForm(request.POST, request.FILES)
if form.is_valid():
Photo.objects.create(
original=form.cleaned_data['original'],
thumbnail=form.cleaned_data['original'],
editor=form.cleaned_data['original'],
owner=form.cleaned_data['owner']
)
messages.success(request, "Photo successfully uploaded!")
else:
messages.error(request, "The photo failed to upload.")
return HttpResponseRedirect(reverse('photos:list'))
我很困惑，我可能只是不明白一些事情。帮忙？
几个月来，我一直在努力解决这个内存使用问题，最终将其缩小到这个范围。
FWIW，我使用的是 django1.6 python 2.7，webfaction 托管，amazon s3 存储桶。
【问题讨论】：
-
你解决过这个问题吗？我遇到了与 django 的 FileField 和 s3 存储桶存储完全相同的问题
标签： python django apache file-upload amazon-s3