【问题标题】:django-pipeline and s3boto storage don't seem to work togetherdjango-pipeline 和 s3boto 存储似乎不能一起工作
【发布时间】:2012-03-25 10:09:10
【问题描述】:

我正在尝试将 django-pipeline-1.1.27 与 s3boto 一起使用来压缩和过滤静态文件,然后将它们上传到 s3 存储桶。如果我只是使用:

PIPELINE_STORAGE = 'pipeline.storage.PipelineFinderStorage'

然后它工作了,我得到一个静态文件夹,其中包含我配置的漂亮版本文件。一旦我切换到

PIPELINE_STORAGE = 'storages.backends.s3boto.S3BotoStorage'

我明白了

Traceback (most recent call last):
  File "manage.py", line 15, in <module>
    execute_manager(settings)
  File "/my/virtual/env/lib/python2.7/site-packages/django/core/management/__init__.py", line 438, in execute_manager
    utility.execute()
  File "/my/virtual/env/lib/python2.7/site-packages/django/core/management/__init__.py", line 379, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/my/virtual/env/lib/python2.7/site-packages/django/core/management/base.py", line 191, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/my/virtual/env/lib/python2.7/site-packages/django/core/management/base.py", line 220, in execute
    output = self.handle(*args, **options)
  File "/my/virtual/env/lib/python2.7/site-packages/pipeline/management/commands/synccompress.py", line 39, in handle
    packager.pack_stylesheets(package, sync=sync, force=force)
  File "/my/virtual/env/lib/python2.7/site-packages/pipeline/packager.py", line 52, in pack_stylesheets
    **kwargs)
  File "/my/virtual/env/lib/python2.7/site-packages/pipeline/packager.py", line 60, in pack
    package['output'], package['paths'])
  File "/my/virtual/env/lib/python2.7/site-packages/pipeline/versioning/__init__.py", line 45, in need_update
    version = self.version(paths)
  File "/my/virtual/env/lib/python2.7/site-packages/pipeline/versioning/__init__.py", line 20, in version
    return getattr(self.versioner, 'version')(paths)
  File "/my/virtual/env/lib/python2.7/site-packages/pipeline/versioning/hash/__init__.py", line 37, in version
    buf = self.concatenate(paths)
  File "/my/virtual/env/lib/python2.7/site-packages/pipeline/versioning/hash/__init__.py", line 27, in concatenate
    return '\n'.join([self.read_file(path) for path in paths])
  File "/my/virtual/env/lib/python2.7/site-packages/pipeline/versioning/hash/__init__.py", line 31, in read_file
    file = storage.open(path, 'rb')
  File "/my/virtual/env/lib/python2.7/site-packages/django/core/files/storage.py", line 33, in open
    file = self._open(name, mode)
  File "/my/virtual/env/lib/python2.7/site-packages/storages/backends/s3boto.py", line 177, in _open
    raise IOError('File does not exist: %s' % name)
IOError: File does not exist: css/style.css

这是我的 source 文件之一。那么为什么当我切换到 s3boto 存储时,管道不再需要执行过滤/连接/压缩步骤呢?

可能是我在做某事。这是其他配置以防万一:

INSTALLED_APPS = (
    ...
    'pipeline',
    'storages',
)

STATICFILES_FINDERS = (
    'pipeline.finders.PipelineFinder',
    'django.contrib.staticfiles.finders.FileSystemFinder',
    'django.contrib.staticfiles.finders.AppDirectoriesFinder',
)

STATIC_ROOT = "/some/path/outside/django_project/deploy_static"
STATICFILES_DIRS = () # All statics in this site are in apps

STATICFILES_STORAGE = 'pipeline.storage.PipelineStorage'
PIPELINE_STORAGE = 'storages.backends.s3boto.S3BotoStorage'

PIPELINE = True
PIPELINE_AUTO = True
PIPELINE_VERSION = True
PIPELINE_VERSION_PLACEHOLDER = 'VERSION'
PIPELINE_VERSIONING = 'pipeline.versioning.hash.SHA1Versioning'

PIPELINE_CSS = {
    'standard': {
        'source_filenames': (
          'css/style.css',
          ...
        ),
        'output_filename': 'css/all-VERSION.css',
        'extra_context': {
            'media': 'screen,projection',
        },
    }
}

我的网站在 Django 1.3.1 上。

我正在运行的命令是:

python manage.py synccompress --force

AWS 信誉也在设置中,但这没有实际意义,因为它甚至还没有达到这一点。

更新添加了 cmets 中请求的完整堆栈和设置

更新 应图书馆作者的要求,我尝试升级到最新的测试版。到目前为止的观察结果:

  1. 我现在不知道如何获取版本化压缩文件
  2. collectstatic 给我留下了压缩文件原件
  3. 在配置 boto 存储时仍然从 django-pipeline 收到相同的错误:它想将我的 source 文件发送到 s3,但我什至看不到它在哪里暂存我的资产。 STATIC_ROOT 中没有任何内容。

更新 我创建了最简单的项目,适用于查找器存储,然后使用 S3Boto 中断。我已将其推送到 github,并包含了堆栈跟踪的捕获。

https://github.com/estebistec/simple_pipeline https://raw.github.com/estebistec/simple_pipeline/master/STACKTRACE

如果有人告诉我我正在做一些非常愚蠢的事情,我会欣喜若狂,而这一切都应该奏效。

【问题讨论】:

  • STATIC_ROOTSTATICFILES_DIRSPIPELINE_ROOT 的设置是什么?完整的回溯也很有用!
  • 添加了完整堆栈、STATIC_ROOT 和 STATICFILES_DIRS。在 1.1.27 文档中提到 PIPELINE_ROOT 是在与绝对路径相关的信息框中,我没有使用它来定位源文件。
  • 那么我的空 STATICFILES_DIRS 是不是把它全部扔掉了?无论文档何时暗示应该使用 PIPELINE_ROOT,都需要它吗?

标签: python django static-content django-pipeline


【解决方案1】:

django-pipeline 1.1.x 对于应该如何使用静态文件有点愚蠢,它更喜欢将所有东西都放在一个地方。 我建议你尝试django-pipeline 1.2 和最新的django-staticfilesdjango 1.4

使用这样的自定义:

STATICFILES_STORAGE = 'your.app.S3PipelineStorage'

代码如下所示:

from staticfiles.storage import CachedFilesMixin

from pipeline.storage import PipelineMixin

from storages.backends.s3boto import S3BotoStorage


class S3PipelineStorage(PipelineMixin, CachedFilesMixin, S3BotoStorage):
     pass

您可以找到如何修复您的应用程序,但是除非您使用 1.2c1 版本,否则编译文件仍然存在错误:https://gist.github.com/1999564

【讨论】:

  • 对不起,我的意思是在管道查找器后面加上省略号。我并没有试图让我很难为我提供帮助:/
  • 哦,哇,我错过了文档中的静态文件依赖项吗?我很快就退出了 1.2 测试版,因为我可以看到它使用了 django 1.3 中没有的东西。我不知道 django-staticfiles 在 django 之外仍然得到增强:/
  • 文档总是需要改进,不要犹豫打开 github 问题以获取文档或其他内容。
  • 我想一旦我看到所有这些工作,我就会这样做。感谢您迄今为止的帮助。
  • 版本控制如何与 1.2 测试版一起使用?我在单个源文件上看到了 sha1 版本,但在我的聚合/过滤文件上却没有。
【解决方案2】:

我刚刚在使用 django-pipeline==1.3.23 的 Django 1.6 项目中遇到了同样的错误,解决方案只是删除了 PIPELINE_STORAGE 设置。

【讨论】:

    【解决方案3】:

    类似的错误消息还有另一个问题,它会影响django-pipeline 的早期版本和当前版本 (1.5.4)。

    错误消息是IOError: File does not exist,它发生在s3boto.py.open()packager.pack_stylesheets()。如果您使用任何编译器(Compass、Sass、Less 等),您可能会遇到问题。我怀疑它也会影响JS编译器,但我没有确认。

    简而言之,编译器在本地静态存储中生成输出文件,接下来,compress 正在尝试在 s3 存储中查找输出。

    如果它影响到您,您可能需要查看https://github.com/cyberdelia/django-pipeline/issues/473。有两个拉取请求(补丁),一个由skirsdeda 制作,另一个由thomasyip(我)制作。两者都可以解决您的问题。如果您希望将已编译(但预压缩)的文件复制到 s3 并可供应用程序使用,您可以使用 thomasyip (me) 的补丁。

    这里是问题的完整追溯:

    Traceback (most recent call last):
      File "apps/manage.py", line 16, in <module>
        execute_from_command_line(sys.argv)
      File "/Users/thomas/Dev/Project/main-server/venv/lib/python2.7/site-packages/django/core/management/__init__.py", line 385, in execute_from_command_line
        utility.execute()
      File "/Users/thomas/Dev/Project/main-server/venv/lib/python2.7/site-packages/django/core/management/__init__.py", line 377, in execute
        self.fetch_command(subcommand).run_from_argv(self.argv)
      File "/Users/thomas/Dev/Project/main-server/venv/lib/python2.7/site-packages/django/core/management/base.py", line 288, in run_from_argv
        self.execute(*args, **options.__dict__)
      File "/Users/thomas/Dev/Project/main-server/venv/lib/python2.7/site-packages/django/core/management/base.py", line 338, in execute
        output = self.handle(*args, **options)
      File "/Users/thomas/Dev/Project/main-server/venv/lib/python2.7/site-packages/django/core/management/base.py", line 533, in handle
        return self.handle_noargs(**options)
      File "/Users/thomas/Dev/Project/main-server/venv/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 171, in handle_noargs
        collected = self.collect()
      File "/Users/thomas/Dev/Project/main-server/venv/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 117, in collect
        for original_path, processed_path, processed in processor:
      File "/Users/thomas/Dev/Project/main-server/venv/lib/python2.7/site-packages/pipeline/storage.py", line 26, in post_process
        packager.pack_stylesheets(package)
      File "/Users/thomas/Dev/Project/main-server/venv/lib/python2.7/site-packages/pipeline/packager.py", line 96, in pack_stylesheets
        variant=package.variant, **kwargs)
      File "/Users/thomas/Dev/Project/main-server/venv/lib/python2.7/site-packages/pipeline/packager.py", line 106, in pack
        content = compress(paths, **kwargs)
      File "/Users/thomas/Dev/Project/main-server/venv/lib/python2.7/site-packages/pipeline/compressors/__init__.py", line 73, in compress_css
        css = self.concatenate_and_rewrite(paths, output_filename, variant)
      File "/Users/thomas/Dev/Project/main-server/venv/lib/python2.7/site-packages/pipeline/compressors/__init__.py", line 137, in concatenate_and_rewrite
        content = self.read_text(path)
      File "/Users/thomas/Dev/Project/main-server/venv/lib/python2.7/site-packages/pipeline/compressors/__init__.py", line 220, in read_text
        content = self.read_bytes(path)
      File "/Users/thomas/Dev/Project/main-server/venv/lib/python2.7/site-packages/pipeline/compressors/__init__.py", line 214, in read_bytes
        file = staticfiles_storage.open(path)
      File "/Users/thomas/Dev/Project/main-server/venv/lib/python2.7/site-packages/django/core/files/storage.py", line 35, in open
        return self._open(name, mode)
      File "/Users/thomas/Dev/Project/main-server/venv/lib/python2.7/site-packages/storages/backends/s3boto.py", line 366, in _open
        raise IOError('File does not exist: %s' % name)
    IOError: File does not exist: sheets/sass/sheets.css
    

    【讨论】:

      【解决方案4】:

      补充答案,您也可以在压缩时使用 GZIP:

      from django.contrib.staticfiles.storage import CachedFilesMixin
      
      from pipeline.storage import PipelineMixin
      
      from storages.backends.s3boto import S3BotoStorage
      
      
      class S3PipelineStorage(PipelineMixin, CachedFilesMixin, S3BotoStorage):
          def __init__(self, *args, **kwargs):
              self.gzip = True
              super(S3PipelineStorage, self).__init__(*args, **kwargs)
      

      使用如下设置:

      COMPRESS_STORAGE = STATICFILES_STORAGE = 'my.apps.main.S3PipelineStorage'
      

      【讨论】:

        【解决方案5】:

        不确定这对其他人是否有效。我按照上面的解决方案,不断收到以下错误:

        Traceback (most recent call last):
          File "manage.py", line 24, in <module>
            execute_from_command_line(sys.argv)
          File "python3.4/site-packages/django/core/management/__init__.py", line 338, in execute_from_command_line
            utility.execute()
          File "python3.4/site-packages/django/core/management/__init__.py", line 330, in execute
            self.fetch_command(subcommand).run_from_argv(self.argv)
          File "python3.4/site-packages/django/core/management/base.py", line 390, in run_from_argv
            self.execute(*args, **cmd_options)
          File "python3.4/site-packages/django/core/management/base.py", line 441, in execute
            output = self.handle(*args, **options)
          File "python3.4/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 168, in handle
            collected = self.collect()
          File "python3.4/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 114, in collect
            for original_path, processed_path, processed in processor:
          File "python3.4/site-packages/pipeline/storage.py", line 26, in post_process
            packager.pack_stylesheets(package)
          File "python3.4/site-packages/pipeline/packager.py", line 96, in pack_stylesheets
            variant=package.variant, **kwargs)
          File "python3.4/site-packages/pipeline/packager.py", line 105, in pack
            paths = self.compile(package.paths, force=True)
          File "python3.4/site-packages/pipeline/packager.py", line 99, in compile
            return self.compiler.compile(paths, force=force)
          File "python3.4/site-packages/pipeline/compilers/__init__.py", line 56, in compile
            return list(executor.map(_compile, paths))
          File "/usr/local/lib/python3.4/concurrent/futures/_base.py", line 549, in result_iterator
            yield future.result()
          File "/usr/local/lib/python3.4/concurrent/futures/_base.py", line 402, in result
            return self.__get_result()
          File "/usr/local/lib/python3.4/concurrent/futures/_base.py", line 354, in __get_result
            raise self._exception
          File "/usr/local/lib/python3.4/concurrent/futures/thread.py", line 54, in run
            result = self.fn(*self.args, **self.kwargs)
          File "python3.4/site-packages/pipeline/compilers/__init__.py", line 42, in _compile
            outdated = compiler.is_outdated(input_path, output_path)
          File "python3.4/site-packages/pipeline/compilers/__init__.py", line 85, in is_outdated
            return self.storage.modified_time(infile) > self.storage.modified_time(outfile)
          File "python3.4/site-packages/storages/backends/s3boto.py", line 480, in modified_time
            return parse_ts(entry.last_modified)
        AttributeError: 'NoneType' object has no attribute 'last_modified'
        

        直到遇到this solution,我才开始找到适合我的方法。这是我最终使用的存储,它将文件保存在本地以及 S3 中,这让我通过了所有错误:

        from django.contrib.staticfiles.storage import ManifestFilesMixin
        from django.core.files.storage import get_storage_class
        from pipeline.storage import PipelineMixin
        from storages.backends.s3boto import S3BotoStorage
        
        
        class StaticStorage(PipelineMixin, ManifestFilesMixin, S3BotoStorage):
            """Custom storage for static content."""
        
            def __init__(self, *args, **kwargs):
                super(StaticStorage, self).__init__(*args, **kwargs)
                self.local_storage = get_storage_class(
                    'django.contrib.staticfiles.storage.StaticFilesStorage')()
        
            def save(self, name, content):
                name = super(StaticStorage, self).save(name, content)
                self.local_storage._save(name, content)
                return name
        

        【讨论】:

        • 所以这最初是针对旧版本的管道。不幸的是,我似乎无法编辑问题标题以放入版本并明确说明:/
        猜你喜欢
        • 2012-08-02
        • 1970-01-01
        • 2017-08-23
        • 1970-01-01
        • 1970-01-01
        • 2013-07-06
        • 1970-01-01
        • 1970-01-01
        • 2015-10-24
        相关资源
        最近更新 更多