柔性环境中的GDAL

Question

在应用引擎灵活环境中，我正在尝试使用 GIS 扩展在 postgresql db 上运行 django。可以在本地运行，最终成功部署到云端。但是，当从云运行时，我得到一个“错误：服务器错误”，并且日志文件具有如下所示的跟踪。

基本上，它说它无法读取文件夹“/env/lib/python3.5/site-packages/django/contrib/gis/gdal”中的文件。该文件夹肯定在本地。这是完整的跟踪：

Traceback (most recent call last):
 File "/env/lib/python3.5/site-packages/gunicorn/arbiter.py", line 578, in spawn_worker
  worker.init_process()
 File "/env/lib/python3.5/site-packages/gunicorn/workers/base.py", line 126, in init_process
  self.load_wsgi()
 File "/env/lib/python3.5/site-packages/gunicorn/workers/base.py", line 135, in load_wsgi
  self.wsgi = self.app.wsgi()
 File "/env/lib/python3.5/site-packages/gunicorn/app/base.py", line 67, in wsgi
  self.callable = self.load()
 File "/env/lib/python3.5/site-packages/gunicorn/app/wsgiapp.py", line 65, in load
  return self.load_wsgiapp()
 File "/env/lib/python3.5/site-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
  return util.import_app(self.app_uri)
 File "/env/lib/python3.5/site-packages/gunicorn/util.py", line 352, in import_app
  __import__(module)
 File "/home/vmagent/app/mysite/wsgi.py", line 22, in <module>
  application = get_wsgi_application()
 File "/env/lib/python3.5/site-packages/django/core/wsgi.py", line 13, in get_wsgi_application
  django.setup(set_prefix=False)
 File "/env/lib/python3.5/site-packages/django/__init__.py", line 27, in setup
  apps.populate(settings.INSTALLED_APPS)
 File "/env/lib/python3.5/site-packages/django/apps/registry.py", line 108, in populate
  app_config.import_models()
 File "/env/lib/python3.5/site-packages/django/apps/config.py", line 202, in import_models
  self.models_module = import_module(models_module_name)
 File "/env/lib/python3.5/importlib/__init__.py", line 126, in import_module
  return _bootstrap._gcd_import(name[level:], package, level)
 File "/env/lib/python3.5/site-packages/django/contrib/auth/models.py", line 4, in <module>
  from django.contrib.auth.base_user import AbstractBaseUser, BaseUserManager
 File "/env/lib/python3.5/site-packages/django/contrib/auth/base_user.py", line 52, in <module>
  class AbstractBaseUser(models.Model):
 File "/env/lib/python3.5/site-packages/django/db/models/base.py", line 124, in __new__
  new_class.add_to_class('_meta', Options(meta, app_label))
 File "/env/lib/python3.5/site-packages/django/db/models/base.py", line 331, in add_to_class
  value.contribute_to_class(cls, name)
 File "/env/lib/python3.5/site-packages/django/db/models/options.py", line 214, in contribute_to_class
  self.db_table = truncate_name(self.db_table, connection.ops.max_name_length())
 File "/env/lib/python3.5/site-packages/django/db/__init__.py", line 33, in __getattr__
  return getattr(connections[DEFAULT_DB_ALIAS], item)
 File "/env/lib/python3.5/site-packages/django/db/utils.py", line 211, in __getitem__
  backend = load_backend(db['ENGINE'])
 File "/env/lib/python3.5/site-packages/django/db/utils.py", line 115, in load_backend
  return import_module('%s.base' % backend_name)
 File "/env/lib/python3.5/importlib/__init__.py", line 126, in import_module
  return _bootstrap._gcd_import(name[level:], package, level)
 File "/env/lib/python3.5/site-packages/django/contrib/gis/db/backends/postgis/base.py", line 6, in <module>
  from .features import DatabaseFeatures
 File "/env/lib/python3.5/site-packages/django/contrib/gis/db/backends/postgis/features.py", line 1, in <module>
  from django.contrib.gis.db.backends.base.features import BaseSpatialFeatures
 File "/env/lib/python3.5/site-packages/django/contrib/gis/db/backends/base/features.py", line 4, in <module>
  from django.contrib.gis.db.models import aggregates
 File "/env/lib/python3.5/site-packages/django/contrib/gis/db/models/__init__.py", line 3, in <module>
  from django.contrib.gis.db.models.aggregates import * # NOQA
 File "/env/lib/python3.5/site-packages/django/contrib/gis/db/models/aggregates.py", line 1, in <module>
  from django.contrib.gis.db.models.fields import ExtentField
 File "/env/lib/python3.5/site-packages/django/contrib/gis/db/models/fields.py", line 3, in <module>
  from django.contrib.gis import forms, gdal
 File "/env/lib/python3.5/site-packages/django/contrib/gis/forms/__init__.py", line 3, in <module>
  from .fields import ( # NOQA
 File "/env/lib/python3.5/site-packages/django/contrib/gis/forms/fields.py", line 4, in <module>
  from django.contrib.gis.geos import GEOSException, GEOSGeometry
 File "/env/lib/python3.5/site-packages/django/contrib/gis/geos/__init__.py", line 5, in <module>
  from .collections import ( # NOQA
 File "/env/lib/python3.5/site-packages/django/contrib/gis/geos/collections.py", line 11, in <module>
  from django.contrib.gis.geos.geometry import GEOSGeometry, LinearGeometryMixin
 File "/env/lib/python3.5/site-packages/django/contrib/gis/geos/geometry.py", line 11, in <module>
  from django.contrib.gis import gdal
 File "/env/lib/python3.5/site-packages/django/contrib/gis/gdal/__init__.py", line 28, in <module>
  from django.contrib.gis.gdal.datasource import DataSource
 File "/env/lib/python3.5/site-packages/django/contrib/gis/gdal/datasource.py", line 39, in <module>
  from django.contrib.gis.gdal.driver import Driver
 File "/env/lib/python3.5/site-packages/django/contrib/gis/gdal/driver.py", line 5, in <module>
  from django.contrib.gis.gdal.prototypes import ds as vcapi, raster as rcapi
 File "/env/lib/python3.5/site-packages/django/contrib/gis/gdal/prototypes/ds.py", line 9, in <module>
  from django.contrib.gis.gdal.libgdal import GDAL_VERSION, lgdal
 File "/env/lib/python3.5/site-packages/django/contrib/gis/gdal/libgdal.py", line 49, in <module>
  lgdal = CDLL(lib_path)
 File "/opt/python3.5/lib/python3.5/ctypes/__init__.py", line 351, in __init__
  self._handle = _dlopen(self._name, mode)
OSError: /env/lib/python3.5/site-packages/django/contrib/gis/gdal: cannot read file data: Is a directory

在 ubuntu 16.04 上开发。

我从 virtualenv 中的简单“pip install django”开始，它创建了“django/contrib/gis/gdal”文件夹，该文件夹说在部署时丢失。我尝试了其他几种安装 GDAL 的方法，以确保没有丢失的头文件。还升级到了 Google Cloud SDK 171 和 gsutil 4.27。曾尝试在 requirements.txt 中包含 GDAL==2.1.3，但随后出现构建错误，因此将其删除。我还尝试了“pip freeze”以确保 requirements.txt 包含所有需要的包。

最后，我按照本教程从头开始：https://cloud.google.com/python/django/flexible-environment。我没有在本地 postgreSQL 数据库上进行测试，而是直接访问了之前使用的 Cloud SQL 实例（其中包含所需的表）。而这次我用的是python2.7。

它在本地仍然可以正常运行，但现在在堆栈跟踪中缺少文件夹“/env/local/lib/python2.7/site-packages/django/contrib/gis/gdal”。这是完整的跟踪：

Traceback (most recent call last):
 File "/env/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 578, in spawn_worker
  worker.init_process()
 File "/env/local/lib/python2.7/site-packages/gunicorn/workers/base.py", line 126, in init_process
  self.load_wsgi()
 File "/env/local/lib/python2.7/site-packages/gunicorn/workers/base.py", line 135, in load_wsgi
  self.wsgi = self.app.wsgi()
 File "/env/local/lib/python2.7/site-packages/gunicorn/app/base.py", line 67, in wsgi
  self.callable = self.load()
 File "/env/local/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 65, in load
  return self.load_wsgiapp()
 File "/env/local/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
  return util.import_app(self.app_uri)
 File "/env/local/lib/python2.7/site-packages/gunicorn/util.py", line 352, in import_app
  __import__(module)
 File "/home/vmagent/app/mysite/wsgi.py", line 22, in <module>
  application = get_wsgi_application()
 File "/env/local/lib/python2.7/site-packages/django/core/wsgi.py", line 13, in get_wsgi_application
  django.setup(set_prefix=False)
 File "/env/local/lib/python2.7/site-packages/django/__init__.py", line 27, in setup
  apps.populate(settings.INSTALLED_APPS)
 File "/env/local/lib/python2.7/site-packages/django/apps/registry.py", line 108, in populate
  app_config.import_models()
 File "/env/local/lib/python2.7/site-packages/django/apps/config.py", line 202, in import_models
  self.models_module = import_module(models_module_name)
 File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
  __import__(name)
 File "/env/local/lib/python2.7/site-packages/django/contrib/auth/models.py", line 4, in <module>
  from django.contrib.auth.base_user import AbstractBaseUser, BaseUserManager
 File "/env/local/lib/python2.7/site-packages/django/contrib/auth/base_user.py", line 52, in <module>
  class AbstractBaseUser(models.Model):
 File "/env/local/lib/python2.7/site-packages/django/db/models/base.py", line 124, in __new__
  new_class.add_to_class('_meta', Options(meta, app_label))
 File "/env/local/lib/python2.7/site-packages/django/db/models/base.py", line 331, in add_to_class
  value.contribute_to_class(cls, name)
 File "/env/local/lib/python2.7/site-packages/django/db/models/options.py", line 214, in contribute_to_class
  self.db_table = truncate_name(self.db_table, connection.ops.max_name_length())
 File "/env/local/lib/python2.7/site-packages/django/db/__init__.py", line 33, in __getattr__
  return getattr(connections[DEFAULT_DB_ALIAS], item)
 File "/env/local/lib/python2.7/site-packages/django/db/utils.py", line 211, in __getitem__
  backend = load_backend(db['ENGINE'])
 File "/env/local/lib/python2.7/site-packages/django/db/utils.py", line 115, in load_backend
  return import_module('%s.base' % backend_name)
 File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
  __import__(name)
 File "/env/local/lib/python2.7/site-packages/django/contrib/gis/db/backends/postgis/base.py", line 6, in <module>
  from .features import DatabaseFeatures
 File "/env/local/lib/python2.7/site-packages/django/contrib/gis/db/backends/postgis/features.py", line 1, in <module>
  from django.contrib.gis.db.backends.base.features import BaseSpatialFeatures
 File "/env/local/lib/python2.7/site-packages/django/contrib/gis/db/backends/base/features.py", line 4, in <module>
  from django.contrib.gis.db.models import aggregates
 File "/env/local/lib/python2.7/site-packages/django/contrib/gis/db/models/__init__.py", line 3, in <module>
  from django.contrib.gis.db.models.aggregates import * # NOQA
 File "/env/local/lib/python2.7/site-packages/django/contrib/gis/db/models/aggregates.py", line 1, in <module>
  from django.contrib.gis.db.models.fields import ExtentField
 File "/env/local/lib/python2.7/site-packages/django/contrib/gis/db/models/fields.py", line 3, in <module>
  from django.contrib.gis import forms, gdal
 File "/env/local/lib/python2.7/site-packages/django/contrib/gis/forms/__init__.py", line 3, in <module>
  from .fields import ( # NOQA
 File "/env/local/lib/python2.7/site-packages/django/contrib/gis/forms/fields.py", line 4, in <module>
  from django.contrib.gis.geos import GEOSException, GEOSGeometry
 File "/env/local/lib/python2.7/site-packages/django/contrib/gis/geos/__init__.py", line 5, in <module>
  from .collections import ( # NOQA
 File "/env/local/lib/python2.7/site-packages/django/contrib/gis/geos/collections.py", line 11, in <module>
  from django.contrib.gis.geos.geometry import GEOSGeometry, LinearGeometryMixin
 File "/env/local/lib/python2.7/site-packages/django/contrib/gis/geos/geometry.py", line 11, in <module>
  from django.contrib.gis import gdal
 File "/env/local/lib/python2.7/site-packages/django/contrib/gis/gdal/__init__.py", line 28, in <module>
  from django.contrib.gis.gdal.datasource import DataSource
 File "/env/local/lib/python2.7/site-packages/django/contrib/gis/gdal/datasource.py", line 39, in <module>
  from django.contrib.gis.gdal.driver import Driver
 File "/env/local/lib/python2.7/site-packages/django/contrib/gis/gdal/driver.py", line 5, in <module>
  from django.contrib.gis.gdal.prototypes import ds as vcapi, raster as rcapi
 File "/env/local/lib/python2.7/site-packages/django/contrib/gis/gdal/prototypes/ds.py", line 9, in <module>
  from django.contrib.gis.gdal.libgdal import GDAL_VERSION, lgdal
 File "/env/local/lib/python2.7/site-packages/django/contrib/gis/gdal/libgdal.py", line 45, in <module>
  % '", "'.join(lib_names)
ImproperlyConfigured: Could not find the GDAL library (tried "gdal", "GDAL", "gdal2.1.0", "gdal2.0.0", "gdal1.11.0", "gdal1.10.0", "gdal1.9.0"). Is GDAL installed? If it is, try setting GDAL_LIBRARY_PATH in your settings.

非常感谢任何帮助。

丹

Answer 1

应用引擎柔性环境附带此处列出的 C 库：https://cloud.google.com/appengine/docs/flexible/python/runtime

由于 GDAL 是一个不在标准运行时中的 C 库，因此我必须使用 Docker 容器构建自定义环境。诀窍是在 requirements.txt 和 Python3 中安装 GDAL 以及必要的 python 库，然后通过 gunicorn 启动 django 应用程序。

这是我的 Dockerfile：

FROM gcr.io/google-appengine/python

RUN apt-get update && apt-get install -y \
  binutils \
  gdal-bin \
  python-gdal

# Create a virtualenv for dependencies. This isolates these packages from
# system-level packages.
RUN virtualenv /env -p python3.6

# Setting these environment variables are the same as running
# source /env/bin/activate.
ENV VIRTUAL_ENV /env
ENV PATH /env/bin:$PATH

# Copy the application's requirements.txt and run pip to install all
# dependencies into the virtualenv.
ADD requirements.txt /app/requirements.txt
RUN pip install -r /app/requirements.txt
# Add the application source code.
ADD . /app

# Run a WSGI server to serve the application. gunicorn must be declared as
# a dependency in requirements.txt.
CMD gunicorn -b :$PORT mysite.wsgi

我的 app.yaml：

# [START runtime]
runtime: custom
#python
env: flex
entrypoint: gunicorn -b :$PORT mysite.wsgi

beta_settings:
    cloud_sql_instances: <your-db-instance-identifying-string>

runtime_config:
  python_version: 3
# [END runtime]

以及最低要求.txt（根据需要添加）：

Django==1.11.4
mysqlclient==1.3.10
wheel==0.29.0
gunicorn==19.7.1
psycopg2==2.7.3

希望这对某人有所帮助，

丹