GCP App Engine 上的 python-magic - 部署失败

Question

我正在开发将部署到 GCP App Engine 的 Python 应用程序。
我尝试使用python-magic 功能。
我已将import magic 添加到我的代码中，并将python-magic 添加到requirements.txt 文件中。
但是当我尝试将代码部署到 App Engine 时，它​​会失败并出现以下错误：

[2020-02-03 15:35:23 +0000] [8] [ERROR] Exception in worker process
Traceback (most recent call last):
  File "/env/lib/python3.6/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
    worker.init_process()
  File "/env/lib/python3.6/site-packages/gunicorn/workers/base.py", line 129, in init_process
    self.load_wsgi()
  File "/env/lib/python3.6/site-packages/gunicorn/workers/base.py", line 138, in load_wsgi
    self.wsgi = self.app.wsgi()
  File "/env/lib/python3.6/site-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
  File "/env/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 52, in load
    return self.load_wsgiapp()
  File "/env/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 41, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/env/lib/python3.6/site-packages/gunicorn/util.py", line 350, in import_app
    __import__(module)
  File "/home/vmagent/app/main.py", line 11, in <module>
    import magic
  File "/env/lib/python3.6/site-packages/magic.py", line 181, in <module>
    raise ImportError('failed to find libmagic.  Check your installation')
ImportError: failed to find libmagic.  Check your installation
[2020-02-03 15:35:23 +0000] [8] [INFO] Worker exiting (pid: 8)
[2020-02-03 15:35:23 +0000] [1] [INFO] Shutting down: Master
[2020-02-03 15:35:23 +0000] [1] [INFO] Reason: Worker failed to boot.

为了完成这项工作，我缺少什么设置？

---

编辑：共享最小设置以重现此错误

main.py

# -*- coding: utf-8 -*-

import json
import base64
import magic

import logging
logging.basicConfig(level=logging.DEBUG)

from flask import Flask, request, abort

app = Flask(__name__)

@app.route('/parse', methods=['POST'])
def parse():

    try:
        #input should be a b64 encoded byte string
        input = request.get_json()['input']
    except Exception as e:
        abort(400, 'An error occured while reading in parameters: {}'.format(str(e)))

    decoded_input_data = base64.decodebytes(input)
    mime = magic.Magic(mime=True)
    file_ext = mime.from_buffer(decoded_input_data)

    returnData = {}
    returnData['Status'] = 200
    returnData['file_ext'] = file_ext

    return json.dumps(returnData), 200

@app.errorhandler(Exception)
def error(e):
    logging.exception(str(e))
    return json.dumps({'Status': e.code,
                       'Message': e.description}), e.code

if __name__ == '__main__':
    # This is used when running locally. Gunicorn is used to run the
    # application on Google App Engine. See entrypoint in app.yaml.
    app.run(host='127.0.0.1', port=8080, debug=True)

requirements.txt

Flask==1.0.2
gunicorn==19.9.0
google-cloud-vision==0.38.0
google-cloud-bigquery==1.11.2
google-cloud-storage==1.15.0
requests-toolbelt==0.9.1
protobuf==3.6.0
python-magic==0.4.15

app.yaml

runtime: python
env: flex
service: myservicename
entrypoint: gunicorn -b :$PORT main:app --timeout 240 --limit-request-line 0

runtime_config:
  python_version: 3

Answer 1

如果您想使用python-magic 库，您可以按照下一个过程：

App Engine 标准

1. 使用 venv module 在项目外部的目录中创建一个隔离的 Python 环境并通过以下方式激活它：

python3 -m venv env
source env/bin/activate

1. 使用 pip 安装 python-magic 库，方法是运行：

pip install python-magic

1. 将库添加到您的 requirements.txt 文件中。例如，如果您正在运行官方文档中的Quickstart，您的 requirements.txt 文件应如下所示：

Flask==1.1.1
python-magic==0.4.15

4. 在应用程序代码的相关模块上导入库。例如，如果您正在运行官方文档中的 Quickstart，则 main.py 文件的相关部分应如下所示：

...
# [START gae_python37_app]
from flask import Flask
import magic

# If `entrypoint` is not defined in app.yaml, App Engine will look for an app
# called `app` in `main.py`.
app = Flask(__name__)
...

App Engine 灵活 libmagic C 库未预安装在 flex 环境中（here 是已安装库的列表）。因此，您需要创建一个custom runtime。

将您的 app.yaml 更改为以下内容：

runtime: custom
env: flex

并将相应的 Dockerfile 添加到您的项目目录中。例如：

FROM python:3.7
WORKDIR /app
COPY . /app
RUN apt-get update &&\
    apt-get install -y libmagic-dev
RUN pip install -r requirements.txt
EXPOSE 8080
CMD ["gunicorn", "main:app", "-b", ":8080", "--timeout", "300"]

应该足以成功部署应用程序。

请注意，Dockerfile 只是安装 libmagic C 库，然后安装您的 requirements.txt，最后在 App Engine 所需的端口上运行 gunicorn。但请随时根据您的需要进行调整。