无服务器python打包numpy依赖错误

Question

我在从已部署的 Python3.7 Lambda 函数调用函数时遇到问题，从错误消息来看，该函数与 numpy 相关。该问题表明无法导入包，尽管尝试了我读过的许多解决方案，但我没有取得任何成功，我想知道接下来要测试什么或如何进一步调试。

我尝试了以下方法：

1. 安装Docker，添加插件serverless-python-requirements，在yml中配置
2. 在app目录下安装包进行捆绑部署，pip install -t src/vendor -r requirements.txt --no-cache-dir
3. 卸载 setuptools 和 numpy 并按此顺序重新安装

错误信息（运行sls invoke -f auth后显示）：

{
    "errorMessage": "Unable to import module 'data': Unable to import required dependencies:\nnumpy: \n\nIMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!\n\nImporting the numpy c-extensions failed.\n- Try uninstalling and reinstalling numpy.\n- If you have already done that, then:\n  1. Check that you expected to use Python3.7 from \"/var/lang/bin/python3.7\",\n     and that you have no directories in your PATH or PYTHONPATH that can\n     interfere with the Python and numpy version \"1.18.1\" you're trying to use.\n  2. If (1) looks fine, you can open a new issue at\n     https://github.com/numpy/numpy/issues.  Please include details on:\n     - how you installed Python\n     - how you installed numpy\n     - your operating system\n     - whether or not you have multiple versions of Python installed\n     - if you built from source, your compiler versions and ideally a build log\n\n- If you're working with a numpy git repository, try `git clean -xdf`\n  (removes all files not under version control) and rebuild numpy.\n\nNote: this error has many possible causes, so please don't comment on\nan existing issue about this - open a new one instead.\n\nOriginal error was: No module named 'numpy.core._multiarray_umath'\n",
    "errorType": "Runtime.ImportModuleError"
}

提供的是我的设置：

OS: Mac OS X
Local Python: /Users/me/miniconda3/bin/python
Local Python version: Python 3.7.4

无服务器环境信息（运行时 = Python3.7）：

 Operating System:          darwin
 Node Version:              12.14.0
 Framework Version:         1.67.3
 Plugin Version:            3.6.6
 SDK Version:               2.3.0
 Components Version:        2.29.1

Docker：

Docker version 19.03.13, build 4484c46d9d

serverless.yml：

service: understand-your-sleep-api

plugins:
  - serverless-python-requirements
  - serverless-offline-python

custom:
  pythonRequirements:
    dockerizePip: true # non-linux
    slim: true
    useStaticCache: false
    useDownloadCache: false
    invalidateCaches: true

provider:
  name: aws
  runtime: python3.7
  stage: ${opt:stage, 'dev'}
  region: us-east-1
  iamRoleStatements:
    - Effect: Allow
      Action:
        - ssm:GetParameter
      Resource: "arn:aws:ssm:us-east-1:*id*:parameter/*"
  environment:
    STAGE: ${self:provider.stage}

functions:
  auth:
    handler: data.auth
    events:
      - http:
          path: /auth
          method: get
          cors: true

package:
  exclude:
    - env.yml
    - node_modules/**

requirements.txt：

pandas==1.0.0
fitbit==0.3.1
oauthlib==3.1.0
requests==2.22.0
requests-oauthlib==1.3.0

数据.py：

import sys
sys.path.insert(0, 'src/vendor') # Location of packages that follow
import json
from datetime import timedelta, datetime, date
import math
import pandas as pd
from requests_oauthlib import OAuth2Session
from urllib.parse import urlparse, parse_qs
import fitbit
import requests
import webbrowser
import base64
import os 
import logging

def auth(event, context):
    ...

Answer 1

• 使用 lambda 层来打包你的所有需求，确保你在 requirements.txt 文件中有 numpy。尝试一次。
• 仅当 serverless-python-requirements 插件列在插件部分时才有效。
• 用此替换您的自定义键，并为函数提供使用该层的参考

custom:
  pythonRequirements:
    layer: true
functions:
  auth:
    handler: data.auth
    events:
      - http:
          path: /auth
          method: get
          cors: true
    layers:
      - { Ref: PythonRequirementsLambdaLayer}

Answer 2

由于您使用的是serverless-python-requirements 插件，它会为您打包库。换句话说，您无需手动执行所有pip install -t src/vendor -r requirements.txt --no-cache-dir 操作。

为解决您的问题，请删除src/vendor 和data.py 中的以下两行：

import sys
sys.path.insert(0, 'src/vendor') # Location of packages that follow

然后坐下来，让serverless-python-requirements 为您完成工作。

Answer 3

我检查了zipinfo .requirements.zip，发现加载的是 macos dynlib 而不是 linux so 文件

我通过使用 dockerizePip: non-linux 解决了这个问题

请注意，如果在工作目录中已存在 .requirements.zip，则不会触发此操作，因此 git clean -xfd 在运行 sls deploy 之前