来自 alpine 包 repo 的 Numpy 无法导入 c 扩展

Question

我正在制作一个需要 pandas 和 numpy 的 docker 映像，但通过 pip 安装大约需要 20 分钟，这对于我的用例来说太长了。然后我选择从 alpine 包 repo 安装 pandas 和 numpy，但似乎无法正确导入 numpy。

这是我的 Dockerfile：

# syntax=docker/dockerfile:experimental
FROM python:3.9.5-alpine as base

FROM base as builder
RUN apk add build-base gcc musl-dev

RUN --mount=type=cache,target=/root/.cache/pip \
    pip install --target="/install" django

FROM base
RUN apk add py3-pandas py3-numpy

COPY --from=builder /install /usr/local/lib/python3.9/site-packages

ENV PYTHONPATH "${PYTHONPATH}:/usr/lib/python3.9/site-packages"

CMD ["python"]

当我尝试导入依赖于 numpy 的 pandas 时，出现错误：

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.9/site-packages/pandas/__init__.py", line 16, in <module>
    raise ImportError(
ImportError: Unable to import required dependencies:
numpy: 

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

    https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

  * The Python version is: Python3.9 from "/usr/local/bin/python"
  * The NumPy version is: "1.20.3"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: No module named 'numpy.core._multiarray_umath'

如果我导入 numpy 会出现错误：

Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/numpy/core/__init__.py", line 22, in <module>
    from . import multiarray
  File "/usr/lib/python3.9/site-packages/numpy/core/multiarray.py", line 12, in <module>
    from . import overrides
  File "/usr/lib/python3.9/site-packages/numpy/core/overrides.py", line 7, in <module>
    from numpy.core._multiarray_umath import (
ModuleNotFoundError: No module named 'numpy.core._multiarray_umath'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.9/site-packages/numpy/__init__.py", line 145, in <module>
    from . import core
  File "/usr/lib/python3.9/site-packages/numpy/core/__init__.py", line 48, in <module>
    raise ImportError(msg)
ImportError: 

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

    https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

  * The Python version is: Python3.9 from "/usr/local/bin/python"
  * The NumPy version is: "1.20.3"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: No module named 'numpy.core._multiarray_umath'

我已经束手无策，试图弄清楚我错过了什么和做错了什么。我已经尝试了错误跟踪给出的 url 中的故障排除提示，但似乎没有任何解决问题的方法。

非常感谢任何帮助。

Answer 1

我知道这个问题已经有一段时间了，您可能已经找到了解决方案，或者从 Alpine 转移到了不同​​的发行版。但我遇到了同样的问题，这是我搜索时出现的第一件事。因此，在花费了几个小时并找到了解决方案之后，我认为值得在这里记录一下。

问题（显然）与 numpy 和 pandas 软件包有关。我使用了社区 repo 中的预制轮子，遇到了和你一样的问题。因此，显然，构建过程本身正在引入这个问题。具体来说，如果您查看，例如，在安装位置 (/usr/lib/python3.9/site-packages) 的 numpy/core 下，您会发现所有 C 扩展名中都有 .cpython-39-x86_64-linux-musl。因此，例如，您遇到问题的模块numpy.core._multiarray_umath 被命名为_multiarray_umath.cpython-39-x86_64-linux-musl.so，而不仅仅是_multiarray_umath.so。从这些文件名中删除 .cpython-39-x86_64-linux-musl 解决了这个问题（编辑：有关详细信息，请参阅附录）。

在安装py3-pandas 和py3-numpy 后，可以在您的Dockerfile 中添加以下行来修复它：

RUN find /usr/lib/python3.9/site-packages -iname "*.so" -exec sh -c 'x="{}"; mv "$x" "${x/cpython-39-x86_64-linux-musl./}"' \;

P.S.：进一步调查问题后，我发现了罪魁祸首：由于某种原因，在 Alpine 下运行的 Python 认为其完整平台扩展后缀（可从 importlib.machinery.EXTENSION_SUFFIXES 获得）应该是 cpython-39-x86_64-linux-gnu.so，而不是 @987654336 @。我不相信它是用 glibc 构建的，但谁知道呢。因此，您只需将上述共享对象名称中的musl 更改为gnu，它也可以正常工作。不知道为什么构建期间生成的扩展后缀与 Python 在运行时使用的不同。