docker build 产生损坏的python库安装

Question

很多时候，我的docker build 运行会导致python 库安装损坏。有时，这会导致 docker build 本身在库安装步骤中失败。但更严重的是，它可能会静默失败，并且只有在运行应用程序时才会看到错误。损坏的确切 python 库因构建而异；重新安装相同版本的损坏库将修复它。使用 --no-cache 似乎更好，但即使使用此构建选项仍然会发生。

一个例子如下所示：

$ docker build --no-cache --force-rm --rm -f Dockerfile . -t mypy                           

$ docker run --rm -it --entrypoint /bin/bash mypy                   
root:/# python
Python 3.6.6 (default, Jul 31 2018, 22:36:30)
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import statsmodels.api as sm_api
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/site-packages/statsmodels/__init__.py", line 8, in <module>
    from .tools.sm_exceptions import (ConvergenceWarning, CacheWriteWarning,
  File "/usr/local/lib/python3.6/site-packages/statsmodels/tools/__init__.py", line 1, in <module>
    from .tools import add_constant, categorical
  File "/usr/local/lib/python3.6/site-packages/statsmodels/tools/tools.py", line 11, in <module>
    from statsmodels.datasets import webuse
  File "/usr/local/lib/python3.6/site-packages/statsmodels/datasets/__init__.py", line 5, in <module>
    from . import (anes96, cancer, committee, ccard, copper, cpunish, elnino,
  File "/usr/local/lib/python3.6/site-packages/statsmodels/datasets/scotland/__init__.py", line 1
    6,�R��A�L
     ^
SyntaxError: invalid syntax

root:/# pip list | grep statsmodels
statsmodels            0.8.0

root:/# pip uninstall statsmodels

root:/# pip install statsmodels==0.8.0
Collecting statsmodels==0.8.0
  Using cached https://files.pythonhosted.org/packages/0d/e9/70d80b48c8c52a8de3ec7cd50e2aa2b1f3cf3f95e42b15fdcb59bd7189f3/statsmodels-0.8.0-cp36-cp36m-manylinux1_x86_64.whl
Requirement already satisfied: pandas in /usr/local/lib/python3.6/site-packages (from statsmodels==0.8.0) (0.19.2)
Requirement already satisfied: scipy in /usr/local/lib/python3.6/site-packages (from statsmodels==0.8.0) (1.1.0)
Requirement already satisfied: patsy in /usr/local/lib/python3.6/site-packages (from statsmodels==0.8.0) (0.5.0)
Requirement already satisfied: numpy>=1.7.0 in /usr/local/lib/python3.6/site-packages (from pandas->statsmodels==0.8.0) (1.12.1)
Requirement already satisfied: python-dateutil>=2 in /usr/local/lib/python3.6/site-packages (from pandas->statsmodels==0.8.0) (2.7.3)
Requirement already satisfied: pytz>=2011k in /usr/local/lib/python3.6/site-packages (from pandas->statsmodels==0.8.0) (2018.5)
Requirement already satisfied: six in /usr/local/lib/python3.6/site-packages (from patsy->statsmodels==0.8.0) (1.11.0)
Installing collected packages: statsmodels
Successfully installed statsmodels-0.8.0
root:/# python
Python 3.6.6 (default, Jul 31 2018, 22:36:30)
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import statsmodels.api as sm_api
>>>
root:/#

而且 Dockerfile 相当简单，例如：

FROM python:3.6-slim
ARG app_dir=/root/myapp
ARG aws_access_key_id
ARG aws_secret_access_key

RUN test -n "$aws_access_key_id" && test -n "$aws_secret_access_key"

RUN mkdir -p ${app_dir} /root/.aws

RUN echo "[default]" > /root/.aws/credentials && \
  echo "aws_access_key_id = $aws_access_key_id" >> /root/.aws/credentials && \
  echo "aws_secret_access_key = $aws_secret_access_key" >> /root/.aws/credentials

ADD requirements.txt ${app_dir}/requirements.txt

RUN apt-get update -qqy && apt-get install -qqy --no-install-recommends \
  build-essential libopenblas-dev gfortran procps less nano wget

RUN pip install -r ${app_dir}/requirements.txt

这些静默错误相当危险，因为我们没有使用针对 docker 映像的应用特定单元测试来执行广泛的验证。我们希望看到可重现的图像构建。如何预防这些错误？

Answer 1

我没有答案，而是一个建议。

如果依赖项只是构建依赖项而不是运行时依赖项，则使用多阶段构建。

通过这种方式，您可以在构建的第一步生产轮子，然后在第二步安装轮子。如果创建的轮子损坏，您可能无法安装它们，不过我不确定。

基于您的 Dockerfile 的示例：

FROM python:3.6-slim as builder

ADD requirements.txt requirements.txt

RUN apt-get update -qqy && apt-get install -qqy --no-install-recommends \
  build-essential libopenblas-dev gfortran gcc gfortran g++  

RUN mkdir /wheels && \
    pip install -U pip wheel setuptools && pip wheel -r requirements.txt --wheel-dir=/wheels

FROM python3.6-slim as prod

ARG app_dir=/root/myapp
ARG aws_access_key_id
ARG aws_secret_access_key

RUN echo "[default]" > /root/.aws/credentials && \
  echo "aws_access_key_id = $aws_access_key_id" >> /root/.aws/credentials && \
  echo "aws_secret_access_key = $aws_secret_access_key" >> /root/.aws/credentials

COPY --from=builder /wheels /tmp/wheels
ADD requirements.txt requirements.txt

RUN pip install --no-cache-dir --no-index --find-links=/tmp/wheels/. -r requirements.txt 

WORKDIR $app_dir

如果您不介意更长的构建过程，您可以在 prod 和 builder 之间添加一个步骤，这将安装并导入所有模块以测试导入是否失败：

FROM python:3.6-slim as builder

ADD requirements.txt requirements.txt

RUN apt-get update -qqy && apt-get install -qqy --no-install-recommends \
  build-essential libopenblas-dev gfortran gcc gfortran g++  

RUN mkdir /wheels && \
    pip install -U pip wheel setuptools && pip wheel -r requirements.txt --wheel-dir=/wheels

FROM python3.6-slim as test

COPY --from=builder /wheels /tmp/wheels
ADD requirements.txt requirements.txt

RUN pip install --no-cache-dir --no-index --find-links=/tmp/wheels/. -r requirements.txt 
# Will fail if one of requirements cannot be imported
RUN python -c 'f = open("requirements.txt") ; modules = [line.strip() for line in f.readlines() if line.strip()] ; f.close() ; list(map(__import__, modules))'

FROM python3.6-slim as prod

ARG app_dir=/root/myapp
ARG aws_access_key_id
ARG aws_secret_access_key

RUN echo "[default]" > /root/.aws/credentials && \
  echo "aws_access_key_id = $aws_access_key_id" >> /root/.aws/credentials && \
  echo "aws_secret_access_key = $aws_secret_access_key" >> /root/.aws/credentials

COPY --from=builder /wheels /tmp/wheels
ADD requirements.txt requirements.txt

RUN pip install --no-cache-dir --no-index --find-links=/tmp/wheels/. -r requirements.txt 

WORKDIR $app_dir

这是一个相当丑陋的提议，但也许它可以帮助你......