【问题标题】:Using confluent-kafka python client in alpine container在 alpine 容器中使用 confluent-kafka python 客户端
【发布时间】:2017-04-27 17:37:50
【问题描述】:

我正在尝试运行一个与 kafka 通信的简单 python 应用程序。我正在寻找使用高山容器。这是我当前的 dockerfile(它不是最佳的......只是试图让事情暂时正常)。

FROM python:3.6-alpine
MAINTAINER Ashic Mahtab (ashic@live.com)

RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app

RUN echo "http://dl-cdn.alpinelinux.org/alpine/edge/community" >> /etc/apk/repositories
RUN apk update && apk --no-cache add librdkafka


COPY requirements.txt /usr/src/app/
RUN pip install --no-cache-dir -r requirements.txt

COPY api /usr/src/app/api
COPY static /usr/src/app/static

CMD ["python", "api/index.py"]

需求文件中有confluent-kafka。构建失败

OK: 8784 distinct packages available
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/community/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/edge/community/x86_64/APKINDEX.tar.gz
ERROR: unsatisfiable constraints:
  so:libcrypto.so.41 (missing):
    required by:
                 librdkafka-0.9.4-r1[so:libcrypto.so.41]
                 librdkafka-0.9.4-r1[so:libcrypto.so.41]
                 librdkafka-0.9.4-r1[so:libcrypto.so.41]
  so:libssl.so.43 (missing):
    required by:
                 librdkafka-0.9.4-r1[so:libssl.so.43]
                 librdkafka-0.9.4-r1[so:libssl.so.43]
                 librdkafka-0.9.4-r1[so:libssl.so.43]

我的问题是 a) 有没有办法在不构建容器的情况下使其工作?如果我可以简单地将库复制到 alpine 就足够了。或者即使我可以复制 librdkafka。 b)如果没有,我怎样才能让 libssl 和 libcryto.so 工作?

【问题讨论】:

  • 适用于使用 confluent-kafka v1.4.0 的人。不要使用它.. 使用 v1.4.2。有一个问题是它不包含 requirements.txt。请参阅this 帖子。

标签: docker apache-kafka alpine confluent-platform kafka-python


【解决方案1】:

这里有一个让它工作的方法:

FROM python:3.6-alpine
MAINTAINER Ashic Mahtab (ashic@live.com)

RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app

RUN sed -i -e 's/v3\.4/edge/g' /etc/apk/repositories \
    && apk upgrade --update-cache --available \
    && apk --no-cache add librdkafka

COPY requirements.txt /usr/src/app/
RUN pip install --no-cache-dir -r requirements.txt

COPY api /usr/src/app/api
COPY static /usr/src/app/static

CMD ["python", "api/index.py"]

这里解释一下:

python:3.6-alpine 镜像基于 alpine linux 3.4:Dockerfile

所以你首先需要正确移动到边缘高山分支:Edge
这是通过以下方式完成的:

RUN sed -i -e 's/v3\.4/edge/g' /etc/apk/repositories \
    && apk upgrade --update-cache --available \

然后你就可以安装librdkafka 和依赖了。

【讨论】:

  • RUN sed -i -e 's/v3\.4/edge/g' /etc/apk/repositories \ && apk upgrade --update-cache --available \ && apk --no-cache add alpine-sdk gcc librdkafka 我得到fatal error: librdkafka/rdkafka.h: No such file or directory
  • 一般头文件在<libsomething>-dev的包里,所以尝试安装librdkafka-dev
  • 更进一步,丢失的文件,可以用这个找到对应的包:pkgs.alpinelinux.org/contents
  • 这会将稳定的高山图像切换到“当前开发树”(边缘)。这对于某些实验来说很好,但在生产中存在风险,我建议不要这样做。我强烈建议您在此答案中添加免责声明。
【解决方案2】:

我维护了扩展官方python:3.6-alpine 图像的ucalgary/python-librdkafka 图像并从其source releases 安装librdkafka。欢迎您使用该映像,或查看 Dockerfile 以了解其构建方式。

【讨论】:

  • 您在此答案中提到了 Python 3.6,但查看 Dockerfile 我看到引用了 Python 3.7。这是误导
【解决方案3】:

既然你处于优势地位,你也可以从那里使用 python3:

% cat Dockerfile 
FROM alpine:edge
MAINTAINER Gerd Flaig <stackoverflow@gerd.flaig.name>

RUN apk update && apk add --no-cache python3 librdkafka && \
    python3 -m ensurepip && \
    rm -r /usr/lib/python*/ensurepip && \
    pip3 install --upgrade pip setuptools && \
    rm -r /root/.cache
% docker build .
Sending build context to Docker daemon  2.048kB
Step 1/3 : FROM alpine:edge
edge: Pulling from library/alpine
71c5a0cc58e4: Pull complete
Digest: sha256:99588bc8883c955c157d18fc3eaa4a3c1400c223e6c7cabca5f600a3e9f8d5cd
Status: Downloaded newer image for alpine:edge
 ---> 8914de95a28d
Step 2/3 : MAINTAINER Gerd Flaig <stackoverflow@gerd.flaig.name>
 ---> Running in 658b056c4e16
 ---> a3b2485fabb0
Removing intermediate container 658b056c4e16
Step 3/3 : RUN apk update && apk add --no-cache python3 librdkafka &&     python3 -m ensurepip &&     rm -r /usr/lib/python*/ensurepip &&     pip3 install --upgrade pip setuptools &&     rm -r /root/.cache
 ---> Running in d7c3f3b30d89
fetch http://dl-cdn.alpinelinux.org/alpine/edge/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/edge/community/x86_64/APKINDEX.tar.gz
v3.5.0-4449-g8922925e5c [http://dl-cdn.alpinelinux.org/alpine/edge/main]
v3.5.0-4450-gb7b4122d6f [http://dl-cdn.alpinelinux.org/alpine/edge/community]
OK: 8348 distinct packages available
fetch http://dl-cdn.alpinelinux.org/alpine/edge/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/edge/community/x86_64/APKINDEX.tar.gz
(1/16) Installing libressl2.5-libcrypto (2.5.3-r1)
(2/16) Installing libgcc (6.3.0-r3)
(3/16) Installing libressl2.5-libssl (2.5.3-r1)
(4/16) Installing libstdc++ (6.3.0-r3)
(5/16) Installing librdkafka (0.9.4-r1)
(6/16) Installing libbz2 (1.0.6-r5)
(7/16) Installing expat (2.2.0-r0)
(8/16) Installing libffi (3.2.1-r3)
(9/16) Installing gdbm (1.12-r0)
(10/16) Installing xz-libs (5.2.3-r0)
(11/16) Installing ncurses-terminfo-base (6.0-r7)
(12/16) Installing ncurses-terminfo (6.0-r7)
(13/16) Installing ncurses-libs (6.0-r7)
(14/16) Installing readline (6.3.008-r5)
(15/16) Installing sqlite-libs (3.18.0-r0)
(16/16) Installing python3 (3.6.1-r1)
Executing busybox-1.26.2-r0.trigger
OK: 75 MiB in 27 packages
Requirement already satisfied: setuptools in /usr/lib/python3.6/site-packages
Requirement already satisfied: pip in /usr/lib/python3.6/site-packages
Requirement already up-to-date: pip in /usr/lib/python3.6/site-packages
Collecting setuptools
  Downloading setuptools-35.0.2-py2.py3-none-any.whl (390kB)
Collecting packaging>=16.8 (from setuptools)
  Downloading packaging-16.8-py2.py3-none-any.whl
Collecting appdirs>=1.4.0 (from setuptools)
  Downloading appdirs-1.4.3-py2.py3-none-any.whl
Collecting six>=1.6.0 (from setuptools)
  Downloading six-1.10.0-py2.py3-none-any.whl
Collecting pyparsing (from packaging>=16.8->setuptools)
  Downloading pyparsing-2.2.0-py2.py3-none-any.whl (56kB)
Installing collected packages: pyparsing, six, packaging, appdirs, setuptools
  Found existing installation: setuptools 28.8.0
    Uninstalling setuptools-28.8.0:
      Successfully uninstalled setuptools-28.8.0
Successfully installed appdirs-1.4.3 packaging-16.8 pyparsing-2.2.0 setuptools-35.0.2 six-1.10.0
 ---> cfb1033ceec0
Removing intermediate container d7c3f3b30d89
Successfully built cfb1033ceec0

【讨论】:

  • librdkafka 从边缘安装,但 pip 安装 confluent-kafka 失败,并出现上述缺少头文件错误。
猜你喜欢
  • 1970-01-01
  • 2022-01-09
  • 2021-02-14
  • 2021-07-05
  • 2019-12-05
  • 1970-01-01
  • 1970-01-01
  • 2018-12-14
  • 2018-10-11
相关资源
最近更新 更多