导入 tf-sentencepiece 时未定义的符号答案

【问题标题】：Undefined symbol when importing tf-sentencepiece导入 tf-sentencepiece 时未定义的符号
【发布时间】：2020-01-06 14:37:31
【问题描述】：

在我的 MacBook（版本 10.14.6）上，我成功运行了一个 Django 应用程序，包括 TensorFlow 和 tf-sentencepiece（特别是使用通用句子编码器模型）。当我执行pipenv lock -r > requirements.txt 时，我得到以下必需的包：

-i https://pypi.org/simple
absl-py==0.8.1
aniso8601==7.0.0
astor==0.8.0
bert-embedding==1.0.1
boto3==1.10.34
boto==2.49.0
botocore==1.13.34
certifi==2019.11.28
chardet==3.0.4
coreapi==2.3.3
coreschema==0.0.4
cycler==0.10.0
defusedxml==0.6.0
diff-match-patch==20181111
django-cors-headers==3.2.0
django-elasticsearch-dsl-drf==0.18
django-elasticsearch-dsl==0.5.1
django-filter==2.2.0
django-import-export==2.0
django-neomodel==0.0.4
django-nine==0.2.2
django-pandas==0.6.1
django-rest-framework==0.1.0
django-rest-swagger==2.2.0
django==2.2.8
djangorestframework==3.10.3
docutils==0.15.2
elasticsearch-dsl==6.1.0
elasticsearch==6.4.0
et-xmlfile==1.0.1
gast==0.3.2
gensim==3.8.1
gluonnlp==0.6.0
graphene-neo4j==2.2.3.9
graphene==2.1.8
graphql-core==2.2.1
graphql-relay==2.0.1
graphviz==0.8.4
grpcio==1.25.0
h5py==2.10.0
idna==2.8
ipaddress==1.0.23
itypes==1.1.0
jdcal==1.4.1
jinja2==2.10.3
jmespath==0.9.4
joblib==0.14.0
keras-applications==1.0.8
keras-preprocessing==1.1.0
kiwisolver==1.1.0
lazy-import==0.2.2
lxml==4.4.2
markdown==3.1.1
markuppy==1.14
markupsafe==1.1.1
matplotlib==3.1.2
mock==3.0.5
mxnet==1.4.0
neo4j-driver==1.6.3
neomodel-next==3.5.2.5
neomodel==3.3.1
neotime==1.7.4
nltk==3.4.5
numpy==1.14.6
odfpy==1.4.0
openapi-codec==1.3.2
openpyxl==3.0.2
pandas==0.25.3
promise==2.2.1
protobuf==3.11.1
pycld2==0.41
pyparsing==2.4.5
python-dateutil==2.8.0
pytorch-pretrained-bert==0.6.2
pytz==2019.3
pyyaml==5.2
regex==2019.11.1
requests==2.22.0
risparser==0.4.3
rx==1.6.1
s3transfer==0.2.1
scikit-learn==0.22
scipy==1.3.3
simplejson==3.17.0
singledispatch==3.4.0.3
six==1.13.0
sklearn==0.0
smart-open==1.9.0
sqlalchemy==1.3.11
sqlparse==0.3.0
tablib==0.14.0
tensorboard==1.13.1
tensorflow-estimator==1.13.0
tensorflow-hub==0.7.0
tensorflow==1.13.1
termcolor==1.1.0
tf-sentencepiece==0.1.82
torch==1.3.1
tqdm==4.40.1
typing==3.6.6
uritemplate==3.0.0
urllib3==1.25.7
werkzeug==0.16.0
wheel==0.33.6 ; python_version >= '3'
xlrd==1.2.0
xlwt==1.3.0

上面的列表也是在我的Dockerfile中生成的：

# The first instruction is what image we want to base our container on
# We Use an official Python runtime as a parent image
FROM python:3.7-stretch

# The enviroment variable ensures that the python output is set straight
# to the terminal with out buffering it first
ENV PYTHONUNBUFFERED 1

# create root directory for our project in the container
RUN mkdir /var/lib/django

# Set the working directory to /var/lib/django
WORKDIR /var/lib/django

# Copy the current directory contents into the container at /var/lib/django
ADD . /var/lib/django/

# Install any needed packages specified in pipfile using piping
RUN pip install --upgrade pip
RUN pip install pipenv

# Below are packages that need to be added to allow C tools
RUN apt update
RUN apt-get -y install libc-dev
RUN apt-get -y install build-essential

# Install requirements
RUN pipenv lock -r > requirements.txt
RUN pip install -r requirements.txt

# Run onDockerStartup file (executes python required functions for startup)
RUN python3 onDockerStartup.py

EXPOSE 8000

CMD ["python3", "manage.py", "runserver", "0.0.0.0:8000", "--noreload"]

但是，当我使用 docker-compose 启动我的应用程序时，我收到以下错误：

django_1  | Traceback (most recent call last):
django_1  |   File "manage.py", line 21, in <module>
django_1  |     main()
django_1  |   File "manage.py", line 17, in main
django_1  |     execute_from_command_line(sys.argv)
django_1  |   File "/usr/local/lib/python3.7/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
django_1  |     utility.execute()
django_1  |   File "/usr/local/lib/python3.7/site-packages/django/core/management/__init__.py", line 375, in execute
django_1  |     self.fetch_command(subcommand).run_from_argv(self.argv)
django_1  |   File "/usr/local/lib/python3.7/site-packages/django/core/management/base.py", line 323, in run_from_argv
django_1  |     self.execute(*args, **cmd_options)
django_1  |   File "/usr/local/lib/python3.7/site-packages/django/core/management/commands/runserver.py", line 60, in execute
django_1  |     super().execute(*args, **options)
django_1  |   File "/usr/local/lib/python3.7/site-packages/django/core/management/base.py", line 364, in execute
django_1  |     output = self.handle(*args, **options)
django_1  |   File "/usr/local/lib/python3.7/site-packages/django/core/management/commands/runserver.py", line 95, in handle
django_1  |     self.run(**options)
django_1  |   File "/usr/local/lib/python3.7/site-packages/django/core/management/commands/runserver.py", line 104, in run
django_1  |     self.inner_run(None, **options)
django_1  |   File "/usr/local/lib/python3.7/site-packages/django/core/management/commands/runserver.py", line 117, in inner_run
django_1  |     self.check(display_num_errors=True)
django_1  |   File "/usr/local/lib/python3.7/site-packages/django/core/management/base.py", line 390, in check
django_1  |     include_deployment_checks=include_deployment_checks,
django_1  |   File "/usr/local/lib/python3.7/site-packages/django/core/management/base.py", line 377, in _run_checks
django_1  |     return checks.run_checks(**kwargs)
django_1  |   File "/usr/local/lib/python3.7/site-packages/django/core/checks/registry.py", line 72, in run_checks
django_1  |     new_errors = check(app_configs=app_configs)
django_1  |   File "/usr/local/lib/python3.7/site-packages/django/core/checks/urls.py", line 40, in check_url_namespaces_unique
django_1  |     all_namespaces = _load_all_namespaces(resolver)
django_1  |   File "/usr/local/lib/python3.7/site-packages/django/core/checks/urls.py", line 57, in _load_all_namespaces
django_1  |     url_patterns = getattr(resolver, 'url_patterns', [])
django_1  |   File "/usr/local/lib/python3.7/site-packages/django/utils/functional.py", line 80, in __get__
django_1  |     res = instance.__dict__[self.name] = self.func(instance)
django_1  |   File "/usr/local/lib/python3.7/site-packages/django/urls/resolvers.py", line 584, in url_patterns
django_1  |     patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
django_1  |   File "/usr/local/lib/python3.7/site-packages/django/utils/functional.py", line 80, in __get__
django_1  |     res = instance.__dict__[self.name] = self.func(instance)
django_1  |   File "/usr/local/lib/python3.7/site-packages/django/urls/resolvers.py", line 577, in urlconf_module
django_1  |     return import_module(self.urlconf_name)
django_1  |   File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module
django_1  |     return _bootstrap._gcd_import(name[level:], package, level)
django_1  |   File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
django_1  |   File "<frozen importlib._bootstrap>", line 983, in _find_and_load
django_1  |   File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
django_1  |   File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
django_1  |   File "<frozen importlib._bootstrap_external>", line 728, in exec_module
django_1  |   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
django_1  |   File "/var/lib/django/backend/urls.py", line 29, in <module>
django_1  |     path(r'search/', include('apps.search.urls')),
django_1  |   File "/usr/local/lib/python3.7/site-packages/django/urls/conf.py", line 34, in include
django_1  |     urlconf_module = import_module(urlconf_module)
django_1  |   File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module
django_1  |     return _bootstrap._gcd_import(name[level:], package, level)
django_1  |   File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
django_1  |   File "<frozen importlib._bootstrap>", line 983, in _find_and_load
django_1  |   File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
django_1  |   File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
django_1  |   File "<frozen importlib._bootstrap_external>", line 728, in exec_module
django_1  |   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
django_1  |   File "/var/lib/django/apps/search/urls.py", line 4, in <module>
django_1  |     from . import views
django_1  |   File "/var/lib/django/apps/search/views.py", line 24, in <module>
django_1  |     from .servicelayer.question_answer import find_answer_to_question
django_1  |   File "/var/lib/django/apps/search/servicelayer/question_answer.py", line 1, in <module>
django_1  |     from ml.use import embed_use
django_1  |   File "/var/lib/django/ml/use.py", line 9, in <module>
django_1  |     import tf_sentencepiece
django_1  |   File "/usr/local/lib/python3.7/site-packages/tf_sentencepiece/__init__.py", line 5, in <module>
django_1  |     from tf_sentencepiece.sentencepiece_processor_ops import *
django_1  |   File "/usr/local/lib/python3.7/site-packages/tf_sentencepiece/sentencepiece_processor_ops.py", line 43, in <module>
django_1  |     _gen_sentencepiece_processor_op = tf.load_op_library(so_file)
django_1  |   File "/usr/local/lib/python3.7/site-packages/tensorflow/python/framework/load_library.py", line 61, in load_op_library
django_1  |     lib_handle = py_tf.TF_LoadLibrary(library_filename)
django_1  | tensorflow.python.framework.errors_impl.NotFoundError: /usr/local/lib/python3.7/site-packages/tf_sentencepiece/_sentencepiece_processor_ops.so.1.13.1: undefined symbol: _ZN10tensorflow12OpDefBuilder4AttrESs

我应该如何解决这个错误才能让sentencepiece成功启动？

【问题讨论】：

请注意，在您的 Dockerfile 中，pipenv lock 命令在容器内运行。因此，您不会在 docker 中复制外部环境。您应该事先创建 requirements.txt，然后构建您的 docker 映像。
@GrégoireRoussel 好建议，我删除了在 de Dockerfile 中创建 requirements.txt 文件的行。然后在本地我1）创建了一个requirements.txt，2）删除了虚拟环境，3）执行pipenv install -r requirements.txt，以确认应用程序仍然可以在本地工作，它做到了！然后我重新创建了 Docker 镜像并运行了docker-compose，但不幸的是容器中出现了同样的错误。

标签： python docker tensorflow pip

【解决方案1】：

问题不在于 Dockerfile 的设置，也不在于 requirements.txt 中生成的包。这与 Tensorflow 与 Python 3.7 不兼容有关（尽管在本地一切正常，但仍然很奇怪）。我在本地机器上迁移到 Python 3.6.5 以测试一切是否正常。然后我将我的 Dockerfile 更改为以

开头

FROM python:3.6-stretch

我的应用程序现在可以成功运行了。

【讨论】：

【解决方案2】：

我没有 Django 的技能，但似乎 tensorflow 试图找到一个包（名字很奇怪）并且失败了。

我首先建议尝试修复您的 docker 容器设置，并检查 pipenv lock -r 在容器内外产生相同的结果。

1）正如您在评论中所说，在主机上

# Install requirements
RUN pipenv lock -r > requirements.txt

2) 在 Docker 容器中复制此环境

# Dockerfile
#...

# Install requirements
ADD requirements.txt .
RUN pip install -r requirements.txt

3) 检查：

docker container run -it --rm <your image> pipenv lock -r

我能带你就这么多，希望够了

【讨论】：

【解决方案3】：

Dockerfile

如 cmets 中所述，首先应该在主机上生成 requirements.txt 文件，而不是在 Dockerfile 中。

# Install any needed packages specified in pipfile using piping
RUN pip install --upgrade pip
RUN pip install pipenv

...

# Install requirements
RUN pipenv lock -r > requirements.txt

将大致生成一个requirements.txt，其中只有pip 和pipenv。

所以，在您的主机上生成您的 requirements.txt 并将其添加到您的容器中：

# Install requirements
ADD requirements.txt .
RUN pip install -r requirements.txt

句子片段

然后，您的错误似乎与已编译的库有关。 Python 正在_sentencepiece_processor_ops.so.1.13.1 库中查找名为_ZN10tensorflow12OpDefBuilder4AttrESs 的符号，但没有找到。这通常发生在版本不匹配时——您正在旧版本的库中寻找新符号。

我不知道sentencepiece，但我看到它存在两个python 包：sentencepiece 和tf_sentencepiece。但是，我看到sentencepiece 没有在您的requirements.txt 中列出（可能是您手动安装的），我觉得tf_sentencepiece 需要它才能工作。

此外，我觉得很奇怪你有_sentencepiece_processor_ops.so.1.13.1，而sentencepiece的最后一个版本是0.1.85。

【讨论】：