如何在 Amazon EMR 上安装多个版本的 numpy 以及如何删除早期版本？答案

【问题标题】：How do I have multiple versions of numpy installed on Amazon EMR and how to I delete the early versions?如何在 Amazon EMR 上安装多个版本的 numpy 以及如何删除早期版本？
【发布时间】：2020-12-06 22:30:48
【问题描述】：

我不明白 Python 如何可以安装单个包的多个版本，或者为什么当我安装了多个版本时，import package 没有给我最新的版本。

我在 AWS EMR 中使用 AWS linux 和 AWS 存储库。当我安装 Python 3.6 时，它默认为我提供 Numpy 版本 1.14.5。我想使用 1.19.1 版本并已将其放入目录中。

$ sudo pip-3.6 install --upgrade --no-index --find-links /mnt/packages/ numpy==1.19.1
Requirement already up-to-date: numpy==1.19.1 in /usr/local/lib64/python3.6/site-packages
$

但请注意：

 $ python
Python 3.6.11 (default, Jul 20 2020, 22:15:17)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> numpy.version.version
'1.14.5'
>>>

因为我同时安装了 1.14.5 版和 1.19.1 版。

我可以卸载 1.19.1 版，然后我可以卸载 1.14.5 版，虽然我似乎必须卸载它两次（请参阅下面的扩展混音）。

那么我该如何让import numpy 获得版本1.19.1 而不是旧版本？

$ sudo pip-3.6 uninstall numpy
Uninstalling numpy-1.14.5:
  /usr/local/lib64/python3.6/site-packages/numpy-1.14.5.data/scripts/f2py
  /usr/local/lib64/python3.6/site-packages/numpy-1.14.5.dist-info/METADATA
  /usr/local/lib64/python3.6/site-packages/numpy-1.14.5.dist-info/RECORD
  /usr/local/lib64/python3.6/site-packages/numpy-1.14.5.dist-info/WHEEL
  /usr/local/lib64/python3.6/site-packages/numpy-1.14.5.dist-info/top_level.txt
  /usr/local/lib64/python3.6/site-packages/numpy/.libs/libgfortran-ed201abd.so.3.0.0
  /usr/local/lib64/python3.6/site-packages/numpy/.libs/libopenblasp-r0-39a31c03.2.18.so
  /usr/local/lib64/python3.6/site-packages/numpy/_import_tools.py
  /usr/local/lib64/python3.6/site-packages/numpy/add_newdocs.py
  /usr/local/lib64/python3.6/site-packages/numpy/core/_dummy.cpython-36m-x86_64-linux-gnu.so
  /usr/local/lib64/python3.6/site-packages/numpy/core/info.py
  /usr/local/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so
  /usr/local/lib64/python3.6/site-packages/numpy/core/multiarray_tests.cpython-36m-x86_64-linux-gnu.so
  /usr/local/lib64/python3.6/site-packages/numpy/core/operand_flag_tests.cpython-36m-x86_64-linux-gnu.so
  /usr/local/lib64/python3.6/site-packages/numpy/core/struct_ufunc_test.cpython-36m-x86_64-linux-gnu.so
  /usr/local/lib64/python3.6/site-packages/numpy/core/test_rational.cpython-36m-x86_64-linux-gnu.so
  /usr/local/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so
  /usr/local/lib64/python3.6/site-packages/numpy/core/umath_tests.cpython-36m-x86_64-linux-gnu.so
  /usr/local/lib64/python3.6/site-packages/numpy/distutils/__version__.py
  /usr/local/lib64/python3.6/site-packages/numpy/distutils/compat.py
  /usr/local/lib64/python3.6/site-packages/numpy/distutils/environment.py
  /usr/local/lib64/python3.6/site-packages/numpy/distutils/info.py
  /usr/local/lib64/python3.6/site-packages/numpy/f2py/info.py
  /usr/local/lib64/python3.6/site-packages/numpy/fft/fftpack.py
  /usr/local/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so
  /usr/local/lib64/python3.6/site-packages/numpy/fft/info.py
  /usr/local/lib64/python3.6/site-packages/numpy/fft/tests/test_fftpack.py
  /usr/local/lib64/python3.6/site-packages/numpy/lib/info.py
  /usr/local/lib64/python3.6/site-packages/numpy/linalg/info.py
  /usr/local/lib64/python3.6/site-packages/numpy/ma/version.py
  /usr/local/lib64/python3.6/site-packages/numpy/random/info.py
  /usr/local/lib64/python3.6/site-packages/numpy/random/randomkit.h
  /usr/local/lib64/python3.6/site-packages/numpy/testing/decorators.py
  /usr/local/lib64/python3.6/site-packages/numpy/testing/nose_tools/__init__.py
  /usr/local/lib64/python3.6/site-packages/numpy/testing/nose_tools/decorators.py
  /usr/local/lib64/python3.6/site-packages/numpy/testing/nose_tools/noseclasses.py
  /usr/local/lib64/python3.6/site-packages/numpy/testing/nose_tools/nosetester.py
  /usr/local/lib64/python3.6/site-packages/numpy/testing/nose_tools/parameterized.py
  /usr/local/lib64/python3.6/site-packages/numpy/testing/nose_tools/utils.py
  /usr/local/lib64/python3.6/site-packages/numpy/testing/noseclasses.py
  /usr/local/lib64/python3.6/site-packages/numpy/testing/nosetester.py
Proceed (y/n)? y
  Successfully uninstalled numpy-1.14.5
You are using pip version 9.0.3, however version 20.2.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
$ python
Python 3.6.11 (default, Jul 20 2020, 22:15:17)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>>
$ sudo pip-3.6 uninstall numpy
Uninstalling numpy-1.14.5:
  /usr/local/lib/python3.6/site-packages/numpy-1.14.5.data/scripts/f2py
  /usr/local/lib/python3.6/site-packages/numpy-1.14.5.dist-info/METADATA
  /usr/local/lib/python3.6/site-packages/numpy-1.14.5.dist-info/RECORD
  /usr/local/lib/python3.6/site-packages/numpy-1.14.5.dist-info/WHEEL
  /usr/local/lib/python3.6/site-packages/numpy-1.14.5.dist-info/top_level.txt
  /usr/local/lib/python3.6/site-packages/numpy/.libs/libgfortran-ed201abd.so.3.0.0
  /usr/local/lib/python3.6/site-packages/numpy/.libs/libopenblasp-r0-39a31c03.2.18.so
  /usr/local/lib/python3.6/site-packages/numpy/LICENSE.txt
  /usr/local/lib/python3.6/site-packages/numpy/__config__.py
  /usr/local/lib/python3.6/site-packages/numpy/__init__.py
  /usr/local/lib/python3.6/site-packages/numpy/_distributor_init.py
  /usr/local/lib/python3.6/site-packages/numpy/_globals.py
  /usr/local/lib/python3.6/site-packages/numpy/_import_tools.py
  /usr/local/lib/python3.6/site-packages/numpy/add_newdocs.py
  /usr/local/lib/python3.6/site-packages/numpy/compat/__init__.py
  /usr/local/lib/python3.6/site-packages/numpy/compat/_inspect.py
  /usr/local/lib/python3.6/site-packages/numpy/compat/py3k.py
  /usr/local/lib/python3.6/site-packages/numpy/compat/setup.py
  /usr/local/lib/python3.6/site-packages/numpy/conftest.py
  /usr/local/lib/python3.6/site-packages/numpy/core/__init__.py
  /usr/local/lib/python3.6/site-packages/numpy/core/_dummy.cpython-36m-x86_64-linux-gnu.so
  /usr/local/lib/python3.6/site-packages/numpy/core/_internal.py
  /usr/local/lib/python3.6/site-packages/numpy/core/_methods.py
  /usr/local/lib/python3.6/site-packages/numpy/core/arrayprint.py
  /usr/local/lib/python3.6/site-packages/numpy/core/cversions.py
  /usr/local/lib/python3.6/site-packages/numpy/core/defchararray.py
  /usr/local/lib/python3.6/site-packages/numpy/core/einsumfunc.py
  /usr/local/lib/python3.6/site-packages/numpy/core/fromnumeric.py
  /usr/local/lib/python3.6/site-packages/numpy/core/function_base.py
  /usr/local/lib/python3.6/site-packages/numpy/core/generate_numpy_api.py
  /usr/local/lib/python3.6/site-packages/numpy/core/getlimits.py
  /usr/local/lib/python3.6/site-packages/numpy/core/include/numpy/__multiarray_api.h
  /usr/local/lib/python3.6/site-packages/numpy/core/include/numpy/__ufunc_api.h
...
  /usr/local/lib/python3.6/site-packages/numpy/distutils/fcompiler/intel.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/fcompiler/lahey.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/fcompiler/mips.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/fcompiler/nag.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/fcompiler/none.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/fcompiler/pathf95.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/fcompiler/pg.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/fcompiler/sun.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/fcompiler/vast.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/from_template.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/info.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/intelccompiler.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/lib2def.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/line_endings.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/log.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/mingw/gfortran_vs2003_hack.c
  /usr/local/lib/python3.6/site-packages/numpy/distutils/mingw32ccompiler.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/misc_util.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/msvc9compiler.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/msvccompiler.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/npy_pkg_config.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/numpy_distribution.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/pathccompiler.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/setup.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/system_info.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/tests/__init__.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/tests/test_exec_command.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/tests/test_fcompiler_gnu.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/tests/test_fcompiler_intel.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/tests/test_fcompiler_nagfor.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/tests/test_misc_util.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/tests/test_npy_pkg_config.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/tests/test_system_info.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/unixccompiler.py
  /usr/local/lib/python3.6/site-packages/numpy/doc/__init__.py
  /usr/local/lib/python3.6/site-packages/numpy/doc/basics.py
  /usr/local/lib/python3.6/site-packages/numpy/doc/broadcasting.py
  /usr/local/lib/python3.6/site-packages/numpy/doc/byteswapping.py
  /usr/local/lib/python3.6/site-packages/numpy/doc/constants.py
...
  /usr/local/lib/python3.6/site-packages/numpy/testing/print_coercion_tables.py
  /usr/local/lib/python3.6/site-packages/numpy/testing/setup.py
  /usr/local/lib/python3.6/site-packages/numpy/testing/tests/__init__.py
  /usr/local/lib/python3.6/site-packages/numpy/testing/tests/test_decorators.py
  /usr/local/lib/python3.6/site-packages/numpy/testing/tests/test_doctesting.py
  /usr/local/lib/python3.6/site-packages/numpy/testing/tests/test_utils.py
  /usr/local/lib/python3.6/site-packages/numpy/testing/utils.py
  /usr/local/lib/python3.6/site-packages/numpy/tests/__init__.py
  /usr/local/lib/python3.6/site-packages/numpy/tests/test_ctypeslib.py
  /usr/local/lib/python3.6/site-packages/numpy/tests/test_matlib.py
  /usr/local/lib/python3.6/site-packages/numpy/tests/test_numpy_version.py
  /usr/local/lib/python3.6/site-packages/numpy/tests/test_reloading.py
  /usr/local/lib/python3.6/site-packages/numpy/tests/test_scripts.py
  /usr/local/lib/python3.6/site-packages/numpy/tests/test_warnings.py
  /usr/local/lib/python3.6/site-packages/numpy/version.py
Proceed (y/n)? Y
  Successfully uninstalled numpy-1.14.5
You are using pip version 9.0.3, however version 20.2.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
$ sudo pip-3.6 uninstall numpy
Cannot uninstall requirement numpy, not installed
You are using pip version 9.0.3, however version 20.2.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
$ sudo pip-3.6 uninstall numpy
Cannot uninstall requirement numpy, not installed
You are using pip version 9.0.3, however version 20.2.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
$ python
Python 3.6.11 (default, Jul 20 2020, 22:15:17)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'numpy'
>>>
$ sudo pip-3.6 install --upgrade --no-index --find-links /mnt/gits/das-vm-config/dist/python numpy==1.19.1
Collecting numpy==1.19.1
Installing collected packages: numpy
Successfully installed numpy-1.19.1
$ python
Python 3.6.11 (default, Jul 20 2020, 22:15:17)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> numpy.version.version
'1.19.1'
>>>

【问题讨论】：

标签： python python-3.x numpy pip

【解决方案1】：

我刚刚遇到了同样的问题。我最终运行了以下 3 次来卸载所有 numpy 版本：

python3 -m pip uninstall numpy

一旦它告诉我没有安装 numpy，我就运行安装，它能够为我提供正确的版本。

很烦人，以前不是问题。

【讨论】：

在 emr5.27 上需要射出 3 次“ sudo python3 -m pip uninstall numpy -y”

【解决方案2】：

我们的解决方案是在 EMR 引导脚本（在每个节点上运行）中执行以下操作：

使用yum安装python36和python36-pip
使用带有--target 选项的pip-3.6 在/mnt/python_additions 中安装numpy-1.19.1。我们在 Bootstrap 操作中执行此操作，以便它发生在每个节点上。
让亚马逊的脚本安装python36-numpy，它会在站点库位置安装numpy-1.14.4。
在从spark-submit 调用的程序中，将/mnt/python_additions 添加到sys.path。

这是一个 hack，但它有效。

【讨论】：

您好，谢谢您的回答。请您为此提供一个示例命令吗？努力弄清楚什么去哪里。
它进入您的集群引导脚本。
可以分享你的 .sh 吗？
不是真的，很遗憾。
在第 3 点需要安装一个 with : sudo yum install python36-numpy so ?

【解决方案3】：

虽然您可以安装一个包的多个版本，但在 Python 环境中只能导入一个版本。如果你想明确你想要使用的包的版本，那么看看 Python 从哪里导入包，你可以在交互式会话中运行以下命令：

>>> import sys
>>> print(sys.path)

这将返回一个导入位置列表。列表中的最后一项显示 Python 将查找的最后一个位置。列表中后面的路径会取代前面的路径。如果您有以下情况：

["site_packages_path_with_numpy_1.19.1", "site_packages_with_numpy_1.14.5"]

Python 将导入 numpy 1.14.5。

如果您没有遇到过，我建议您查看 virtualenv 和 requirements.txt。

Virtualenv 允许您使用所需的任何包创建本地版本的 Python（与系统版本分开）。例如。

python -m venv .venv
source .venv/bin/activate

这将创建一个新的 Python 副本并激活它。对 Python 或 PIP 的调用现在将使用虚拟环境。

您可以像这样将软件包安装到虚拟环境中。

pip install numpy==1.19.1

Requirements 允许您保留要安装的软件包的列表。文件是这样的

# requirements.txt

numpy==1.19.1

您可以像这样将需求的内容安装到虚拟环境中（假设您已经运行了活动命令）：

pip install -r requirements.txt

最后，一个更现代且可以说更好的解决方案是https://python-poetry.org/。

【讨论】：

谢谢。我们在 Amazon EMR 中使用 Spark，这又增加了一层复杂性。

【解决方案4】：

从另一个 SO 问题交叉发布对我有用的答案：https://stackoverflow.com/a/69367430/2643556

【讨论】：