无法在 AWS Glue python shell 上安装 pyarrow

Question

我想在 Python shell Glue 脚本中导入 pyarrow，因为我需要将数据框导出为镶木地板（即使用 DataFrame.to_parquet()）。

AWS 文档中建议的添加自定义依赖项的方法是使用 .egg 或 .whl 文件 (https://docs.aws.amazon.com/glue/latest/dg/add-job-python.html#create-python-extra-library)。

库pyarrow 具有numpy 和six 作为依赖项：

• numpy 已经预安装在 Glue 上，版本为 1.16.2，我使用简单的 print(numpy.version.version) 进行了检查

• six 没有预装，所以我从 Pypi 下载了six-1.14.0-py2.py3-none-any.whl 并将其上传到 S3。

• pyarrow 没有预装，所以我从 Pypi 下载了 wheel 文件 pyarrow-0.16.0-cp36-cp36m-manylinux2014_x86_64.whl 并上传到 S3。

脚本本身是这样的：

import pandas as pd
import six
import numpy
from pyarrow import *

data = [['Alex',10],['Bob',12],['Clarke',13]]
df = pd.DataFrame(data,columns=['Name','Age'],dtype=float)
df.to_parquet('test.parquet')

当我运行将six 和pyarrow 的轮文件添加为库的脚本时，我收到以下消息：

Processing ./glue-python-libs-f8nyy9el/six-1.14.0-py2.py3-none-any.whl
Installing collected packages: six
Successfully installed six-1.14.0
Processing ./glue-python-libs-f8nyy9el/pyarrow-0.16.0-cp36-cp36m-manylinux2014_x86_64.whl

还有以下错误：

WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f7d78c96e10>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/numpy/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f7d78c96c88>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/numpy/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f7d78c96dd8>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/numpy/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f7d78c969b0>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/numpy/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f7d78c96898>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/numpy/
ERROR: Could not find a version that satisfies the requirement numpy>=1.14 (from pyarrow==0.16.0) (from versions: none)
ERROR: No matching distribution found for numpy>=1.14 (from pyarrow==0.16.0)
Traceback (most recent call last):
  File "/tmp/runscript.py", line 112, in <module>
    download_and_install(args.extra_py_files)
  File "/tmp/runscript.py", line 62, in download_and_install
    subprocess.check_call([sys.executable, "-m", "pip", "install", "--target=
{}
".format(install_path), local_file_path])
  File "/usr/local/lib/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/usr/local/bin/python', '-m', 'pip', 'install', '--target=/glue/lib/installation', '/tmp/glue-python-libs-f8nyy9el/pyarrow-0.16.0-cp36-cp36m-manylinux2014_x86_64.whl']' returned non-zero exit status 1.

所以起初，似乎 6 已正确安装，但随后看起来该作业没有意识到 numpy 已经存在兼容版本。

然后我尝试将我从 Pypi 下载的轮子文件 s3://risultati-navigazione-wt-ga/libs/numpy-1.18.2-cp36-cp36m-manylinux1_x86_64.whl 也上传到 S3。在这种情况下，我收到消息：

Processing ./glue-python-libs-xzfdvgzd/numpy-1.18.2-cp36-cp36m-manylinux1_x86_64.whl
Installing collected packages: numpy
Successfully installed numpy-1.18.2
Processing ./glue-python-libs-xzfdvgzd/six-1.14.0-py2.py3-none-any.whl
Installing collected packages: six
Successfully installed six-1.14.0
Processing ./glue-python-libs-xzfdvgzd/pyarrow-0.16.0-cp36-cp36m-manylinux2014_x86_64.whl

和错误：

WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fca02861cc0>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/six/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fca02861cc0>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/six/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fca02861cc0>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/six/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fca02861cc0>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/six/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fca02861cc0>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/six/
ERROR: Could not find a version that satisfies the requirement six>=1.0.0 (from pyarrow==0.16.0) (from versions: none)
ERROR: No matching distribution found for six>=1.0.0 (from pyarrow==0.16.0)
Traceback (most recent call last):
  File "/tmp/runscript.py", line 112, in <module>
    download_and_install(args.extra_py_files)
  File "/tmp/runscript.py", line 62, in download_and_install
    subprocess.check_call([sys.executable, "-m", "pip", "install", "--target=
{}
".format(install_path), local_file_path])
  File "/usr/local/lib/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/usr/local/bin/python', '-m', 'pip', 'install', '--target=/glue/lib/installation', '/tmp/glue-python-libs-xzfdvgzd/pyarrow-0.16.0-cp36-cp36m-manylinux2014_x86_64.whl']' returned non-zero exit status 1.

所以，这一次，numpy 在安装 pyarrow 期间被识别，但据我所知，虽然six 安装正确，但由于某种原因，pyarrow 在安装过程中找不到它，实际上它试图从 Internet 下载（在该操作过程中会卡住几分钟）。

有人可以帮我吗？谢谢！

Answer 1

要使用pyarrow，需要构建自己的包。在setup.py 中，您必须将pyarrow 指定为install_requires 之一。

setup(
    name="PACKAGE_NAME",
    version="0.1",
    packages=find_packages(),
    install_requires=[
        'pyarrow'
    ],
)

Here您可以找到有关为 AWS 胶水构建包的更多信息。

根据documentation，pandas也应该以这种方式添加。

pandas（需要通过python setuptools配置，setup.py安装）

Answer 2

更新： 这不再有效，必须使用 WHL 添加库。

---

以前的答案： 尝试添加：

import os
import site
import importlib
from setuptools.command import easy_install
install_path = os.environ['GLUE_INSTALLATION']

libraries = ["pyarrow"]
for lib in libraries:
    easy_install.main( ["--install-dir", install_path, lib] )

importlib.reload(site)

在 Glue Python Shell 代码的开头。这似乎比在 Python 库路径中添加 .whl 库更好。

Answer 3

当我将这个 Linux 版本的 pyarrow 上传到 S3 时为我工作，它是编辑作业模式中 Python 库路径框的路径： https://files.pythonhosted.org/packages/04/57/f9a96302f27f0008f5afcd4232d4df66a6af5e568445128cac52f64ee4fd/pyarrow-3.0.0-cp36-cp36m-manylinux2010_x86_64.whl

您可以像上面 Mauro 的回答那样使用易于安装^，但我相信很快就会弃用。