使用 livy 向 AWS EMR 提交具有虚拟环境的 pyspark 作业

Question

我按照 AWS 文档创建了一个具有以下配置的 EMR 集群

https://aws.amazon.com/premiumsupport/knowledge-center/emr-pyspark-python-3x/

{
    "Classification": "livy-conf",
    "Properties": {
      "livy.spark.deploy-mode": "cluster",
      "livy.impersonation.enabled": "true",
      "livy.spark.yarn.appMasterEnv.PYSPARK_PYTHON": "/usr/bin/python3"
    }
  },

当我通过以下发布请求使用 livy 提交 pyspark 作业时

```
payload = {
    'file': self.py_file,
    'pyFiles': self.py_files,
    'name': self.job_name,
    'archives': ['s3://test.test.bucket/venv.zip#venv', 's3://test.test.bucket/requirements.pip'],
    'proxyUser': 'hadoop',
    "conf": {
      "PYSPARK_PYTHON": "./venv/bin/python",
      "spark.yarn.appMasterEnv.PYSPARK_PYTHON": "./venv/bin/python",
      "spark.yarn.executorEnv.PYSPARK_PYTHON": "./venv/bin/python",
      "spark.yarn.appMasterEnv.VIRTUAL_ENV": "./venv/bin/python",
      "spark.yarn.executorEnv.VIRTUAL_ENV": "./venv/bin/python",
      "livy.spark.yarn.appMasterEnv.PYSPARK_PYTHON": "./venv/bin/python",
      "livy.spark.yarn.appMasterEnv.PYSPARK_PYTHON": "./venv/bin/python",
      "spark.pyspark.virtualenv.enabled": "true",
      "spark.pyspark.virtualenv.type": "native",
      "spark.pyspark.virtualenv.requirements": "s3://test.test.bucket/requirements.pip",
      "spark.pyspark.virtualenv.path": "./venv/bin/python"
     }
}

```

我收到以下错误消息：

```
Log Type: stdout
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Fatal Python error: Py_Initialize: Unable to get the locale encoding
ImportError: No module named 'encodings'
Current thread 0x00007efc72b57740 (most recent call first)
```

我也试过在虚拟环境下把PYTHONHOMEPYTHONPATH改成python的bin文件的父文件夹，但是没有用。

```
"spark.yarn.appMasterEnv.PYTHONPATH": "./venv/bin/",
"spark.yarn.executorEnv.PYTHONPATH": "./venv/bin/",
"livy.spark.yarn.appMasterEnv.PYTHONPATH": "./venv/bin/",
"livy.spark.yarn.executorEnv.PYTHONPATH": "./venv/bin/",
#
"spark.yarn.appMasterEnv.PYTHONHOME": "./venv/bin/",
"spark.yarn.executorEnv.PYTHONHOME": "./venv/bin/",
"livy.spark.yarn.appMasterEnv.PYTHONHOME": "./venv/bin/",
"livy.spark.yarn.executorEnv.PYTHONHOME": "./venv/bin/",
```

错误：

Fatal Python error: Py_Initialize: Unable to get the locale encoding ImportError: No module named 'encodings' Current thread 0x00007f7351d53740 (most recent call first):

这就是我创建虚拟环境的方式

```
python3 -m venv venv/
source venv/bin/activate
python3 -m pip install -r requirements.pip
deactivate
pushd venv/
zip -rq ../venv.zip *
popd
```

虚拟环境结构：

drwxrwxr-x  2   4096 Oct 15 12:37 bin/
drwxrwxr-x  2   4096 Oct 15 12:37 include/
drwxrwxr-x  3   4096 Oct 15 12:37 lib/
lrwxrwxrwx  1      3 Oct 15 12:37 lib64 -> lib/
-rw-rw-r--  1     59 Oct 15 12:37 pip-selfcheck.json
-rw-rw-r--  1     69 Oct 15 12:37 pyvenv.cfg
drwxrwxr-x  3   4096 Oct 15 12:37 share/

bin 目录：

activate  activate.csh  activate.fish  chardetect  easy_install  easy_install-3.5  pip  pip3  pip3.5  python  python3

库目录：

python3.5/site-packages/

Aws 支持说这是一个持续存在的错误。

https://issues.apache.org/jira/browse/SPARK-13587

https://issues.apache.org/jira/browse/ZEPPELIN-2233

有什么建议吗？

谢谢！

Answer 1

我需要提交一个带有虚拟环境的 PySpark 作业。为了在 5.x 发行版中使用 EMR 中的虚拟虚拟环境，我这样做了：

进入代码文件夹的根目录（例如：/home/hadoop）并运行：

virtualenv -p /usr/bin/python3 <your-venv_name>

source <your-venv_name>/bin/activate

进入 /bin 并运行：

./pip3 freeze --> ensure that is empty

sudo ./pip3 install -r <CODE FOLDER PATH>/requirements.txt

./pip3 freeze --> ensure that is populated

为了提交我的工作，我使用了（带有基本配置）这个命令：

spark-submit --conf spark.pyspark.virtualenv.bin.path=<path-to-your-venv_name>- --conf spark.pyspark.python=<path-to-your-venv_name>/bin/python3 --conf spark.pyspark.driver.python=<path-to-your-venv_name>/bin/python3 <path-to-your-main.py>

在 main.py 代码中，您还必须在环境中指定“PYSPARK PYTHON”：

import os
os.environ["PYSPARK_PYTHON"]="/usr/bin/python3"