【发布时间】:2020-06-26 23:10:05
【问题描述】:
在 Google Colab 中,当前目录为 /myContent,该目录包含以下内容:
setup.py spark-2.4.5-bin-hadoop2.7.tgz trainer/
在 trainer 文件夹中,它有 __init__.py task.py。 task.py 有我的 Python 代码,import pyspark
这是安装 spark 依赖文件的setup.py 中的代码 sn-p:
from setuptools import find_packages
from setuptools import setup
REQUIRED_PACKAGES = ['spark-2.4.5-bin-hadoop2.7.tgz']
setup(
name='trainer',
version='0.1',
install_requires=REQUIRED_PACKAGES,
packages=find_packages(),
include_package_data=True,
description='My training application package.'
)
当我向 Google AI Cloud 提交训练作业时,在 /myContent 目录运行以下代码:
!gcloud ai-platform jobs submit training $JOB_NAME \
--package-path $ACKAGE_PATH \
--module-name $MODULE \
--staging-bucket $STAGING_PATH \
--scale-tier custom \
--master-machine-type complex_model_l_gpu \
--worker-machine-type complex_model_l_gpu \
--worker-count 2 \
--runtime-version 2.1 \
--python-version 3.7 \
--packages spark-2.4.5-bin-hadoop2.7.tgz \
--job-dir $JOB_DIR \
-- \
--param_A=1 \
--param_B=2 \
作业失败,日志中显示错误消息:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/usr/lib/python3.7/tokenize.py", line 447, in open
buffer = _builtin_open(filename, 'rb')
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pip-req-build-b_xhvahl/setup.py'
1)我已经将setup.py提交到谷歌AI平台,为什么找不到那个py文件?
2) 如何在Google AI Platform中预先安装spark依赖文件?在 Google Colab Jupyter Notebook 中,我总是在单元格中运行以下代码:
# install spark
%cd
!apt-get install openjdk-8-jdk-headless -qq
!wget https://archive.apache.org/dist/spark/spark-2.4.5/spark-2.4.5-bin-hadoop2.7.tgz
!tar -xvf spark-2.4.5-bin-hadoop2.7.tgz > /dev/null
!pip install -q findspark
os.environ["JAVA_HOME"] = "/usr/lib/jvm/java-8-openjdk-amd64"
os.environ["SPARK_HOME"] = "/root/spark-2.4.5-bin-hadoop2.7"
谢谢
【问题讨论】:
标签: python apache-spark google-cloud-platform