AWS EMR Spark“没有名为 pyspark 的模块”答案

【问题标题】：AWS EMR Spark "No Module named pyspark"AWS EMR Spark“没有名为 pyspark 的模块”
【发布时间】：2015-01-11 05:35:31
【问题描述】：

我创建了一个 spark 集群，通过 ssh 连接到 master，然后启动 shell：

MASTER=yarn-client ./spark/bin/pyspark

当我执行以下操作时：

x = sc.textFile("s3://location/files.*")
xt = x.map(lambda x: handlejson(x))
table= sqlctx.inferSchema(xt)

我收到以下错误：

Error from python worker:
  /usr/bin/python: No module named pyspark
PYTHONPATH was:
  /mnt1/var/lib/hadoop/tmp/nm-local-dir/usercache/hadoop/filecache/11/spark-assembly-1.1.0-hadoop2.4.0.jar
java.io.EOFException
        java.io.DataInputStream.readInt(DataInputStream.java:392)
        org.apache.spark.api.python.PythonWorkerFactory.startDaemon(PythonWorkerFactory.scala:151)
        org.apache.spark.api.python.PythonWorkerFactory.createThroughDaemon(PythonWorkerFactory.scala:78)
        org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:54)
        org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:97)
        org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:66)
        org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
        org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
        org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62)
        org.apache.spark.scheduler.Task.run(Task.scala:54)
        org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:177)
        java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        java.lang.Thread.run(Thread.java:745)

我还检查了 PYTHONPATH

 >>> os.environ['PYTHONPATH'] '/home/hadoop/spark/python/lib/py4j-0.8.2.1-src.zip:/home/hadoop/spark/python/:/‌home/hadoop/spark/lib/spark-assembly-1.1.0-hadoop2.4.0.jar'

然后在罐子里寻找 pyspark，它就在那里：

jar -tf /home/hadoop/spark/lib/spark-assembly-1.1.0-hadoop2.4.0.jar | grep pyspark
pyspark/
pyspark/shuffle.py
pyspark/resultiterable.py
pyspark/files.py
pyspark/accumulators.py
pyspark/sql.py
pyspark/java_gateway.py
pyspark/join.py
pyspark/serializers.py
pyspark/shell.py
pyspark/rddsampler.py
pyspark/rdd.py
....

以前有人遇到过这种情况吗？谢谢！

【问题讨论】：

我还检查了 PYTHONGPATH 环境：>>> os.environ['PYTHONPATH'] '/home/hadoop/spark/python/lib/py4j-0.8.2.1-src.zip:/home/hadoop/spark/python/:/home/hadoop/spark/lib/spark-assembly-1.1.0-hadoop2.4.0.jar'，看起来没问题。
另外，jar -tf /home/hadoop/spark/lib/spark-assembly-1.1.0-hadoop2.4.0.jar | grep pyspark 表明 pyspark 是 jar 的一部分。

标签： python amazon-web-services apache-spark amazon-emr

【解决方案1】：

您需要参考这些 Spark 问题：

解决方案（假设您不想重建 jar）：

unzip -d foo spark/lib/spark-assembly-1.1.0-hadoop2.4.0.jar
cd foo
# if you don't have openjdk 1.6:
# yum install -y java-1.6.0-openjdk-devel.x86_64
/usr/lib/jvm/openjdk-1.6.0/bin/jar cvmf META-INF/MANIFEST ../spark/lib/spark-assembly-1.1.0-hadoop2.4.0.jar .
# don't neglect the dot at the end of that command

【讨论】：

【解决方案2】：

这在 EMR 的后续版本中已修复。有关发行说明和说明，请参阅 https://github.com/awslabs/emr-bootstrap-actions/tree/master/spark。

【讨论】：