带有 spark 的 ipython 笔记本在 sparkcontext 中出现错误答案

【问题标题】：ipython notebook with spark gets error with sparkcontext带有 spark 的 ipython 笔记本在 sparkcontext 中出现错误
【发布时间】：2016-07-13 14:31:19
【问题描述】：

我正在我的 macbook osx 10.10.5 上用这个例子测试 turi https://turi.com/learn/gallery/notebooks/spark_and_graphlab_create.html

当到达这一步时

# Set up the SparkContext object
# this can be 'local' or 'yarn-client' in PySpark
# Remember if using yarn-client then all the paths should be accessible
# by all nodes in the cluster.
sc = SparkContext('local')

出现以下错误

---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
<ipython-input-12-dc1befb4186c> in <module>()
      3 # Remember if using yarn-client then all the paths should be accessible
      4 # by all nodes in the cluster.
----> 5 sc = SparkContext()

/usr/local/Cellar/apache-spark/1.6.2/libexec/python/pyspark/context.pyc in __init__(self, master, appName, sparkHome, pyFiles, environment, batchSize, serializer, conf, gateway, jsc, profiler_cls)
    110         """
    111         self._callsite = first_spark_call() or CallSite(None, None, None)
--> 112         SparkContext._ensure_initialized(self, gateway=gateway)
    113         try:
    114             self._do_init(master, appName, sparkHome, pyFiles, environment, batchSize, serializer,

/usr/local/Cellar/apache-spark/1.6.2/libexec/python/pyspark/context.pyc in _ensure_initialized(cls, instance, gateway)
    243         with SparkContext._lock:
    244             if not SparkContext._gateway:
--> 245                 SparkContext._gateway = gateway or launch_gateway()
    246                 SparkContext._jvm = SparkContext._gateway.jvm
    247 

/usr/local/Cellar/apache-spark/1.6.2/libexec/python/pyspark/java_gateway.pyc in launch_gateway()
     92                 callback_socket.close()
     93         if gateway_port is None:
---> 94             raise Exception("Java gateway process exited before sending the driver its port number")
     95 
     96         # In Windows, ensure the Java child processes do not linger after Python has exited.

Exception: Java gateway process exited before sending the driver its port number

快速谷歌搜索没有帮助。

这是我的 .bash_profile

# added by Anaconda2 4.1.1 installer
export PATH="/Users/me/anaconda/bin:$PATH"

export SCALA_HOME=/usr/local/Cellar/scala/2.11.8/libexec
export SPARK_HOME=/usr/local/Cellar/apache-spark/1.6.2/libexec
export PYTHONPATH=$SPARK_HOME/python/pyspark:$PYTHONPATH
export PYTHONPATH=$SPARK_HOME/python/lib/py4j-0.9-src.zip:$PYTHONPATH 
export PYTHONPATH=$SPARK_HOME/python/:$PYTHONPATH

有人知道如何解决这个错误吗？

谢谢

【问题讨论】：

SPARK_HOME 路径是否正确？您是否在环境变量中设置了PYSPARK_SUBMIT_ARGS="--master spark://<host>:<port>"？这可能是您缺少的 port 号码
spark_home 是正确的，我没有配置 PYSPARK_SUBMIT_ARGS，这种情况我应该指定什么？？
试试这个export PYSPARK_SUBMIT_ARGS="--master local[2]"
@KartikKannapur 我认为这确实有效，您可以编辑您的答案以便我接受吗？非常感谢
很高兴为您提供帮助。将添加答案。

标签： python apache-spark ipython pyspark jupyter-notebook

【解决方案1】：

这可能由于两个原因而发生：

环境变量SPARK_HOME 可能指向错误的路径
设置export PYSPARK_SUBMIT_ARGS="--master local[2]" - 这是您希望PySpark 开始的配置。

【讨论】：