【发布时间】:2017-11-08 19:11:40
【问题描述】:
我在 python 3 中使用 PySpark,我刚刚用命令下载了 pyspark
conda install pyspark
并成功下载,我将文件导入Jupyter Notebook,然后尝试运行以下代码
sc1 = sp.SparkContext.getOrCreate()
我收到以下异常
Exception Traceback (most recent call last)
<ipython-input-9-a4daca8dcb9d> in <module>()
----> 1 sc1 = sp.SparkContext.getOrCreate()
~\Anaconda3\lib\site-packages\pyspark\context.py in getOrCreate(cls, conf)
332 with SparkContext._lock:
333 if SparkContext._active_spark_context is None:
--> 334 SparkContext(conf=conf or SparkConf())
335 return SparkContext._active_spark_context
336
~\Anaconda3\lib\site-packages\pyspark\context.py in __init__(self, master, appName, sparkHome, pyFiles, environment, batchSize, serializer, conf, gateway, jsc, profiler_cls)
113 """
114 self._callsite = first_spark_call() or CallSite(None, None, None)
--> 115 SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)
116 try:
117 self._do_init(master, appName, sparkHome, pyFiles, environment, batchSize, serializer,
~\Anaconda3\lib\site-packages\pyspark\context.py in _ensure_initialized(cls, instance, gateway, conf)
281 with SparkContext._lock:
282 if not SparkContext._gateway:
--> 283 SparkContext._gateway = gateway or launch_gateway(conf)
284 SparkContext._jvm = SparkContext._gateway.jvm
285
~\Anaconda3\lib\site-packages\pyspark\java_gateway.py in launch_gateway(conf)
93 callback_socket.close()
94 if gateway_port is None:
---> 95 raise Exception("Java gateway process exited before sending the driver its port number")
96
97 # In Windows, ensure the Java child processes do not linger after Python has exited.
Exception: Java gateway process exited before sending the driver its port number
我尝试了几个解决方案,但没有一个有效.. 有什么建议吗??
【问题讨论】:
-
我得到了 windows 的解决方案,medium.com/@GalarnykMichael/… 按照此链接上的说明进行操作,现在一切正常
-
很遗憾 conda install pyspark 没有更简单的解决方案
标签: python-3.x exception pyspark