无法在 Jupyter 笔记本中启动 Spark 会话

Question

首先，这不是 question 的副本。我刚刚在 Windows 中安装了 pyspark，设置了 SPARK_HOME 变量并运行了 findspark.init() 以确保没有安装问题。

运行 pyspark shell，会自动创建 spark (SparkSession) 变量，一切正常，但是当我想从 Jupyter 启动 spark 会话时，出现以下错误

import pyspark
from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
    Exception                                 Traceback (most recent call last)
<ipython-input-20-39c4e6ac7c9b> in <module>
      3 from pyspark.sql import SparkSession
      4 
----> 5 spark = SparkSession.builder.getOrCreate()
      6 

c:\users\ahg01\miniconda3\lib\site-packages\pyspark\sql\session.py in getOrCreate(self)
    184                             sparkConf.set(key, value)
    185                         # This SparkContext may be an existing one.
--> 186                         sc = SparkContext.getOrCreate(sparkConf)
    187                     # Do not update `SparkConf` for existing `SparkContext`, as it's shared
    188                     # by all sessions.

c:\users\ahg01\miniconda3\lib\site-packages\pyspark\context.py in getOrCreate(cls, conf)
    374         with SparkContext._lock:
    375             if SparkContext._active_spark_context is None:
--> 376                 SparkContext(conf=conf or SparkConf())
    377             return SparkContext._active_spark_context
    378 

c:\users\ahg01\miniconda3\lib\site-packages\pyspark\context.py in __init__(self, master, appName, sparkHome, pyFiles, environment, batchSize, serializer, conf, gateway, jsc, profiler_cls)
    131                 " is not allowed as it is a security risk.")
    132 
--> 133         SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)
    134         try:
    135             self._do_init(master, appName, sparkHome, pyFiles, environment, batchSize, serializer,

c:\users\ahg01\miniconda3\lib\site-packages\pyspark\context.py in _ensure_initialized(cls, instance, gateway, conf)
    323         with SparkContext._lock:
    324             if not SparkContext._gateway:
--> 325                 SparkContext._gateway = gateway or launch_gateway(conf)
    326                 SparkContext._jvm = SparkContext._gateway.jvm
    327 

c:\users\ahg01\miniconda3\lib\site-packages\pyspark\java_gateway.py in launch_gateway(conf, popen_kwargs)
    103 
    104             if not os.path.isfile(conn_info_file):
--> 105                 raise Exception("Java gateway process exited before sending its port number")
    106 
    107             with open(conn_info_file, "rb") as info:

Exception: Java gateway process exited before sending its port number

没有运行现有的 SparkContext，为什么我会收到此消息？没有预定义的 spark 或 sc 变量，所以我不知道如何调试这个问题

Answer 1

试试这个应该可以的

spark = SparkSession.builder.appName('Test')