如何在纱线客户端上从 jupyter 运行 spark

Question

我使用 cloudera manager 部署了一个集群并安装了 spark parcel， 在 shell 中输入 pyspark 时，它可以工作，但在 jupyter 上运行以下代码会引发异常

代码

import sys
import py4j
from pyspark.sql import SparkSession
from pyspark import SparkContext, SparkConf
conf = SparkConf()
conf.setMaster('yarn-client')
conf.setAppName('SPARK APP')
sc = SparkContext(conf=conf)
# sc= SparkContext.getOrCreate()
# sc.stop()

def mod(x):
    import numpy as np
    return (x, np.mod(x, 2))

rdd = sc.parallelize(range(1000)).map(mod).take(10)
print (rdd)

例外

/usr/lib/python3.6/site-packages/pyspark/context.py in _do_init(self, master, appName, sparkHome, pyFiles, environment, batchSize, serializer, conf, jsc, profiler_cls)
    187         self._accumulatorServer = accumulators._start_update_server(auth_token)
    188         (host, port) = self._accumulatorServer.server_address
--> 189         self._javaAccumulator = self._jvm.PythonAccumulatorV2(host, port, auth_token)
    190         self._jsc.sc().register(self._javaAccumulator)
    191 

TypeError: 'JavaPackage' object is not callable

Answer 1

搜索升技后，spark使用的版本1.6与python3.7不兼容，必须使用python2.7运行