PySpark 数据框查看不适用于创建的数据框

Question

我正在尝试为 spark sql 创建一个视图，但我无法从字符串列表创建它。

所以我决定按照 pyspark.sql 文档逐字逐句，还是不行：

testd = [{'name': 'Alice', 'age': 1}]
spark.createDataFrame(testd).collect()

错误跟踪：

Py4JJavaError                             Traceback (most recent call last)
<ipython-input-55-d4321f74b607> in <module>()
      1 testd = [{'name': 'Alice', 'age': 1}]
      2 
----> 3 spark.createDataFrame(testd).collect()

/opt/app/anaconda2/python27/lib/python2.7/site-packages/pyspark/sql/dataframe.pyc in collect(self)
    389         """
    390         with SCCallSiteSync(self._sc) as css:
--> 391             port = self._jdf.collectToPython()
    392         return list(_load_from_socket(port, BatchedSerializer(PickleSerializer())))
    393 

/opt/app/anaconda2/python27/lib/python2.7/site-packages/py4j/java_gateway.pyc in __call__(self, *args)
   1131         answer = self.gateway_client.send_command(command)
   1132         return_value = get_return_value(
-> 1133             answer, self.gateway_client, self.target_id, self.name)
   1134 
   1135         for temp_arg in temp_args:

/opt/app/anaconda2/python27/lib/python2.7/site-packages/pyspark/sql/utils.pyc in deco(*a, **kw)
     61     def deco(*a, **kw):
     62         try:
---> 63             return f(*a, **kw)
     64         except py4j.protocol.Py4JJavaError as e:
     65             s = e.java_exception.toString()

/opt/app/anaconda2/python27/lib/python2.7/site-packages/py4j/protocol.pyc in get_return_value(answer, gateway_client, target_id, name)
    317                 raise Py4JJavaError(
    318                     "An error occurred while calling {0}{1}{2}.\n".
--> 319                     format(target_id, ".", name), value)
    320             else:
    321                 raise Py4JError(

Py4JJavaError: An error occurred while calling o896.collectToPython.
....
TypeError: range() integer end argument expected, got list.

同时，教程中的这个：

l = [('Alice', 1)]
spark.createDataFrame(l, ['name', 'age']).collect()

得到相同的基本错误跟踪'range() integer end argument expected, got list.'

这是怎么回事？？？

以下是我启动 spark 实例的方法：

os.environ['SPARK_HOME']='/path/to/spark2-client'
os.environ['PY4JPATH']='/path/to/spark2-client/python/lib/py4j-0.10.4-src.zip'
sys.path.insert(0, os.path.join(os.environ['SPARK_HOME'],'python'))
sys.path.insert(1, os.path.join(os.environ['SPARK_HOME'],'python/lib'))
os.environ['HADOOP_CONF_DIR']='/etc/hadoop/conf'
os.environ['MASTER']="yarn"
os.environ['SPARK_MAJOR_VERSION']="2"
spark = (SparkSession
            .builder
            .appName('APPNAME')
            .config("spark.executor.instances","8")
            .config("spark.executor.memory","32g")
            .config("spark.driver.memory","64g")
            .config("spark.driver.maxResultSize","32g")
            .enableHiveSupport()
            .getOrCreate())

所有其他 spark 函数都可以正常工作，包括 hive 查询、数据帧连接等。只有当我尝试从本地内存创建某些东西时，它才不起作用。

感谢您提供任何见解。

Answer 1

spark.createDataFrame(['Alice',1], ['name', 'age']).collect()

根据文档 https://spark.apache.org/docs/2.3.1/api/python/pyspark.sql.html https://spark.apache.org/docs/2.3.1/api/python/_modules/pyspark/sql/dataframe.html#DataFrame.unionByName