pyspark 从熊猫创建数据框，其中包含一列元组列表答案

【问题标题】：pyspark create dataframe from pandas with a column of list of tuplespyspark 从熊猫创建数据框，其中包含一列元组列表
【发布时间】：2021-02-08 10:01:10
【问题描述】：

我正在尝试从 pandas 数据帧创建一个 pyspark 数据帧。

import pandas as pd
from pyspark.sql.types import StructType, StructField, IntegerType, DoubleType

a_dict = {0: [(0, 9.821), (1, 82.185)]}

a_pd = pd.DataFrame.from_dict(a_dict.items())
a_pd.columns = ["row_num", "val"]

a_str = StructType([StructField("id", IntegerType(), True), StructField("prob", DoubleType(), True)])
 my_schema = StructType([ StructField("row_num", LongType(), True),StructField("val", list(a_str), True)]) # error 
                   
a_df = spark.createDataFrame(a_pd, schema=my_schema)

错误：

 AssertionError: dataType [StructField(id,IntegerType,true), StructField(prob,DoubleType,true)] should be an instance of <class 'pyspark.sql.types.DataType'>

如何定义一个有效的架构

 list of tuple of (int, DoubleType)

这样pyspark也能理解？

谢谢

【问题讨论】：

标签： pandas dataframe apache-spark pyspark

【解决方案1】：

对于值列表，您必须使用 ArrayType。以下是您通过示例复制的代码。

import pandas as pd
from pyspark.sql.types import StructType, StructField, IntegerType, DoubleType

a_dict = {0: [(0, 9.821), (1, 82.185)],
          1: [(0, 9.821), (1, 8.10), (3, 2.385)],
          2: [(0, 9.821), (1, 1.4485), (4, 5.15), (5, 6.104)]}


a_pd = pd.DataFrame.from_dict(a_dict.items())
a_pd.columns = ["row_num", "val"]


print(a_pd.head())



a_str = StructType([StructField("id", IntegerType(), True), StructField("prob", DoubleType(), True)])
my_schema = StructType([StructField("row_num", LongType(), True), StructField("val", ArrayType(a_str), True)])  # error

a_df = sqlContext.createDataFrame(a_pd, schema=my_schema)


print(a_df.show(truncate=False))
print(a_df.printSchema())

输出：

+-------+------------------------------------------------+
|row_num|val                                             |
+-------+------------------------------------------------+
|0      |[[0, 9.821], [1, 82.185]]                       |
|1      |[[0, 9.821], [1, 8.1], [3, 2.385]]              |
|2      |[[0, 9.821], [1, 1.4485], [4, 5.15], [5, 6.104]]|
+-------+------------------------------------------------+

root
 |-- row_num: long (nullable = true)
 |-- val: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- id: integer (nullable = true)
 |    |    |-- prob: double (nullable = true)

【讨论】：