创建结构 PySpark 的 DataFrame

Question

enter image description here我怎样才能创建一个空结构的数据框。？ 谢谢。

dataxx = []
schema = StructType(
[
    StructField('Info1',
        StructType([
            StructField('fld', IntegerType(),True),
            StructField('fld1', IntegerType(),True),
            StructField('fld2', IntegerType(),True),
            StructField('fld3', IntegerType(),True),
            StructField('fld4', IntegerType(),True),   
            ])
    ),
]
)
df = sqlCtx.createDataFrame(dataxx, schema)

感谢您的帮助

Answer 1

如果您想创建具有特定架构但不包含数据的 DataFrame，只需向 createDataFrame 函数提供空列表即可：

from pyspark.sql.types import *

schema = StructType(
[
    StructField('Info1',
        StructType([
            StructField('fld', IntegerType(),True),
            StructField('fld1', IntegerType(),True),
            StructField('fld2', IntegerType(),True),
            StructField('fld3', IntegerType(),True),
            StructField('fld4', IntegerType(),True),   
            ])
    ),
]
)
df = spark.createDataFrame([], schema)

df.printSchema()

root
 |-- Info1: struct (nullable = true)
 |    |-- fld: integer (nullable = true)
 |    |-- fld1: integer (nullable = true)
 |    |-- fld2: integer (nullable = true)
 |    |-- fld3: integer (nullable = true)
 |    |-- fld4: integer (nullable = true)

这里spark 是sparkSession。