使用嵌套的 json 数据 PySpark 在 Spark df 中读取 json 文件

Question

我需要将多个 JSON 文件读入 spark df。 JSON 数据如下所示：

{"f0_":{"id":"138307057680","ActionName":"Complete","Time":"2020-04-23-12:40:04"}}
{"f0_":{"id":"138313115245","ActionName":"Midpoint","Time":"2020-06-16-20:41:16"}}

我需要去掉包含所有列的第一个键。 我试过了：

jsonFiles = spark.read.json("Resources") # path to all json files
jsonFile.printSchema()

输出是：

root
 |-- f0_: struct (nullable = true)
 |    |-- id string (nullable = true)
 |    |-- ActionName: string (nullable = true)
 |    |-- Time: string (nullable = true)

Answer 1

这对您来说可能是一个可行的解决方案----

#在这里创建数据框

df_new = spark.createDataFrame([(str({"f0_":{"id":"138307057680","ActionName":"Complete","Time":"2020-04-23-12:40:04"}})), (str({"f0_":{"id":"138313115245","ActionName":"Midpoint","Time":"2020-06-16-20:41:16"}}))],T.StringType())

df_new = df_new.withColumn('col', F.from_json("value",T.MapType(T.StringType(), T.StringType())))
df_new = df_new.select(F.explode("col").alias("x", "y"))


df_new = df_new.withColumn('y', F.from_json("y",T.MapType(T.StringType(), T.StringType())))

df_new = df_new.withColumn("id", df_new.y.getItem("id")).withColumn("ActionName", df_new.y.getItem("ActionName")).withColumn("Time", df_new.y.getItem("Time"))
df_new.show(truncate=False)

在此处输出

+---+-------------------------------------------------------------------------+------------+----------+-------------------+
|x  |y                                                                        |id          |ActionName|Time               |
+---+-------------------------------------------------------------------------+------------+----------+-------------------+
|f0_|[id -> 138307057680, ActionName -> Complete, Time -> 2020-04-23-12:40:04]|138307057680|Complete  |2020-04-23-12:40:04|
|f0_|[id -> 138313115245, ActionName -> Midpoint, Time -> 2020-06-16-20:41:16]|138313115245|Midpoint  |2020-06-16-20:41:16|
+---+-------------------------------------------------------------------------+------------+----------+-------------------+