Scala - 将 Dataframe 转换为 json 文件，并为不同的不同列从列名和列值生成键

Question

我想将 Spark DataFrame 转换为 Json 文件。下面是输入和输出格式。 任何帮助表示赞赏。

Input :

+-------------------------+
|Name|Age|City |Data      |
+-------------------------+
|Ram |30 |Delhi|[A -> ABC]|
|-------------------------|
|Shan|25 |Delhi|[X -> XYZ]|  
|-------------------------|
|Riya|12 |U.P. |[M -> MNO]|
+-------------------------+

Output : 

{"Name":"Ram","Age":"30","City":"Delhi","Delhi":{"A":"ABC"}}
{"Name":"Shan","Age":"25","City":"Delhi","Delhi":{"X":"XYZ"}}
{"Name":"Riya","Age":"12","City":"U.P.","U.P.":{"M":"MNO"}}

Answer 1

您可以在 DataFrame 上调用write.json。

val df: DataFrame = ....
df.write.json("/jsonFilPath")

这是一个使用数据集的示例

scala> case class Data(key: String, value: String)

scala> case class Person(name: String, age: Long, city: String, data: Data)

scala> val peopleDS = Seq(Person("Ram", 30, "Delhi", Data("A", "ABC")), Person("Shan", 25, "Delhi", Data("X", "XYZ")), Person("Riya", 12, "U.P", Data("M", "MNO"))).toDS()

scala> peopleDS.show()
+----+---+-----+--------+
|name|age| city|    data|
+----+---+-----+--------+
| Ram| 30|Delhi|[A, ABC]|
|Shan| 25|Delhi|[X, XYZ]|
|Riya| 12|  U.P|[M, MNO]|
+----+---+-----+--------+

scala> peopleDS.write.json("pathToData/people")

然后你会在给定的文件夹中找到写入的 json 文件。

> cd pathToData/people
> ls -l

part-00000-6bd00826-5a8e-4ab9-bfb0-65d722394108-c000.json

> cat part-00000-6bd00826-5a8e-4ab9-bfb0-65d722394108-c000.json

{"name":"Ram","age":30,"city":"Delhi","data":{"key":"A","value":"ABC"}}
{"name":"Shan","age":25,"city":"Delhi","data":{"key":"X","value":"XYZ"}}
{"name":"Riya","age":12,"city":"U.P","data":{"key":"M","value":"MNO"}}

Answer 2

Scala：从你的数据出发，

val df = Seq(("Ram",30,"Delhi",Map("A" -> "ABC")), ("Shan",25,"Delhi",Map("X" -> "XYZ")), ("Riya",12,"U.P.",Map("M" -> "MNO"))).toDF("Name", "Age", "City", "Data")
df.show

// +----+---+-----+----------+
// |Name|Age| City|      Data|
// +----+---+-----+----------+
// | Ram| 30|Delhi|[A -> ABC]|
// |Shan| 25|Delhi|[X -> XYZ]|
// |Riya| 12| U.P.|[M -> MNO]|
// +----+---+-----+----------+

要将密钥更改为City 而不是Data，

val df2 = df.groupBy("Name", "Age", "City").pivot("City").agg(first("Data"))
df2.show

// +----+---+-----+----------+----------+
// |Name|Age| City|     Delhi|      U.P.|
// +----+---+-----+----------+----------+
// |Riya| 12| U.P.|      null|[M -> MNO]|
// |Shan| 25|Delhi|[X -> XYZ]|      null|
// | Ram| 30|Delhi|[A -> ABC]|      null|
// +----+---+-----+----------+----------+

然后使用toJson 并收集它。

val jsonArray = df.toJSON.collect
jsonArray.foreach(println)

它会打印出如下结果：

{"Name":"Riya","Age":12,"City":"U.P.","U.P.":{"M":"MNO"}}
{"Name":"Shan","Age":25,"City":"Delhi","Delhi":{"X":"XYZ"}}
{"Name":"Ram","Age":30,"City":"Delhi","Delhi":{"A":"ABC"}}