在 Dataframe 中创建具有嵌套列表聚合的列

Question

我有这种结构的数据框

val df = Seq(
  ("john", "tomato", 1),
  ("john", "carrot", 4),
  ("bill", "apple", 1),
  ("john", "tomato", 2),
  ("bill", "taco", 2)      
).toDF("name", "food", "price")

我需要像这样聚合嵌套列表

name | acc                       |
-----+---------------------------+
john |[(tomato, 3), (carrot, 4)] |
bill |[(apple, 1), (taco,2 )]    |

我尝试过这种方式，但它不正确。

 dff.groupBy($"name")
  .agg(collect_list(struct($"food", $"price")).as("foods"))
  .show(false)

+----+------------------------------------+
|name|set                                 |
+----+------------------------------------+
|john|[[tomato,1], [carrot,4], [tomato,2]]|
|bill|[[apple,1], [taco,2]]               |
+----+------------------------------------+

我怎样才能得到它？

Answer 1

您需要两个groupBy 和aggregations 并使用collect_list、struct 和sum 内置函数

import org.apache.spark.sql.functions._
df.groupBy("name", "food").agg(sum("price").as("price"))
  .groupBy("name").agg(collect_list(struct("food", "price")).as("acc"))

您将输出dataframe 为

+----+------------------------+
|name|acc                     |
+----+------------------------+
|john|[[carrot,4], [tomato,3]]|
|bill|[[taco,2], [apple,1]]   |
+----+------------------------+