scala/spark - 对数据框进行分组并从其他列中选择值作为数据框答案

【问题标题】：scala/spark - group dataframe and select values from other column as dataframescala/spark - 对数据框进行分组并从其他列中选择值作为数据框
【发布时间】：2021-01-31 14:58:32
【问题描述】：

我正在尝试从 Scala/Spark 中的现有 DataFrame 创建一个新的 DataFrame。

下面是我现有的DataFrame：

+----------+-----------------------+
|     group|                  value|
+----------+-----------------------+
|         4|[blah blah blah blah...|
|         0|[blah blah blah blah...|
|         1|[blah blah blah blah...|
|         1|[blah blah blah blah...|
|         0|[blah blah blah blah...|
|         2|[blah blah blah blah...|
|         0|[blah blah blah blah...|
|         2|[blah blah blah blah...|
//and so on
+----------+--------------------+

现在我想按 group 列对 DataFrame 上方进行分组，然后对于每个组，将 value 聚合到原始值列表中，以生成如下所示的内容：

+----------+---------------------------------+
|     group|                            value|
+----------+---------------------------------+
|         0|[[blah blah...],[blah blah...]...|
|         1|[[blah blah...],[blah blah...]...|
|         2|[[blah blah...],[blah blah...]...|
|         3|[[blah blah...],[blah blah...]...|
|         4|[[blah blah...],[blah blah...]...|
+----------+---------------------------------+

我怎样才能实现它？

【问题讨论】：

标签： scala dataframe apache-spark

【解决方案1】：

试试下面的代码。

scala> df.show(false)
+-----+---------------------+
|group|value                |
+-----+---------------------+
|4    |[blah blah blah blah]|
|0    |[blah blah blah blah]|
|1    |[blah blah blah blah]|
|1    |[blah blah blah blah]|
|0    |[blah blah blah blah]|
|2    |[blah blah blah blah]|
|0    |[blah blah blah blah]|
|2    |[blah blah blah blah]|
+-----+---------------------+

scala> 
df
.groupBy($"group")
.agg(collect_list($"value").as("value"))
.orderBy($"group".asc)
.show(false)

+-----+---------------------------------------------------------------------+
|group|value                                                                |
+-----+---------------------------------------------------------------------+
|0    |[[blah blah blah blah], [blah blah blah blah], [blah blah blah blah]]|
|1    |[[blah blah blah blah], [blah blah blah blah]]                       |
|2    |[[blah blah blah blah], [blah blah blah blah]]                       |
|4    |[[blah blah blah blah]]                                              |
+-----+---------------------------------------------------------------------+

【讨论】：