根据一列获取不同的行答案

【问题标题】：Get distinct rows based on one column根据一列获取不同的行
【发布时间】：2019-08-13 11:03:40
【问题描述】：

+---------------+---------+-----------------+-------+-------------------+-----------+--------------------+--------------------+---------------+-------+-------------------+-------------------+
|ID_NOTIFICATION|ID_ENTITE|ID_ENTITE_GARANTE|CD_ETAT|DT_ETAT            |CD_ANOMALIE|CD_TYPE_DESTINATAIRE|CD_TYPE_EVENEMENT   |CD_SYS_APPELANT|TYP_MVT|DT_DEBUT           |DT_FIN             |
+---------------+---------+-----------------+-------+-------------------+-----------+--------------------+--------------------+---------------+-------+-------------------+-------------------+
|3110305        |GNE      |GNE              |AT     |2019-06-12 00:03:14|null       |null                |REL_CP_ULTIME_PAPIER|SIGMA          |C      |2019-06-12 00:03:22|2019-06-12 00:03:32|
|3110305        |GNE      |GNE              |AN     |2019-06-12 00:03:28|017        |IDGRC               |REL_CP_ULTIME_PAPIER|SIGMA          |M      |2019-06-12 00:03:22|2019-06-12 15:08:43|
|3110305        |GNE      |GNE              |AN     |2019-06-12 00:03:28|017        |IDGRC               |REL_CP_ULTIME_PAPIER|SIGMA          |M      |2019-06-12 00:03:22|2019-06-12 15:10:06|
|3110305        |GNE      |GNE              |AN     |2019-06-12 15:10:02|017        |IDGRC               |REL_CP_ULTIME_PAPIER|SIGMA          |M      |2019-06-12 00:03:22|2019-06-12 15:10:51|
|3110305        |GNE      |GNE              |AN     |2019-06-12 15:10:02|017        |IDGRC               |REL_CP_ULTIME_PAPIER|SIGMA          |M      |2019-06-12 00:03:22|2019-06-12 15:11:35|

有没有办法在每个不同的 CD_ETAT 列中获取一行？在这种情况下，它将是前两行。

类似于this SQL solution 但请在Scala 中使用DF 函数。谢谢你

【问题讨论】：

标签： scala dataframe apache-spark distinct

【解决方案1】：

你可以用partitionByCD_ETAT窗口函数并选择orderBy得到第一个

import org.apache.spark.sql.expressions.Window
import org.apache.spark.sql.functions._

val window = Window.partitionBy("CD_ETAT").orderBy("DT_ETAT")

df.withColumn("row_num", row_number().over(window))
  .filter($"row_num" === 1)
  .drop("row_num")

输出：

+---------------+---------+-----------------+-------+-------------------+-----------+--------------------+--------------------+---------------+-------+-------------------+-------------------+
|ID_NOTIFICATION|ID_ENTITE|ID_ENTITE_GARANTE|CD_ETAT|            DT_ETAT|CD_ANOMALIE|CD_TYPE_DESTINATAIRE|   CD_TYPE_EVENEMENT|CD_SYS_APPELANT|TYP_MVT|           DT_DEBUT|             DT_FIN|
+---------------+---------+-----------------+-------+-------------------+-----------+--------------------+--------------------+---------------+-------+-------------------+-------------------+
|        3110305|      GNE|              GNE|     AT|2019-06-12 00:03:14|       null|                null|REL_CP_ULTIME_PAPIER|          SIGMA|      C|2019-06-12 00:03:22|2019-06-12 00:03:32|
|        3110305|      GNE|              GNE|     AN|2019-06-12 00:03:28|        017|               IDGRC|REL_CP_ULTIME_PAPIER|          SIGMA|      M|2019-06-12 00:03:22|2019-06-12 15:08:43|
+---------------+---------+-----------------+-------+-------------------+-----------+--------------------+--------------------+---------------+-------+-------------------+-------------------+

【讨论】：

感谢您的回答。我有这个错误：:83: error: not found: value Window I am on Spark2 shell.
你应该导入import org.apache.spark.sql.expressions.Window