Spark 窗口上的条件滞后函数

Question

我有一个数据框，其中值 label 与 (id, bin, date, hour) 相关联：

+----------+----+-----+---+-------------------+
|      date|hour|   id|bin|              label|
+----------+----+-----+---+-------------------+
|2019_12_20|   8|    1|  0|  151.7050821002368|
|2019_12_20|   8|    1|  2| 101.13672140015788|
|2019_12_20|   8|    1|  3| 16.856120233359647|
...

我想在前一天的同一小时、前一天的一小时前等将多个列附加到与label 对应的此数据帧中。我知道如何使用滞后函数获得第一个：

val dateWindow = Window.partitionBy($"id", $"bin").orderBy($"hour", $"date")
val expandedDf = data.withColumn("yesterdaySameHour", lag($"label", 1, 0.0).over(dateWindow))

但是，我不知道如何在前一天在hour - 1 获得值label。有没有办法有条件滞后，我可以过滤掉大于或等于当前行小时的hour？如果没有，这样做的正确方法是什么？

非常感谢。

Answer 1

您必须根据您的目的指定Window 函数。您可能需要使用两次lag 函数。

import org.apache.spark.sql.expressions.Window

val dW = Window.partitionBy("id", "bin", "hour").orderBy("date")
val hW = Window.partitionBy("id", "bin", "date").orderBy("hour")

df.withColumn("yesterdaySameHour", lag("label", 1, 0.0).over(dW))
  .withColumn("todayPreviousHour", lag("label", 1, 0.0).over(hW))
  .withColumn("yestedayPreviousHour", lag(lag("label", 1, 0.0).over(dW), 1, 0.0).over(hW))
  .orderBy("date", "hour", "bin")
  .show(false)

这会给你结果：

+----------+----+---+---+-----+-----------------+-----------------+--------------------+
|date      |hour|id |bin|label|yesterdaySameHour|todayPreviousHour|yestedayPreviousHour|
+----------+----+---+---+-----+-----------------+-----------------+--------------------+
|2019_12_19|7   |1  |0  |-1   |0                |0                |0                   |
|2019_12_19|7   |1  |2  |-2   |0                |0                |0                   |
|2019_12_19|7   |1  |3  |-3   |0                |0                |0                   |
|2019_12_19|8   |1  |0  |1    |0                |-1               |0                   |
|2019_12_19|8   |1  |2  |2    |0                |-2               |0                   |
|2019_12_19|8   |1  |3  |3    |0                |-3               |0                   |
|2019_12_20|7   |1  |0  |4    |-1               |0                |0                   |
|2019_12_20|7   |1  |2  |5    |-2               |0                |0                   |
|2019_12_20|7   |1  |3  |6    |-3               |0                |0                   |
|2019_12_20|8   |1  |0  |7    |1                |4                |-1                  |
|2019_12_20|8   |1  |2  |8    |2                |5                |-2                  |
|2019_12_20|8   |1  |3  |9    |3                |6                |-3                  |
+----------+----+---+---+-----+-----------------+-----------------+--------------------+