使用 PySpark 根据另一个表的值从一个表中获取最大值

Question

我有两个 Spark DataFrame，第一个 (Events) 包含以下事件信息：

Event_id	Date	User_id
1	2019-04-19	1
2	2019-05-30	2
3	2020-01-20	1

第二个（User）包含来自用户的信息如下：

Id	User_id	Date	Weight-kg
1	1	2019-04-05	78
2	1	2019-04-17	75
3	2	2019-10-10	50
4	1	2020-02-10	76

我想知道的是如何使用 PySpark 在 Event Date 之前从 User 获取最新的权重值？

此代码的返回必须如下表：

Event_id	Date	User_id	Weight-kg
1	2019-04-19	1	75
2	2019-05-30	2	null
3	2020-01-20	1	75

Answer 1

这个想法是让用户加入事件，然后根据日期对权重进行排名以获得最新的事件

from pyspark.sql import functions as F
from pyspark.sql import Window as W

(event
    # left join to keep all events
    # note the join condition where
    # event's date >= user's date
    .join(
        user,
        on=[
            event['User_id'] == user['User_id'],
            event['Date'] >= user['Date'],
        ],
        how='left'
    )

    # rank user's weight to get the latest
    # based on the dates that already filtered by event's date
    .withColumn('rank_weight', F.rank().over(W.partitionBy(user['User_id']).orderBy(User['Date'].desc())))
    .where(F.col('rank_weight') == 1)
    .drop('rank_weight')
 
    # drop unnecessary columns
    .drop(user['User_id'])
    .drop(user['Date'])
    .drop('Id')


    .orderBy('Event_id')
    .show()
)

# Output
# +--------+----------+-------+------+
# |Event_id|      Date|User_id|Weight|
# +--------+----------+-------+------+
# |       1|2019-04-19|      1|    75|
# |       2|2019-05-30|      2|  null|
# |       3|2020-01-20|      1|    75|
# +--------+----------+-------+------+