【发布时间】:2020-03-19 13:02:05
【问题描述】:
我有如下 JSON 原始数据
{"event" : "login","time" : "2019-11-20 00:14:46","user_id" : 978699}
{"event" : "logout","time" : "2019-11-20 00:14:46","user_id" : 992210}
{"event" : "login","time" : "2019-11-20 00:14:46","user_id" : 823323}
{"event" : "logout","time" : "2019-11-20 00:14:47","user_id" : 978699}
{"event" : "logout","time" : "2019-11-20 00:14:47","user_id" : 823323}
{"event" : "login","time" : "2019-11-20 00:14:50","user_id" : 978699}
{"event" : "logout","time" : "2019-11-20 00:14:57","user_id" : 978699}
我为上述原始数据创建了数据框
val rawDataFrame = sparkSession.read.option("multiline", "true").json(cleanJsonLines)
我需要找出每个用户登录我们系统的秒数。 预期的最终结果如下所示
{"user_id": 978699, "logged_in_sec":8} // (2019-11-20 00:14:47 - 2019-11-20 00:14:46) + (2019-11-20 00:14:57 - 2019-11-20 00:14:50)
{"user_id": 992210, "logged_in_sec":0}
{"user_id": 823323, "logged_in_sec":1}
我是 spark 和 scala 的新手,不知道如何在这里使用窗口函数来解决这个问题。
我不想编写一种程序化的代码方式来迭代每一行数据帧并计算每个 user_id 的先前“登录”和当前“注销”事件之间的差异。
请指导我解决此问题的方法。 感谢阅读。
更新:
原始数据中有一个问题。登录和注销不必总是连续的。我的意思是说每个 user_id 事件也可以按以下顺序排列。多次注销后可能会发生另一次注销,类似地,登录可能会在多次登录事件后发生。
{"event" : "login","time" : "2019-11-20 00:14:46","user_id" : 978699}
{"event" : "logout","time" : "2019-11-20 00:14:46","user_id" : 992210}
{"event" : "login","time" : "2019-11-20 00:14:46","user_id" : 823323}
{"event" : "logout","time" : "2019-11-20 00:14:47","user_id" : 978699}
{"event" : "logout","time" : "2019-11-20 00:14:47","user_id" : 823323}
{"event" : "login","time" : "2019-11-20 00:14:50","user_id" : 978699}
{"event" : "logout","time" : "2019-11-20 00:14:55","user_id" : 978699}
{"event" : "logout","time" : "2019-11-20 00:14:56","user_id" : 978699}
{"event" : "logout","time" : "2019-11-20 00:14:57","user_id" : 978699}
因此,对于请求的聚合。如果我收到任何一组连续的“注销”事件,我必须在第一个“注销”事件之前从一组登录事件中获取最后一次登录并创建一个窗口。
login
login
login <- window start
logout <- window end
logout
login <- window start
logout <- window end
login <- window start
logout <- window end
logout
更新 2
因此,对于 user_id "978699" 窗口按照上述方法将如下所示
窗口 1
{"event" : "login","time" : "2019-11-20 00:14:46","user_id" : 978699}
{"event" : "logout","time" : "2019-11-20 00:14:47","user_id" : 978699}
窗口 2
{"event" : "login","time" : "2019-11-20 00:14:50","user_id" : 978699}
{"event" : "logout","time" : "2019-11-20 00:14:55","user_id" : 978699}
登录时间总和为 (47-46) + (55-50) = 6 秒
【问题讨论】:
标签: scala dataframe apache-spark apache-spark-sql apache-spark-dataset