将后续记录中的时间戳与 pandas 进行比较

Question

我有一个 30000 KB 的大型数据集（保存为“熊猫”数据框），用于专家和用户之间的聊天对话。

每一行代表专家或用户发送的消息。我想测量用户发送的第二条消息和专家的第二条响应之间的时间。

（请注意，有时专家和用户会键入并发送一组连续的消息，这些消息应作为一条大消息处理，并且还注意到有时某些数据丢失，例如 - sessionId 111 中的消息 0）

例如：在 sessionId 222 中，我想测量索引 3 和索引 4 之间的时间（本例中为 22 分钟）

这里是以列表形式呈现的数据：

import datetime
import matplotlib.pyplot as plt
import pandas as pd
import numpy as np

df = [[222.0, 0.0, 'user', '12/6/2017 20:12'],
 [222.0, 1.0, 'user', '12/6/2017 20:41'],
 [222.0, 2.0, 'expert', '12/6/2017 21:15'],
 [222.0, 3.0, 'user', '12/6/2017 21:45'],
 [222.0, 4.0, 'expert', '12/6/2017 22:07'],
 [222.0, 5.0, 'expert', '12/6/2017 23:36'],
 [443.0, 0.0, 'user', '12/6/2017 20:41'],
 [443.0, 1.0, 'expert', '12/6/2017 21:15'],
 [443.0, 2.0, 'user', '12/6/2017 21:45'],
 [111.0, 1.0, 'expert', '12/6/2017 21:45'],
 [111.0, 2.0, 'user', '12/6/2017 22:07'],
 [243.0, 0.0, 'user', '12/6/2017 20:12'],
 [243.0, 1.0, 'expert', '12/6/2017 20:41'],
 [243.0, 2.0, 'user', '12/6/2017 21:15'],
 [243.0, 3.0, 'expert', '12/6/2017 21:45'],
 [243.0, 4.0, 'user', '12/6/2017 22:07'],
 [243.0, 5.0, 'expert', '12/6/2017 23:36'],
 [243.0, 6.0, 'user', '12/7/2017 0:05'],
 [243.0, 7.0, 'user', '12/7/2017 0:58'],
 [243.0, 8.0, 'user', '12/7/2017 0:58']]

我尝试了什么：

一个。使用 pd.groupby "sessionId" 和 "interaction"

b.创建一个新的 userType 列向下移动 1 行

c。将原始 userType 与转移后的 userType 进行比较并找出不匹配的地方

d。在每三个不匹配中 - 找出不匹配的消息和前一个消息（交互）之间的时间。

您能否向我展示这种方法或可能的其他方法的工作示例？

Answer 1

如果尚未完成，则转换时间戳

df['timestamp'] = pd.to_datetime(df.timestamp)

#Order by session and time before taking differences.
df.sort_values('sessionId','timestamp', inplace = True)

df['delta_time'] = df.groupby('sessionId').timestamp.diff()

结果：

    sessionId   interaction userType    timestamp   delta_time
9   111.0   1.0 expert  2017-12-06 21:45:00 
10  111.0   2.0 user    2017-12-06 22:07:00 0 days 00:22:00.000000000
0   222.0   0.0 user    2017-12-06 20:12:00 
1   222.0   1.0 user    2017-12-06 20:41:00 0 days 00:29:00.000000000
2   222.0   2.0 expert  2017-12-06 21:15:00 0 days 00:34:00.000000000
3   222.0   3.0 user    2017-12-06 21:45:00 0 days 00:30:00.000000000
4   222.0   4.0 expert  2017-12-06 22:07:00 0 days 00:22:00.000000000
5   222.0   5.0 expert  2017-12-06 23:36:00 0 days 01:29:00.000000000
11  243.0   0.0 user    2017-12-06 20:12:00 
12  243.0   1.0 expert  2017-12-06 20:41:00 0 days 00:29:00.000000000
13  243.0   2.0 user    2017-12-06 21:15:00 0 days 00:34:00.000000000
14  243.0   3.0 expert  2017-12-06 21:45:00 0 days 00:30:00.000000000
15  243.0   4.0 user    2017-12-06 22:07:00 0 days 00:22:00.000000000
16  243.0   5.0 expert  2017-12-06 23:36:00 0 days 01:29:00.000000000
17  243.0   6.0 user    2017-12-07 00:05:00 0 days 00:29:00.000000000
18  243.0   7.0 user    2017-12-07 00:58:00 0 days 00:53:00.000000000
19  243.0   8.0 user    2017-12-07 00:58:00 0 days 00:00:00.000000000
6   443.0   0.0 user    2017-12-06 20:41:00 
7   443.0   1.0 expert  2017-12-06 21:15:00 0 days 00:34:00.000000000
8   443.0   2.0 user    2017-12-06 21:45:00 0 days 00:30:00.000000000

另一种方法是为下一条消息创建一个新列，如下所示：

df['nextMessage'] = df.groupby('sessionId').timestamp.shift(-1)
df['deltaTime'] = df.nextMessage- df.timestamp

结果

sessionId   interaction,    userType    timestamp   nextMessage deltaTime
9   111.0   1.0 expert  2017-12-06 21:45:00 2017-12-06 22:07:00 0 days 00:22:00.000000000
10  111.0   2.0 user    2017-12-06 22:07:00     
0   222.0   0.0 user    2017-12-06 20:12:00 2017-12-06 20:41:00 0 days 00:29:00.000000000
1   222.0   1.0 user    2017-12-06 20:41:00 2017-12-06 21:15:00 0 days 00:34:00.000000000
2   222.0   2.0 expert  2017-12-06 21:15:00 2017-12-06 21:45:00 0 days 00:30:00.000000000
3   222.0   3.0 user    2017-12-06 21:45:00 2017-12-06 22:07:00 0 days 00:22:00.000000000
4   222.0   4.0 expert  2017-12-06 22:07:00 2017-12-06 23:36:00 0 days 01:29:00.000000000
5   222.0   5.0 expert  2017-12-06 23:36:00     
11  243.0   0.0 user    2017-12-06 20:12:00 2017-12-06 20:41:00 0 days 00:29:00.000000000
12  243.0   1.0 expert  2017-12-06 20:41:00 2017-12-06 21:15:00 0 days 00:34:00.000000000
13  243.0   2.0 user    2017-12-06 21:15:00 2017-12-06 21:45:00 0 days 00:30:00.000000000
14  243.0   3.0 expert  2017-12-06 21:45:00 2017-12-06 22:07:00 0 days 00:22:00.000000000
15  243.0   4.0 user    2017-12-06 22:07:00 2017-12-06 23:36:00 0 days 01:29:00.000000000
16  243.0   5.0 expert  2017-12-06 23:36:00 2017-12-07 00:05:00 0 days 00:29:00.000000000
17  243.0   6.0 user    2017-12-07 00:05:00 2017-12-07 00:58:00 0 days 00:53:00.000000000
18  243.0   7.0 user    2017-12-07 00:58:00 2017-12-07 00:58:00 0 days 00:00:00.000000000
19  243.0   8.0 user    2017-12-07 00:58:00     
6   443.0   0.0 user    2017-12-06 20:41:00 2017-12-06 21:15:00 0 days 00:34:00.000000000
7   443.0   1.0 expert  2017-12-06 21:15:00 2017-12-06 21:45:00 0 days 00:30:00.000000000
8   443.0   2.0 user    2017-12-06 21:45:00