Python中日期时间列中多个事件的持续时间答案

【问题标题】：Duration of multiple events from a datetime column in PythonPython中日期时间列中多个事件的持续时间
【发布时间】：2020-10-22 13:17:06
【问题描述】：

我有来自多个运动传感器的以下示例数据 (multiple_sensors.csv)：

sensorid,date_time,value
303,2012-06-25 11:15:35,0
404,2012-06-25 11:15:35,0
101,2012-06-25 11:15:35,0
202,2012-06-25 11:15:35,0
303,2012-06-25 11:15:36,0
404,2012-06-25 11:15:36,0
101,2012-06-25 11:15:36,0
202,2012-06-25 11:15:36,1
303,2012-06-25 11:15:37,0
404,2012-06-25 11:15:37,0
101,2012-06-25 11:15:37,0
202,2012-06-25 11:15:37,1
303,2012-06-25 11:15:38,0
404,2012-06-25 11:15:38,0
101,2012-06-25 11:15:38,0
202,2012-06-25 11:15:38,0
303,2012-06-25 11:15:39,0
404,2012-06-25 11:15:39,1
101,2012-06-25 11:15:39,0
202,2012-06-25 11:15:39,0
303,2012-06-25 11:15:40,0
404,2012-06-25 11:15:40,1
101,2012-06-25 11:15:40,0
202,2012-06-25 11:15:40,0
303,2012-06-25 11:15:41,1
404,2012-06-25 11:15:41,0
101,2012-06-25 11:15:41,0
202,2012-06-25 11:15:41,0
303,2012-06-25 11:15:42,1
404,2012-06-25 11:15:42,0
101,2012-06-25 11:15:42,0
202,2012-06-25 11:15:42,0
303,2012-06-25 11:15:43,1
404,2012-06-25 11:15:43,0
101,2012-06-25 11:15:43,0
202,2012-06-25 11:15:43,0
303,2012-06-25 11:15:44,0

我需要按发生顺序返回每个运动传感器事件的 id 和 duration（请参阅 expected_output.png）。 value 列确定是否触发了动作（1 - 表示已触发动作，0 - 表示无动作），date_time 列表示动作开始或结束的时间。

目前，我设法使用下面的单个运动传感器 (single_sensor.csv) 提取了 id 和持续时间（请参阅 single_sensor_output.png）。

sensorid,date_time,value
202,2012-06-25 00:01:07,0
202,2012-06-25 00:01:08,1
202,2012-06-25 00:01:09,1
202,2012-06-25 00:01:10,0
202,2012-06-25 00:02:12,0
202,2012-06-25 00:02:13,1
202,2012-06-25 00:02:14,1
202,2012-06-25 00:02:15,1
202,2012-06-25 00:02:16,0
202,2012-06-25 00:03:40,0
202,2012-06-25 00:03:41,1
202,2012-06-25 00:03:42,1
202,2012-06-25 00:03:43,1
202,2012-06-25 00:03:44,0
202,2012-06-25 00:05:11,0
202,2012-06-25 00:05:12,1
202,2012-06-25 00:05:13,1
202,2012-06-25 00:05:14,0
202,2012-06-25 00:06:19,0
202,2012-06-25 00:06:20,1
202,2012-06-25 00:06:21,1
202,2012-06-25 00:06:22,0

对于涉及单个传感器的代码，我遵循此处的示例 (Calculate duration between events with pandas)

import pandas as pd
import numpy as np
from pandas import read_csv
from datetime import datetime
from datetime import timedelta

data_time_format = '%Y-%m-%d %H:%M:%S'

df = read_csv('single_sensor.csv')
df['date_time'] = pd.to_datetime(df['date_time'], format=data_time_format)

a = (df['value'] != 1).cumsum().mask(df['value'] == 1)
df['value group'] = a.bfill()

df_final = df.groupby('value group').filter(lambda x: set(x['value']) == set([1,0]))\
           .groupby('value group')['date_time'].agg(['first','last'])\
           .rename(columns={'first':'start','last':'end'})\
           .reset_index()

df_final['id'] = df['sensorid']
df_final['duration'] = df_final['end'].values - df_final['start']
df_final['duration'] = df_final['duration'].dt.total_seconds().astype(int)
print(df_final)

如何扩展它以使用 multiple_sensors.csv

实现我的预期输出

【问题讨论】：

列值是多少？您认为它们何时是开始时间和停止时间？

标签： python pandas dataframe csv time-series

【解决方案1】：

IIUC，

让我们试试这个：

def f(df):
    a = (df['value'] != 1).cumsum().mask(df['value'] == 1)
    df['value group'] = a.bfill()

    df_final = df.groupby('value group').filter(lambda x: set(x['value']) == set([1,0]))\
           .groupby('value group')['date_time'].agg(['first','last'])\
           .rename(columns={'first':'start','last':'end'})\
           .reset_index()
    if df_final.shape[0] == 0:
        return
    df_final['id'] = df['sensorid']
    df_final['duration'] = df_final['end'].values - df_final['start']
    df_final['duration'] = df_final['duration'].dt.total_seconds().astype(int)
    return df_final

df_out = df.groupby('sensorid').apply(f).reset_index().drop(['level_1', 'value group', 'id'], axis=1)
df_out = df_out.sort_values('start')
df_out

输出：

   sensorid               start                 end  duration
0       202 2012-06-25 11:15:36 2012-06-25 11:15:38         2
1       303 2012-06-25 11:15:41 2012-06-25 11:15:44         3
2       404 2012-06-25 11:15:39 2012-06-25 11:15:41         2

注意：这可能需要更健壮的测试用例。但是，在 groupby 'sensorid' 调用的自定义函数中使用前面的逻辑。

【讨论】：

非常感谢斯科特。该解决方案通过按 sensorid 分组来实现我所需要的，这意味着它首先找到 202 的所有持续时间，然后再移动到 303，最后是 404。正如我的问题中所提到的，我怎样才能按发生顺序实现相同，即基于'date_time' 而不是 sensorid。如有必要，我可以与您分享更长的数据集。再次感谢。
正是我想要的。谢谢斯科特。

【解决方案2】：

对于单个传感器：

import pandas as pd
df = pd.read_csv('single_censor.csv')
df['date_time'] = pd.to_datetime(df['date_time'])

# Assume that your data format first value=0 ignore, start value=1 end value=0
selected_rows = df['value'] != df['value'].shift(1)
selected_rows[0] = False

df2 = df[selected_rows].copy()

df2['start'] = df2['date_time']
df2['end'] = df2['date_time'].shift(-1)
df2.drop(['date_time'], axis=1, inplace=True)

df3 = df2[df2['value'] == 1].copy()

df3['duration'] = df3['end'] - df3['start']
df3.drop('value', axis=1, inplace=True)

输出

    sensorid    start   end duration
1   202 2012-06-25 00:01:08 2012-06-25 00:01:10 00:00:02
5   202 2012-06-25 00:02:13 2012-06-25 00:02:16 00:00:03
10  202 2012-06-25 00:03:41 2012-06-25 00:03:44 00:00:03
15  202 2012-06-25 00:05:12 2012-06-25 00:05:14 00:00:02
19  202 2012-06-25 00:06:20 2012-06-25 00:06:22 00:00:02

多个传感器：

import pandas as pd
df = pd.read_csv('multiple_sensors.csv')
df['date_time'] = pd.to_datetime(df['date_time'])
df2 = df.sort_values(['sensorid', 'date_time'])

selected_rows = df2['value'] != df2['value'].shift(1)
selected_rows[0] = False

df3 = df2[selected_rows].copy()
df3['start'] = df3['date_time']
df3['end'] = df3['date_time'].shift(-1)
df3.drop(['date_time'], axis=1, inplace=True)

df4 = df3[df3['value'] == 1].copy()
df4['duration'] = df4['end'] - df4['start']
df4.drop('value', axis=1, inplace=True)
df4.sort_values('start')

输出

    sensorid               start                 end duration
7        202 2012-06-25 11:15:36 2012-06-25 11:15:38 00:00:02
17       404 2012-06-25 11:15:39 2012-06-25 11:15:41 00:00:02
24       303 2012-06-25 11:15:41 2012-06-25 11:15:44 00:00:03

去除重叠时间：

data = [
    (202, pd.to_datetime('2012-06-25 00:11:47'),
     pd.to_datetime('2012-06-25 00:11:49'), 2),
    (404, pd.to_datetime('2012-06-25 00:11:48'),
     pd.to_datetime('2012-06-25 00:11:50'), 2)
]
df = pd.DataFrame(data, columns=['sensor_id', 'start', 'end', 'duration'])

df['end_shift'] = df['end'].shift().fillna(pd.to_datetime('1971-01-01'))
df.loc[0, 'end_shift'] = pd.to_datetime('1971-01-01')
df[df['start'] >= df['end_shift']].drop('end_shift', axis=1)

输出

   sensor_id               start                 end  duration
0        202 2012-06-25 00:11:47 2012-06-25 00:11:49         2

小组持续时间：

data = [
(202, pd.to_datetime('2020-06-25 00:11:43'), pd.to_datetime('2020-06-25 00:11:45'),2), 
(202, pd.to_datetime('2020-06-25 00:11:47'), pd.to_datetime('2020-06-25 00:11:49'),2),
(404, pd.to_datetime('2020-06-25 00:11:51'), pd.to_datetime('2020-06-25 00:11:54'),3),
(404, pd.to_datetime('2020-06-25 00:11:55'), pd.to_datetime('2020-06-25 00:11:57'),2),
(202, pd.to_datetime('2020-06-25 00:11:58'), pd.to_datetime('2020-06-25 00:12:01'),3),
(202, pd.to_datetime('2020-06-25 00:12:18'), pd.to_datetime('2020-06-25 00:12:21'),3),
(101, pd.to_datetime('2020-06-25 00:12:21'), pd.to_datetime('2020-06-25 00:12:23'),2),
(101, pd.to_datetime('2020-06-25 00:12:32'), pd.to_datetime('2020-06-25 00:12:34'),2),
]
df=pd.DataFrame(data, columns=['sensor_id', 'start', 'end', 'duration'])

df['id'] = df['sensor_id'].shift(-1)
df['cumsum'] = df['duration'].cumsum()
df2 = df[df['id'] != df['sensor_id']].copy()
df2['duration2'] = df2['cumsum'] - df2['cumsum'].shift().fillna(0)
df2[['sensor_id', 'duration2']]

输出

   sensor_id  duration2
1        202        4.0
3        404        5.0
5        202        6.0
7        101        4.0

要求从一开始就不清楚。所有原始计算的持续时间都将被丢弃，并重新计算新的持续时间。如果要求明确就更好了。解决方案将被缩短。

data = [
(202, pd.to_datetime('2020-06-25 00:11:43'), pd.to_datetime('2020-06-25 00:11:45'),2), 
(202, pd.to_datetime('2020-06-25 00:11:47'), pd.to_datetime('2020-06-25 00:11:49'),2),
(404, pd.to_datetime('2020-06-25 00:11:51'), pd.to_datetime('2020-06-25 00:11:54'),3),
(404, pd.to_datetime('2020-06-25 00:11:55'), pd.to_datetime('2020-06-25 00:11:57'),2),
(202, pd.to_datetime('2020-06-25 00:11:58'), pd.to_datetime('2020-06-25 00:12:01'),3),
(202, pd.to_datetime('2020-06-25 00:12:18'), pd.to_datetime('2020-06-25 00:12:21'),3),
(101, pd.to_datetime('2020-06-25 00:12:21'), pd.to_datetime('2020-06-25 00:12:23'),2),
(101, pd.to_datetime('2020-06-25 00:12:32'), pd.to_datetime('2020-06-25 00:12:34'),2),
]
df=pd.DataFrame(data, columns=['sensor_id', 'start', 'end', 'duration'])

df['id1'] = df['sensor_id'].shift(-1)
df['id2'] = df['sensor_id'].shift(1)

df2 = df[df['id1'] != df['sensor_id']].copy().reset_index()
df2['start'] = df[df['id2'] != df['sensor_id']].reset_index()['start']

df2['duration'] = df2['end'] - df2['start']
df2.drop(['id1', 'id2'], axis=1, inplace=True) 
df2

输出

   index  sensor_id               start                 end duration
0      1        202 2020-06-25 00:11:43 2020-06-25 00:11:49 00:00:06
1      3        404 2020-06-25 00:11:51 2020-06-25 00:11:57 00:00:06
2      5        202 2020-06-25 00:11:58 2020-06-25 00:12:21 00:00:23
3      7        101 2020-06-25 00:12:21 2020-06-25 00:12:34 00:00:13

【讨论】：

不错的解决方案 Pramote，完美运行。正如我的帖子中提到的，如何根据发生的顺序（即“日期时间”而不是传感器 ID）获取每个传感器事件的持续时间。目前，该解决方案首先遍历每个 sensorid，然后再移动到下一个 sensorid。如有必要，我很乐意提供更多信息。再次感谢。
df4.sort_values('start')
作为我的问题的后续，我的输出有一些重叠如下：sensor_id,start,end,duration202,2012-06-25 00:11:47,2012-06-25 00:11:49,2404,2012-06-25 00:11:48,2012-06-25 00:11:50,2结束时间[2012-06-25 00:11:49 ] 第一行的开始时间应该小于第二行的开始时间[2012-06-25 00:11:48] 如何去除重叠部分？ b>
您要调整时间还是删除它们？如果要调整时间，哪一个：开始时间还是结束时间？
您的数据框必须是带有持续时间的输出，而不是原始数据框。只需删除列 end_shift，您将获得删除了重叠行的输出数据框。