在对数据帧进行上采样时拆分持续时间答案

【问题标题】：Split up duration while upsampling dataframe在对数据帧进行上采样时拆分持续时间
【发布时间】：2022-01-16 00:54:50
【问题描述】：

如何在对数据帧进行上采样时拆分持续时间，如下例所示。我可以用例如替换for循环吗？ group_by 函数？

我想用 pandas 像这样转换数据：

  activity name         time started           time ended
0       Bedtime  2021-10-25 00:00:00  2021-10-25 08:25:42
1        videos  2021-10-25 08:25:42  2021-10-25 08:51:54
2       Commute  2021-10-25 08:51:54  2021-10-25 09:29:34

进入这个：

time started        Bedtime         videos           Commute                   
2021-10-25 00:00:00 0 days 01:00:00 0 days 00:00:00  0 days
2021-10-25 01:00:00 0 days 01:00:00 0 days 00:00:00  0 days
2021-10-25 02:00:00 0 days 01:00:00 0 days 00:00:00  0 days
2021-10-25 03:00:00 0 days 01:00:00 0 days 00:00:00  0 days
2021-10-25 04:00:00 0 days 01:00:00 0 days 00:00:00  0 days
2021-10-25 05:00:00 0 days 01:00:00 0 days 00:00:00  0 days
2021-10-25 06:00:00 0 days 01:00:00 0 days 00:00:00  0 days
2021-10-25 07:00:00 0 days 01:00:00 0 days 00:00:00  0 days
2021-10-25 08:00:00 0 days 00:25:42 0 days 00:26:12  0 days 00:08:06
...

我已经走到这一步了：

import pandas as pd
df=pd.DataFrame({'activity name':['Bedtime','videos','Commute'],'time started':["2021-10-25 00:00:00","2021-10-25 08:25:42","2021-10-25 08:51:54"],'time ended':["2021-10-25 08:25:42","2021-10-25 08:51:54","2021-10-25 09:29:34"]})
# converting strings to datetime
df['time ended']=pd.to_datetime(df['time ended'])
df['time started']=pd.to_datetime(df['time started'])

# calclating the duration
df['duration']=df['time ended']-df['time started']

# changeing index
df.index=df['time started']
df=df.drop(columns=['time started','time ended'])

for a in df['activity name'].unique():
    df[a]=(df['activity name']==a)*df['duration']

df=df.drop(columns=['activity name','duration'])
df.resample('H').first()

time started                                               
2021-10-25 00:00:00 0 days 08:25:42 0 days 00:00:00  0 days
2021-10-25 01:00:00             NaT             NaT     NaT
2021-10-25 02:00:00             NaT             NaT     NaT
2021-10-25 03:00:00             NaT             NaT     NaT
2021-10-25 04:00:00             NaT             NaT     NaT
2021-10-25 05:00:00             NaT             NaT     NaT
2021-10-25 06:00:00             NaT             NaT     NaT
2021-10-25 07:00:00             NaT             NaT     NaT
2021-10-25 08:00:00 0 days 00:00:00 0 days 00:26:12  0 days

【问题讨论】：

由于通勤发生在小时的尾声，在您想要的输出 df 的最后通勤行中不应该有0 days 00:08:06 的时间增量吗？
@DerekO 是的，你是对的

标签： python pandas dataframe datetime

【解决方案1】：

试试这个：

import pandas as pd
from io import StringIO

txtfile = StringIO(
    """  activity name         time started           time ended
0       Bedtime  2021-10-25 00:00:00  2021-10-25 08:25:42
1        videos  2021-10-25 08:25:42  2021-10-25 08:51:54
2       Commute  2021-10-25 08:51:54  2021-10-25 09:29:34"""
)

df = pd.read_csv(txtfile, sep="\s\s+", engine="python")

df[["time started", "time ended"]] = df[["time started", "time ended"]].apply(
    pd.to_datetime
)
df_e = df.assign(
    date=[
        pd.date_range(s, e, freq="s")
        for s, e in zip(df["time started"], df["time ended"])
    ]
).explode("date")

df_out = (
    df_e.groupby(["activity name", pd.Grouper(key="date", freq="H")])["activity name"]
    .count()
    .unstack(0)
    .apply(pd.to_timedelta, unit="s")
)

print(df_out)

输出：

activity name               Bedtime         Commute          videos
date                                                               
2021-10-25 00:00:00 0 days 01:00:00             NaT             NaT
2021-10-25 01:00:00 0 days 01:00:00             NaT             NaT
2021-10-25 02:00:00 0 days 01:00:00             NaT             NaT
2021-10-25 03:00:00 0 days 01:00:00             NaT             NaT
2021-10-25 04:00:00 0 days 01:00:00             NaT             NaT
2021-10-25 05:00:00 0 days 01:00:00             NaT             NaT
2021-10-25 06:00:00 0 days 01:00:00             NaT             NaT
2021-10-25 07:00:00 0 days 01:00:00             NaT             NaT
2021-10-25 08:00:00 0 days 00:25:43 0 days 00:08:06 0 days 00:26:13
2021-10-25 09:00:00             NaT 0 days 00:29:35             NaT

地址@DerekO 评论：

import pandas as pd
from io import StringIO

txtfile = StringIO(
    """  activity name         time started           time ended
0       Bedtime  2021-10-25 00:00:00  2021-10-25 08:25:42
1        videos  2021-10-25 08:25:42  2021-10-25 08:51:54
2       Commute  2021-10-25 08:51:54  2021-10-25 09:29:34
3       Bedtime  2021-10-25 11:00:00  2021-10-25 13:04:31"""
)

df = pd.read_csv(txtfile, sep="\s\s+", engine="python")

df[["time started", "time ended"]] = df[["time started", "time ended"]].apply(
    pd.to_datetime
)
df_e = df.assign(
    date=[
        pd.date_range(s, e, freq="s")
        for s, e in zip(df["time started"], df["time ended"])
    ]
).explode("date")

df_out = (
    df_e.groupby(["activity name", pd.Grouper(key="date", freq="H")])["activity name"]
    .count()
    .unstack(0)
    .apply(pd.to_timedelta, unit="s")
    .sort_index()
)

print(df_out)

输出：

activity name               Bedtime         Commute          videos
date                                                               
2021-10-25 00:00:00 0 days 01:00:00             NaT             NaT
2021-10-25 01:00:00 0 days 01:00:00             NaT             NaT
2021-10-25 02:00:00 0 days 01:00:00             NaT             NaT
2021-10-25 03:00:00 0 days 01:00:00             NaT             NaT
2021-10-25 04:00:00 0 days 01:00:00             NaT             NaT
2021-10-25 05:00:00 0 days 01:00:00             NaT             NaT
2021-10-25 06:00:00 0 days 01:00:00             NaT             NaT
2021-10-25 07:00:00 0 days 01:00:00             NaT             NaT
2021-10-25 08:00:00 0 days 00:25:43 0 days 00:08:06 0 days 00:26:13
2021-10-25 09:00:00             NaT 0 days 00:29:35             NaT
2021-10-25 11:00:00 0 days 01:00:00             NaT             NaT
2021-10-25 12:00:00 0 days 01:00:00             NaT             NaT
2021-10-25 13:00:00 0 days 00:04:32             NaT             NaT

【讨论】：

您的解决方案肯定比我的更优雅，但是如果重复活动名称，我不确定它是否有效。我知道原始问题中没有假设这一点，但如果添加更多数据，我觉得这是一个合理的假设......如果你分组activity name 和其他一些工程功能，应该有办法解决这个问题？
非常好！是的，我认为这就是这样做的方法。如果您重复其中一个活动名称并获得预期结果，那么我认为您的答案非常无懈可击
@DerekO 不需要它。它按原样工作。

【解决方案2】：

虽然我同意最好使用groupby 和resample，但我无法使这样的解决方案发挥作用。您可以通过为原始 DataFrame 的每一行创建一个新的 DataFrame 并将它们连接在一起来强制解决问题。

它的工作方式是我们使用pd.date_range 在开始和结束时间的地板之间创建一个DatetimeIndex，并将开始和结束时间也插入到 DatetimeIndex 中。那么这个 DatetimeIndex 中所有日期时间之间的差异就是你的新 DataFrame 的值。

为了使我的解决方案尽可能稳健，我在您的原始 DataFrame 中添加了两个额外的行，其中包含重复的类别，并测试了开始时间正好是整点而不是整点的情况。

import pandas as pd
from pandas._libs.tslibs.timedeltas import Timedelta

df=pd.DataFrame({
    'activity name':['Bedtime','videos','Commute','Work','Commute'],
    'time started':["2021-10-25 00:00:00","2021-10-25 08:25:42","2021-10-25 08:51:54","2021-10-25 09:29:34","2021-10-25 17:00:00"],
    'time ended':["2021-10-25 08:25:42","2021-10-25 08:51:54","2021-10-25 09:29:34","2021-10-25 17:00:00","2021-10-25 18:01:00"]})

# converting strings to datetime
df['time ended']=pd.to_datetime(df['time ended'])
df['time started']=pd.to_datetime(df['time started'])

## column names with spaces can't be accessed by name when using iterruples to iterate through the df
df.columns = [col.replace(" ","_") for col in df.columns]

开始df：

>>> df
  activity_name        time_started          time_ended
0       Bedtime 2021-10-25 00:00:00 2021-10-25 08:25:42
1        videos 2021-10-25 08:25:42 2021-10-25 08:51:54
2       Commute 2021-10-25 08:51:54 2021-10-25 09:29:34
3          Work 2021-10-25 09:29:34 2021-10-25 17:00:00
4       Commute 2021-10-25 17:00:00 2021-10-25 18:01:00

## we use the start and end times to determine what daterange we create
start_time = df['time_started'].min().floor('h')
end_time = df['time_started'].max().ceil('h')

## setup an empty DataFrame to hold the final result
new_columns = list(df.activity_name.unique())
df_new = pd.DataFrame(columns=new_columns)

for row in df.itertuples(index=True):
    new_row = {}
    daterange_start = row.time_started.floor('1h')
    daterange_end = row.time_ended.floor('1h')
    datetimes_index = pd.date_range(daterange_start, daterange_end, freq='1h')

    all_datetimes = datetimes_index.union([row.time_started, row.time_ended])

    ## take the difference and shift by -1 to drop the first NaT
    new_row[row.activity_name] = all_datetimes.to_series().diff().shift(-1)
    
    ## if the first row starts in the middle of an hour, we don't want the difference between the beginning of the hour and the time in that row
    if (row.Index == 0) & (row.time_started > daterange_start):
        df_new = df_new.append(pd.DataFrame(new_row))[1:]
    else:
        df_new = df_new.append(pd.DataFrame(new_row))

df_new.index.name = 'time_started'
df_new.reset_index(inplace=True)

结果：

>>> df_new
          time_started         Bedtime          videos         Commute            Work
0  2021-10-25 00:00:00 0 days 01:00:00             NaT             NaT             NaT
1  2021-10-25 01:00:00 0 days 01:00:00             NaT             NaT             NaT
2  2021-10-25 02:00:00 0 days 01:00:00             NaT             NaT             NaT
3  2021-10-25 03:00:00 0 days 01:00:00             NaT             NaT             NaT
4  2021-10-25 04:00:00 0 days 01:00:00             NaT             NaT             NaT
5  2021-10-25 05:00:00 0 days 01:00:00             NaT             NaT             NaT
6  2021-10-25 06:00:00 0 days 01:00:00             NaT             NaT             NaT
7  2021-10-25 07:00:00 0 days 01:00:00             NaT             NaT             NaT
8  2021-10-25 08:00:00 0 days 00:25:42             NaT             NaT             NaT
9  2021-10-25 08:25:42             NaT             NaT             NaT             NaT
10 2021-10-25 08:00:00             NaT 0 days 00:25:42             NaT             NaT
11 2021-10-25 08:25:42             NaT 0 days 00:26:12             NaT             NaT
12 2021-10-25 08:51:54             NaT             NaT             NaT             NaT
13 2021-10-25 08:00:00             NaT             NaT 0 days 00:51:54             NaT
14 2021-10-25 08:51:54             NaT             NaT 0 days 00:08:06             NaT
15 2021-10-25 09:00:00             NaT             NaT 0 days 00:29:34             NaT
16 2021-10-25 09:29:34             NaT             NaT             NaT             NaT
17 2021-10-25 09:00:00             NaT             NaT             NaT 0 days 00:29:34
18 2021-10-25 09:29:34             NaT             NaT             NaT 0 days 00:30:26
19 2021-10-25 10:00:00             NaT             NaT             NaT 0 days 01:00:00
20 2021-10-25 11:00:00             NaT             NaT             NaT 0 days 01:00:00
21 2021-10-25 12:00:00             NaT             NaT             NaT 0 days 01:00:00
22 2021-10-25 13:00:00             NaT             NaT             NaT 0 days 01:00:00
23 2021-10-25 14:00:00             NaT             NaT             NaT 0 days 01:00:00
24 2021-10-25 15:00:00             NaT             NaT             NaT 0 days 01:00:00
25 2021-10-25 16:00:00             NaT             NaT             NaT 0 days 01:00:00
26 2021-10-25 17:00:00             NaT             NaT             NaT             NaT
27 2021-10-25 17:00:00             NaT             NaT 0 days 01:00:00             NaT
28 2021-10-25 18:00:00             NaT             NaT 0 days 00:01:00             NaT
29 2021-10-25 18:01:00             NaT             NaT             NaT             NaT

对于每个活动，我们创建了一个新的 DataFrame，用于获取带有all_datetimes.to_series().diff().shift(-1) 的时间差异，这意味着活动中的每个更改之间存在NaT。这些没有用，因此我们将删除所有活动为NaT 的行。

然后我们在time_started 列中删除重复的时间戳并保留这些重复的第一个值，并在time_started 列中取所有时间戳的下限：

df_new = df_new.dropna(subset=new_columns, how='all').drop_duplicates(subset=['time_started'], keep='first')
df_new['time_started'] = df_new['time_started'].apply(lambda x: x.floor('1h'))

结果：

>>> df_new
          time_started         Bedtime          videos         Commute            Work
0  2021-10-25 00:00:00 0 days 01:00:00             NaT             NaT             NaT
1  2021-10-25 01:00:00 0 days 01:00:00             NaT             NaT             NaT
2  2021-10-25 02:00:00 0 days 01:00:00             NaT             NaT             NaT
3  2021-10-25 03:00:00 0 days 01:00:00             NaT             NaT             NaT
4  2021-10-25 04:00:00 0 days 01:00:00             NaT             NaT             NaT
5  2021-10-25 05:00:00 0 days 01:00:00             NaT             NaT             NaT
6  2021-10-25 06:00:00 0 days 01:00:00             NaT             NaT             NaT
7  2021-10-25 07:00:00 0 days 01:00:00             NaT             NaT             NaT
8  2021-10-25 08:00:00 0 days 00:25:42             NaT             NaT             NaT
11 2021-10-25 08:00:00             NaT 0 days 00:26:12             NaT             NaT
14 2021-10-25 08:00:00             NaT             NaT 0 days 00:08:06             NaT
15 2021-10-25 09:00:00             NaT             NaT 0 days 00:29:34             NaT
18 2021-10-25 09:00:00             NaT             NaT             NaT 0 days 00:30:26
19 2021-10-25 10:00:00             NaT             NaT             NaT 0 days 01:00:00
20 2021-10-25 11:00:00             NaT             NaT             NaT 0 days 01:00:00
21 2021-10-25 12:00:00             NaT             NaT             NaT 0 days 01:00:00
22 2021-10-25 13:00:00             NaT             NaT             NaT 0 days 01:00:00
23 2021-10-25 14:00:00             NaT             NaT             NaT 0 days 01:00:00
24 2021-10-25 15:00:00             NaT             NaT             NaT 0 days 01:00:00
25 2021-10-25 16:00:00             NaT             NaT             NaT 0 days 01:00:00
27 2021-10-25 17:00:00             NaT             NaT 0 days 01:00:00             NaT
28 2021-10-25 18:00:00             NaT             NaT 0 days 00:01:00             NaT

现在我们用pd.Timedelta("0s") 填充所有NaT，然后我们可以对time_started 列中的值进行分组并将它们相加：

df_new = df_new.fillna(pd.Timedelta(0)).groupby("time_started").sum().reset_index()

最终结果：

>>> df_new
          time_started         Bedtime          videos         Commute            Work
0  2021-10-25 00:00:00 0 days 01:00:00 0 days 00:00:00 0 days 00:00:00 0 days 00:00:00
1  2021-10-25 01:00:00 0 days 01:00:00 0 days 00:00:00 0 days 00:00:00 0 days 00:00:00
2  2021-10-25 02:00:00 0 days 01:00:00 0 days 00:00:00 0 days 00:00:00 0 days 00:00:00
3  2021-10-25 03:00:00 0 days 01:00:00 0 days 00:00:00 0 days 00:00:00 0 days 00:00:00
4  2021-10-25 04:00:00 0 days 01:00:00 0 days 00:00:00 0 days 00:00:00 0 days 00:00:00
5  2021-10-25 05:00:00 0 days 01:00:00 0 days 00:00:00 0 days 00:00:00 0 days 00:00:00
6  2021-10-25 06:00:00 0 days 01:00:00 0 days 00:00:00 0 days 00:00:00 0 days 00:00:00
7  2021-10-25 07:00:00 0 days 01:00:00 0 days 00:00:00 0 days 00:00:00 0 days 00:00:00
8  2021-10-25 08:00:00 0 days 00:25:42 0 days 00:26:12 0 days 00:08:06 0 days 00:00:00
9  2021-10-25 09:00:00 0 days 00:00:00 0 days 00:00:00 0 days 00:29:34 0 days 00:30:26
10 2021-10-25 10:00:00 0 days 00:00:00 0 days 00:00:00 0 days 00:00:00 0 days 01:00:00
11 2021-10-25 11:00:00 0 days 00:00:00 0 days 00:00:00 0 days 00:00:00 0 days 01:00:00
12 2021-10-25 12:00:00 0 days 00:00:00 0 days 00:00:00 0 days 00:00:00 0 days 01:00:00
13 2021-10-25 13:00:00 0 days 00:00:00 0 days 00:00:00 0 days 00:00:00 0 days 01:00:00
14 2021-10-25 14:00:00 0 days 00:00:00 0 days 00:00:00 0 days 00:00:00 0 days 01:00:00
15 2021-10-25 15:00:00 0 days 00:00:00 0 days 00:00:00 0 days 00:00:00 0 days 01:00:00
16 2021-10-25 16:00:00 0 days 00:00:00 0 days 00:00:00 0 days 00:00:00 0 days 01:00:00
17 2021-10-25 17:00:00 0 days 00:00:00 0 days 00:00:00 0 days 01:00:00 0 days 00:00:00
18 2021-10-25 18:00:00 0 days 00:00:00 0 days 00:00:00 0 days 00:01:00 0 days 00:00:00

【讨论】：