【问题标题】:Speeding up detection of overlapping time intervals加快重叠时间间隔的检测
【发布时间】:2021-05-08 15:49:41
【问题描述】:

我正在撰写关于电动汽车充电的论文,并且我正在尝试绘制一个时间序列图,以显示有多少汽车在一段时间内使用充电站。例如,我创建了以下数据:

base = datetime.datetime(2021,3,2)
start_time = np.array([base + datetime.timedelta(hours=12),
         base + datetime.timedelta(hours=12,minutes=3),
         base + datetime.timedelta(hours=12,minutes=20),
         base + datetime.timedelta(hours=12,minutes=50)])

end_time = np.array([start_time[0] + datetime.timedelta(minutes=25),
                   start_time[1] + datetime.timedelta(minutes=32),
                   start_time[2] + datetime.timedelta(minutes=25),
                     start_time[3] + datetime.timedelta(minutes=15)])

df = pd.DataFrame(columns=['start_time','end_time'])
df['start_time'] = start_time
df['end_time'] = end_time
df['is_obs'] = 1

我真正想知道的是,如果每 5 分钟有多少观察是活着,那么在它们的开始时间和结束时间之间。首先,我构建了间隔的时间步长:

# create range of datetime objects with 5 min intervals. 
indx = pd.date_range(start=min(df.start_time),end=max(df.end_time), freq='5T')
# convert to numpy array
indx = np.array(indx.to_pydatetime())
# add extra interval at the end so the full spectrum of an observation is within the interval
indx = np.append(indx, (indx[-1] + datetime.timedelta(minutes = 5)))

最后,我循环遍历每个时间间隔并检查每个观察的时间跨度是否在时间间隔内。为此,我使用包datetimerange

count_obs = np.zeros(len(indx)-1) 
for i in range(0,len(indx)-1): # iterate over time intervals
    time_range = DateTimeRange(indx[i],indx[i+1]) 
    for index, row in df.iterrows(): # iterate over all observations in dataframe
        x = DateTimeRange(row.start_time, row.end_time) 
        if time_range.is_intersection(x):
            count_obs[i] += 1

这给了我结果:

array([2., 2., 2., 3., 3., 3., 2., 1., 1., 1., 1., 1.])

但是,如果我尝试针对总时间跨度约为 2 年(因此超过 210.000 个 5 分钟的时间间隔)和超过 2000 次观察的完整数据集进行扩展,则需要很长时间。

你们中有人知道我可以如何加快这个过程吗?

提前致谢! :-)

【问题讨论】:

    标签: python pandas performance numpy datetime


    【解决方案1】:

    迭代时间间隔而不是输入数据可能更容易。对于您的玩具示例,可能看起来像

    t0, tmax = df['start_time'].min(), df['end_time'].max()
    t1 = t0 + pd.Timedelta(minutes=5)
    d = {'t0': [], 't1': [], 'nobs': []}
    
    while t0 < tmax:
        # lower boundary included, upper boundary excluded
        m = (df['end_time']>t0) & (df['start_time']<=t1)
        d['t0'].append(t0)
        d['t1'].append(t1)
        d['nobs'].append(m.sum())
        t0, t1 = t1, t1 + pd.Timedelta(minutes=5)
    
    df_out = pd.DataFrame(d)
    
    # df_out
    #                     t0                  t1  nobs
    # 0  2021-03-02 12:00:00 2021-03-02 12:05:00     2
    # 1  2021-03-02 12:05:00 2021-03-02 12:10:00     2
    # 2  2021-03-02 12:10:00 2021-03-02 12:15:00     2
    # 3  2021-03-02 12:15:00 2021-03-02 12:20:00     3
    # 4  2021-03-02 12:20:00 2021-03-02 12:25:00     3
    # 5  2021-03-02 12:25:00 2021-03-02 12:30:00     2
    # 6  2021-03-02 12:30:00 2021-03-02 12:35:00     2
    # 7  2021-03-02 12:35:00 2021-03-02 12:40:00     1
    # 8  2021-03-02 12:40:00 2021-03-02 12:45:00     1
    # 9  2021-03-02 12:45:00 2021-03-02 12:50:00     1
    # 10 2021-03-02 12:50:00 2021-03-02 12:55:00     1
    # 11 2021-03-02 12:55:00 2021-03-02 13:00:00     1
    # 12 2021-03-02 13:00:00 2021-03-02 13:05:00     1
    

    【讨论】:

      猜你喜欢
      • 1970-01-01
      • 1970-01-01
      • 1970-01-01
      • 2022-01-20
      • 2019-08-12
      • 1970-01-01
      • 1970-01-01
      • 1970-01-01
      • 2019-04-12
      相关资源
      最近更新 更多