生成可索引滑动窗口时间序列的优雅方式答案

【问题标题】：Elegant way to generate indexable sliding window timeseries生成可索引滑动窗口时间序列的优雅方式
【发布时间】：2019-05-16 10:33:02
【问题描述】：

要实现pytorch的DataSet类__get_item__()方法，需要支持索引，以便dataset[i]可以用来获取ith样本。

假设我有一个时间序列ser：

2017-12-29 14:44:00  69.90
2017-12-29 14:45:00  69.91
2017-12-29 14:46:00  69.87
2017-12-29 14:47:00  69.85
2017-12-29 14:48:00  69.86
2017-12-29 14:49:00  69.92
2017-12-29 14:50:00  69.90
2017-12-29 14:51:00  70.00
2017-12-29 14:52:00  69.97
2017-12-29 14:53:00  69.99
2017-12-29 14:54:00  69.99
2017-12-29 14:55:00  69.85

因为我需要索引到滚动窗口。我使用以下方法生成窗口长度3 时间序列：

l3_list = list()
def t(x):
  l3_list.append(x.copy())
ser.rolling(3).apply(t)

l3_list 变为：

[array([69.9 , 69.91, 69.87]),
 array([69.91, 69.87, 69.85]),
 array([69.87, 69.85, 69.86]),
 array([69.85, 69.86, 69.92]),
 array([69.86, 69.92, 69.9 ]),
 array([69.92, 69.9 , 70.  ]),
 array([69.9 , 70.  , 69.97]),
 array([70.  , 69.97, 69.99]),
 array([69.97, 69.99, 69.99]),
 array([69.99, 69.99, 69.85])]

这样我就可以在 l3_list 中建立索引。即l3_list[i]是ith滑动窗口。有没有更节省内存的方法来做到这一点？

【问题讨论】：

我对 pandas 的了解有限，但为什么不是 return ser[i:i+3].copy() in __getitem__？（为什么copy见this answer）

标签： python pandas pytorch

【解决方案1】：

这是另一个获得滑动窗口的技巧：

设置：

d = {pd.Timestamp('2017-12-29 14:44:00'): 69.9,
 pd.Timestamp('2017-12-29 14:45:00'): 69.91,
 pd.Timestamp('2017-12-29 14:46:00'): 69.87,
 pd.Timestamp('2017-12-29 14:47:00'): 69.85,
 pd.Timestamp('2017-12-29 14:48:00'): 69.86,
 pd.Timestamp('2017-12-29 14:49:00'): 69.92,
 pd.Timestamp('2017-12-29 14:50:00'): 69.9,
 pd.Timestamp('2017-12-29 14:51:00'): 70.0,
 pd.Timestamp('2017-12-29 14:52:00'): 69.97,
 pd.Timestamp('2017-12-29 14:53:00'): 69.99,
 pd.Timestamp('2017-12-29 14:54:00'): 69.99,
 pd.Timestamp('2017-12-29 14:55:00'): 69.85}

ser = pd.Series(d)

rolling 使用空列表，apply 使用 append：

lol = []
ser.rolling(3).apply((lambda x: lol.append(x.values) or 0), raw=False)
lol

输出：

[array([69.9 , 69.91, 69.87]),
 array([69.91, 69.87, 69.85]),
 array([69.87, 69.85, 69.86]),
 array([69.85, 69.86, 69.92]),
 array([69.86, 69.92, 69.9 ]),
 array([69.92, 69.9 , 70.  ]),
 array([69.9 , 70.  , 69.97]),
 array([70.  , 69.97, 69.99]),
 array([69.97, 69.99, 69.99]),
 array([69.99, 69.99, 69.85])]

【讨论】：

【解决方案2】：

您可以添加一列，有点像这里所做的：Pandas rolling window to return an array

from io import StringIO

data = """
2017-12-29 14:44:00  69.90
2017-12-29 14:45:00  69.91
2017-12-29 14:46:00  69.87
2017-12-29 14:47:00  69.85
2017-12-29 14:48:00  69.86
2017-12-29 14:49:00  69.92
2017-12-29 14:50:00  69.90
2017-12-29 14:51:00  70.00
2017-12-29 14:52:00  69.97
2017-12-29 14:53:00  69.99
2017-12-29 14:54:00  69.99
2017-12-29 14:55:00  69.85 
"""

df = pd.read_csv(StringIO(data), sep='\s+', header = None)

stride = np.lib.stride_tricks.as_strided  
arr = stride(df[2], (len(df), 3), (df[2].values.strides * 2))
df['array'] = pd.Series(arr.tolist(), index=df.index[:])

             0         1      2                         array
0   2017-12-29  14:44:00  69.90          [69.9, 69.91, 69.87]
1   2017-12-29  14:45:00  69.91         [69.91, 69.87, 69.85]
2   2017-12-29  14:46:00  69.87         [69.87, 69.85, 69.86]
3   2017-12-29  14:47:00  69.85         [69.85, 69.86, 69.92]
4   2017-12-29  14:48:00  69.86          [69.86, 69.92, 69.9]
5   2017-12-29  14:49:00  69.92           [69.92, 69.9, 70.0]
6   2017-12-29  14:50:00  69.90           [69.9, 70.0, 69.97]
7   2017-12-29  14:51:00  70.00          [70.0, 69.97, 69.99]
8   2017-12-29  14:52:00  69.97         [69.97, 69.99, 69.99]
9   2017-12-29  14:53:00  69.99         [69.99, 69.99, 69.85]
10  2017-12-29  14:54:00  69.99     [69.99, 69.85, 5.53e-322]
11  2017-12-29  14:55:00  69.85  [69.85, 5.53e-322, 5.6e-322]

【讨论】：