如何从 Pandas 的时间序列中删除重复样本？

Question

目前我正在处理 Pandas 中的时间序列数据。该系列是几个标记的 3D 位置，所以我的 Dataframe 如下所示：

[A.x、A.y、A.z、B.x、B.y、B.z、C.x、C.y、C.z ...等]

现在系统有时会丢失一个标记，因此该位置在几帧内保持不变。我想将这些值设置为 NaN（以便稍后对它们进行插值），但我不知道如何做到这一点。所以：

 A.x  A.y  A.z      A.x  A.y  A.z
[0.1, 0.2, 0.2]    [0.1, 0.2, 0.2]
[0.1, 0.2, 0.2]    [NaN, NaN, NaN]
[0.1, 0.2, 0.2] -> [NaN, NaN, NaN]
[0.3, 0.2, 0.2]    [0.3, 0.2, 0.2] <- Kept because at least one position was different
[0.2, 0.2, 0.2]    [0.2, 0.2, 0.2]
[0.3, 0.2, 0.2]    [0.1, 0.2, 0.2] <- Kept as it was not the same as the immediately preceding frame

删除重复项不起作用，因为它不查找“重复”值，而是查找一般重复项。我认为同时查看 3 列（所以 1 分）的解决方案是最好的？

Answer 1

下面是简单版。

通用版：

import numpy as np
import pandas as pd

df = pd.DataFrame(
    [
        [0.1, 0.2, 0.2, 0.3, 0.2, 0.2], 
        [0.1, 0.2, 0.2, 0.3, 0.2, 0.2],
        [0.1, 0.2, 0.2, 0.2, 0.2, 0.2],
        [0.3, 0.2, 0.2, 0.3, 0.2, 0.2],
        [0.2, 0.2, 0.2, 0.1, 0.2, 0.2],
        [0.3, 0.2, 0.2, 0.1, 0.2, 0.2],
        [0.3, 0.2, 0.2, 0.1, 0.2, 0.2],
    ],
    columns="A.x A.y A.z B.x B.y B.z".split(),
)
#    A.x  A.y  A.z  B.x  B.y  B.z
# 0  0.1  0.2  0.2  0.3  0.2  0.2
# 1  0.1  0.2  0.2  0.3  0.2  0.2
# 2  0.1  0.2  0.2  0.2  0.2  0.2
# 3  0.3  0.2  0.2  0.3  0.2  0.2
# 4  0.2  0.2  0.2  0.1  0.2  0.2
# 5  0.3  0.2  0.2  0.1  0.2  0.2
# 6  0.3  0.2  0.2  0.1  0.2  0.2

# identify repeating data
diff = (df.values[:-1]  == df.values[1:])
# [[ True,  True,  True,  True,  True,  True],
#  [ True,  True,  True, False,  True,  True],
#  [False,  True,  True, False,  True,  True],
#  [False,  True,  True, False,  True,  True],
#  [False,  True,  True,  True,  True,  True],
#  [ True,  True,  True,  True,  True,  True]]

allfalse = np.full((1, diff.shape[1]), False)
# [[False, False, False, False, False, False]]

# add allfalse as first row
diff2 = np.concatenate((allfalse, diff), axis=0)

# grouped into 3s
grouped = diff2.reshape(diff2.shape[0], diff2.shape[1] // 3, 3)

# [[[False, False, False], [False, False, False]],
#  [[ True,  True,  True], [ True,  True,  True]],
#  [[ True,  True,  True], [False,  True,  True]],
#  [[False,  True,  True], [False,  True,  True]],
#  [[False,  True,  True], [False,  True,  True]],
#  [[False,  True,  True], [ True,  True,  True]],
#  [[ True,  True,  True], [ True,  True,  True]]]

# mask for triplets
mask = np.all(grouped, axis=2)
# [[False, False],
#  [ True,  True],
#  [ True, False],
#  [False, False],
#  [False, False],
#  [False,  True],
#  [ True,  True]]

grouped[~mask] = False
# [[[False, False, False], [False, False, False]],
#  [[ True,  True,  True], [ True,  True,  True]],
#  [[ True,  True,  True], [False, False, False]],
#  [[False, False, False], [False, False, False]],
#  [[False, False, False], [False, False, False]],
#  [[False, False, False], [ True,  True,  True]],
#  [[ True,  True,  True], [ True,  True,  True]]]

# finally reshape back into original shape
repeated = grouped.reshape(diff2.shape[0], diff2.shape[1])
# [[False, False, False, False, False, False],
#  [ True,  True,  True,  True,  True,  True],
#  [ True,  True,  True, False, False, False],
#  [False, False, False, False, False, False],
#  [False, False, False, False, False, False],
#  [False, False, False,  True,  True,  True],
#  [ True,  True,  True,  True,  True,  True]]

# set repeating values to NAN
df.values[repeated] = np.nan
#    A.x  A.y  A.z  B.x  B.y  B.z
# 0  0.1  0.2  0.2  0.3  0.2  0.2
# 1  NaN  NaN  NaN  NaN  NaN  NaN
# 2  NaN  NaN  NaN  0.2  0.2  0.2
# 3  0.3  0.2  0.2  0.3  0.2  0.2
# 4  0.2  0.2  0.2  0.1  0.2  0.2
# 5  0.3  0.2  0.2  NaN  NaN  NaN
# 6  NaN  NaN  NaN  NaN  NaN  NaN

简单（r）版本：

import numpy as np
import pandas as pd

df = pd.DataFrame(
    [
        [0.1, 0.2, 0.2],
        [0.1, 0.2, 0.2],
        [0.1, 0.2, 0.2],
        [0.3, 0.2, 0.2],
        [0.2, 0.2, 0.2],
        [0.3, 0.2, 0.2],
        [0.3, 0.2, 0.2],
    ],
    columns="A.x  A.y  A.z".split(),
)

#    A.x  A.y  A.z
# 0  0.1  0.2  0.2
# 1  0.1  0.2  0.2
# 2  0.1  0.2  0.2
# 3  0.3  0.2  0.2
# 4  0.2  0.2  0.2
# 5  0.3  0.2  0.2
# 6  0.3  0.2  0.2


# difference between consecutive values
diff = (df.values[:-1]  == df.values[1:])

# [[ True,  True,  True],
#  [ True,  True,  True],
#  [False,  True,  True],
#  [False,  True,  True],
#  [False,  True,  True],
#  [ True,  True,  True]]

# collapse rows into single value np.all(..., axis=1)
# make array len == number of rows in original DF
repeated = np.insert(np.all(diff, axis=1), 0, False)
# [False,  True,  True, False, False, False,  True]

# modify df in-place
df.values[repeated] = [np.nan, np.nan, np.nan]

#    A.x  A.y  A.z
# 0  0.1  0.2  0.2
# 1  NaN  NaN  NaN
# 2  NaN  NaN  NaN
# 3  0.3  0.2  0.2
# 4  0.2  0.2  0.2
# 5  0.3  0.2  0.2
# 6  NaN  NaN  NaN

我确信这可以做得更漂亮、更高效，但这是第 2 步 :)

我会考虑 B.x...C.x 部分...将发布更新。

享受吧！