熊猫有选择地丢弃连续重复答案

【问题标题】：pandas drop consecutive duplicates selectively熊猫有选择地丢弃连续重复
【发布时间】：2020-05-09 11:15:15
【问题描述】：

我一直在查看有关如何在 pandas 数据框中选择性地删除连续重复项的所有问题/答案，但仍然无法弄清楚以下情况：

import pandas as pd
import numpy as np

def random_dates(start, end, n, freq, seed=None):
    if seed is not None:
        np.random.seed(seed)

    dr = pd.date_range(start, end, freq=freq)
    return pd.to_datetime(np.sort(np.random.choice(dr, n, replace=False)))

date = random_dates('2018-01-01', '2018-01-12', 20, 'H', seed=[3, 1415])

data = {'Timestamp': date, 
        'Message': ['Message received.','Sending...', 'Sending...', 'Sending...', 'Work in progress...', 'Work in progress...', 
                    'Message received.','Sending...', 'Sending...','Work in progress...',
                    'Message received.','Sending...', 'Sending...', 'Sending...','Work in progress...', 'Work in progress...', 'Work in progress...',
                    'Message received.','Sending...', 'Sending...']}

df = pd.DataFrame(data, columns = ['Timestamp', 'Message'])

我有以下数据框：

             Timestamp              Message
0  2018-01-02 03:00:00    Message received.
1  2018-01-02 11:00:00           Sending...
2  2018-01-03 04:00:00           Sending...
3  2018-01-04 11:00:00           Sending...
4  2018-01-04 16:00:00  Work in progress...
5  2018-01-04 17:00:00  Work in progress...
6  2018-01-05 05:00:00    Message received.
7  2018-01-05 11:00:00           Sending...
8  2018-01-05 17:00:00           Sending...
9  2018-01-06 02:00:00  Work in progress...
10 2018-01-06 14:00:00    Message received.
11 2018-01-07 07:00:00           Sending...
12 2018-01-07 20:00:00           Sending...
13 2018-01-08 01:00:00           Sending...
14 2018-01-08 02:00:00  Work in progress...
15 2018-01-08 15:00:00  Work in progress...
16 2018-01-09 00:00:00  Work in progress...
17 2018-01-10 03:00:00    Message received.
18 2018-01-10 09:00:00           Sending...
19 2018-01-10 14:00:00           Sending...

我只想在 'Message' 为 'Work in progress...' 时删除 df['Message'] 列中的连续重复项并保留第一个实例（此处例如需要删除索引 5、15 和 16 )，理想情况下我想得到：

             Timestamp              Message
0  2018-01-02 03:00:00    Message received.
1  2018-01-02 11:00:00           Sending...
2  2018-01-03 04:00:00           Sending...
3  2018-01-04 11:00:00           Sending...
4  2018-01-04 16:00:00  Work in progress...
6  2018-01-05 05:00:00    Message received.
7  2018-01-05 11:00:00           Sending...
8  2018-01-05 17:00:00           Sending...
9  2018-01-06 02:00:00  Work in progress...
10 2018-01-06 14:00:00    Message received.
11 2018-01-07 07:00:00           Sending...
12 2018-01-07 20:00:00           Sending...
13 2018-01-08 01:00:00           Sending...
14 2018-01-08 02:00:00  Work in progress...
17 2018-01-10 03:00:00    Message received.
18 2018-01-10 09:00:00           Sending...
19 2018-01-10 14:00:00           Sending...

我尝试过类似帖子中提供的解决方案，例如：

df['Message'].loc[df['Message'].shift(-1) != df['Message']]

我还计算了消息的长度：

df['length'] = df['Message'].apply(lambda x: len(x))

并写了一个有条件的drop：

df.loc[(df['length'] ==17) | (df['length'] ==10) | ~df['Message'].duplicated(keep='first')]

看起来更好，但索引 14、15 和 16 仍然被完全删除，因此表现不佳，请参阅：

             Timestamp              Message  length
0  2018-01-02 03:00:00    Message received.      17
1  2018-01-02 11:00:00           Sending...      10
2  2018-01-03 04:00:00           Sending...      10
3  2018-01-04 11:00:00           Sending...      10
4  2018-01-04 16:00:00  Work in progress...      19
6  2018-01-05 05:00:00    Message received.      17
7  2018-01-05 11:00:00           Sending...      10
8  2018-01-05 17:00:00           Sending...      10
10 2018-01-06 14:00:00    Message received.      17
11 2018-01-07 07:00:00           Sending...      10
12 2018-01-07 20:00:00           Sending...      10
13 2018-01-08 01:00:00           Sending...      10
17 2018-01-10 03:00:00    Message received.      17
18 2018-01-10 09:00:00           Sending...      10
19 2018-01-10 14:00:00           Sending...      10

感谢您的时间和帮助！

【问题讨论】：

标签： python pandas duplicates data-manipulation

【解决方案1】：

首先过滤第一个连续值，比较 Series.shift，链掩码过滤所有没有 Work in progress... 值的行：

df = df[(df['Message'].shift() != df['Message']) | (df['Message'] != 'Work in progress...')]
print (df)
             Timestamp              Message
0  2018-01-02 03:00:00    Message received.
1  2018-01-02 11:00:00           Sending...
2  2018-01-03 04:00:00           Sending...
3  2018-01-04 11:00:00           Sending...
4  2018-01-04 16:00:00  Work in progress...
6  2018-01-05 05:00:00    Message received.
7  2018-01-05 11:00:00           Sending...
8  2018-01-05 17:00:00           Sending...
9  2018-01-06 02:00:00  Work in progress...
10 2018-01-06 14:00:00    Message received.
11 2018-01-07 07:00:00           Sending...
12 2018-01-07 20:00:00           Sending...
13 2018-01-08 01:00:00           Sending...
14 2018-01-08 02:00:00  Work in progress...
17 2018-01-10 03:00:00    Message received.
18 2018-01-10 09:00:00           Sending...
19 2018-01-10 14:00:00           Sending...

【讨论】：

我超级接近，没注意我可以在 shift 子句中添加第二个参数！精彩的。系统告诉我要再等 6 分钟才能接受，这是我这边接受的，你好。

【解决方案2】：

您可以先获取所有带有“Work in Progress”的消息，并将它们与上一个元素进行比较，然后进行过滤：

condition = (df['Message'] == 'Work in progress...') & (df['Message']==df['Message'].shift(1))

df[~condition]

     Timestamp           Message
0   2018-01-02 03:00:00 Message received.
1   2018-01-02 11:00:00 Sending...
2   2018-01-03 04:00:00 Sending...
3   2018-01-04 11:00:00 Sending...
4   2018-01-04 16:00:00 Work in progress...
6   2018-01-05 05:00:00 Message received.
7   2018-01-05 11:00:00 Sending...
8   2018-01-05 17:00:00 Sending...
9   2018-01-06 02:00:00 Work in progress...
10  2018-01-06 14:00:00 Message received.
11  2018-01-07 07:00:00 Sending...
12  2018-01-07 20:00:00 Sending...
13  2018-01-08 01:00:00 Sending...
14  2018-01-08 02:00:00 Work in progress...
17  2018-01-10 03:00:00 Message received.
18  2018-01-10 09:00:00 Sending...
19  2018-01-10 14:00:00 Sending...

【讨论】：