从字典中删除重复的熊猫数据框答案

【问题标题】：Remove duplicate pandas data frames from dictionary从字典中删除重复的熊猫数据框
【发布时间】：2016-11-12 10:16:45
【问题描述】：

我有一个字典，其中包含具有相同列名的 pandas 数据框，我想删除具有相同值和行 ID 的重复数据框。

假设这是我的数据框字典：

>>> dd[0]
              Origin           Destination                Time
0           New York                Boston 2016-03-28 02:00:00
1           New York           Los Angeles 2016-03-28 04:00:00
2             Boston           Los Angeles 2016-03-28 06:00:00
>>> dd[1]
              Origin           Destination                Time
0           New York                Boston 2016-03-28 02:00:00
1           New York           Los Angeles 2016-03-28 04:00:00
2             Boston           Los Angeles 2016-03-28 06:00:00
>>> dd[2]
              Origin           Destination                Time
0           New York                Boston 2016-03-28 02:00:00
1           New York           Los Angeles 2016-03-28 04:00:00
2             Boston           Los Angeles 2016-03-28 06:00:00
>>> dd[3]
              Origin           Destination                Time
1           New York           Los Angeles 2016-03-28 04:00:00
2           Los Angeles             Boston 2016-03-28 06:00:00
3             Boston              New York 2016-03-28 08:00:00
>>> dd[4]
              Origin           Destination                Time
1           New York           Los Angeles 2016-03-28 04:00:00
2           Los Angeles             Boston 2016-03-28 06:00:00
3             Boston              New York 2016-03-28 08:00:00
>>> dd[5]
              Origin           Destination                Time
3             Boston              New York 2016-03-28 08:00:00
4           New York           Los Angeles 2016-03-28 12:00:00
>>> dd[6]
              Origin           Destination                Time
3             Boston              New York 2016-03-28 08:00:00
4           New York           Los Angeles 2016-03-28 12:00:00

我希望结果如下所示：

>>> dd[0]
              Origin           Destination                Time
0           New York                Boston 2016-03-28 02:00:00
1           New York           Los Angeles 2016-03-28 04:00:00
2             Boston           Los Angeles 2016-03-28 06:00:00
>>> dd[3]
              Origin           Destination                Time
1           New York           Los Angeles 2016-03-28 04:00:00
2           Los Angeles             Boston 2016-03-28 06:00:00
3             Boston              New York 2016-03-28 08:00:00
>>> dd[5]
              Origin           Destination                Time
3             Boston              New York 2016-03-28 08:00:00
4           New York           Los Angeles 2016-03-28 12:00:00

这是导致上述示例的代码：

# Load data as pandas data frame
data = pd.read_csv("website.txt", names = ["Time", "Origin", `"Destination"])`
data["Time"] = pd.to_datetime(data["Time"], infer_datetime_format=True)
# Reverse data frame by index to loop backwards
data = data.reindex(index=df.index[::-1]) 
dd = {} 
for i, e in reverse.iterrows(): 
    dd[i] = data[ (data['Time'] > e['Time']-pd.Timedelta('4 hours')) & (data['Time'] < e['Time'] + pd.Timedelta('4 hours'))]

原文：

{"Time": "2016-03-28T02:00:00Z", "Origin": "New York", "Destination": "Boston"}
{"Time": "2016-03-28T02:00:00Z", "Origin": "New York", "Destination": "Boston"}
{"Time": "2016-03-28T02:00:00Z", "Origin": "New York", "Destination": "Boston"}
{"Time": "2016-03-28T04:00:00Z", "Origin": "New York", "Destination": "Los Angeles"}
{"Time": "2016-03-28T04:00:00Z", "Origin": "New York", "Destination": "Los Angeles"}
{"Time": "2016-03-28T04:00:00Z", "Origin": "New York", "Destination": "Los Angeles"}
{"Time": "2016-03-28T06:00:00Z", "Origin": "Boston", "Destination": "Los Angeles"}
{"Time": "2016-03-28T06:00:00Z", "Origin": "Boston", "Destination": "Los Angeles"}
{"Time": "2016-03-28T06:00:00Z", "Origin": "Boston", "Destination": "Los Angeles"}
{"Time": "2016-03-28T08:00:00Z", "Origin": "Boston", "Destination": "New York"}
{"Time": "2016-03-28T08:00:00Z", "Origin": "Boston", "Destination": "New York"}
{"Time": "2016-03-28T12:00:00Z", "Origin": "New York", "Destination": "Los Angeles"}
{"Time": "2016-03-28T12:00:00Z", "Origin": "New York", "Destination": "Los Angeles"}

【问题讨论】：

Removing Duplicates From Dictionary的可能重复
约定使用'pd'作为pandas的缩写“import pandas as pd”约定'df'是dataframe的缩写。如果您的代码难以阅读 vs pandas/python 约定。
嘿梅林，我早就注意到你的 cmets 下面了。请给我时间进行调整。
@Merlin 嘿，Merlin，我想知道您是否可以提出一种方法来将这个问题与我提出的另一个问题联系起来，并且您已经回答了。提前非常感谢！
@Provisional.Modulation 桥接是什么意思？

标签： python pandas dictionary duplicates

【解决方案1】：

一个班轮

{k: v.unstack() for k, v in pd.DataFrame({k: v.stack() for k, v in dd.iteritems()}).T.drop_duplicates().iterrows()}

解释版

# iterate through key, value pairs of dictionary,
# stacking each dataframe into a series so that we
# can pass the resulting dataframe into the pd.DataFrame constructor.
df1 = pd.DataFrame({k: v.stack() for k, v in dd.iteritems()})
# Each column is now one key, value pair from the original dictionary
# Transpose and drop duplicates
df2 = df1.T.drop_duplicates()
# reverse the original stacking and convert back to dictionary
# we could have used df2.T.iteritems() but df2.iterrows() took
# one fewer operations and fewer characters to type.
dd_ = {k: v.unstack() for k, v in df2.iterrows()}

for k, v in dd_.iteritems():
    print 'key {}:'.format(k)
    print v
    print '-' * 10

key 0:
   a  b
0  1  2
1  3  4
----------
key 2:
   a  b
0  2  3
1  4  5
----------

设置以获得与我相同的结果（复制并粘贴此）

from StringIO import StringIO
import pandas as pd

text0 = """              Origin           Destination                 Time
0           New York                Boston  2016-03-28 02:00:00
1           New York           Los Angeles  2016-03-28 04:00:00
2             Boston           Los Angeles  2016-03-28 06:00:00"""


text1 = """              Origin           Destination                 Time
0           New York                Boston  2016-03-28 02:00:00
1           New York           Los Angeles  2016-03-28 04:00:00
2             Boston           Los Angeles  2016-03-28 06:00:00"""

text2 = """              Origin           Destination                 Time
0           New York                Boston  2016-03-28 02:00:00
1           New York           Los Angeles  2016-03-28 04:00:00
2           Los Angeles             Boston  2016-03-28 06:00:00"""

dd = {}

dd[0] = pd.read_csv(StringIO(text0), sep='\s{2,}', index_col=0, engine='python')
dd[0].Time = pd.to_datetime(dd[0].Time)

dd[1] = pd.read_csv(StringIO(text1), sep='\s{2,}', index_col=0, engine='python')
dd[1].Time = pd.to_datetime(dd[1].Time)

dd[2] = pd.read_csv(StringIO(text2), sep='\s{2,}', index_col=0, engine='python')
dd[2].Time = pd.to_datetime(dd[2].Time)

# Then run solutions above:

df1 = pd.DataFrame({k: v.stack() for k, v in dd.iteritems()})
df2 = df1.T.drop_duplicates()
dd_ = {k: v.unstack() for k, v in df2.iterrows()}

for k, v in dd_.iteritems():
    print 'key {}:'.format(k)
    print v
    print '-' * 10

你应该得到这个：

key 0:
     Origin  Destination                 Time
0  New York       Boston  2016-03-28 02:00:00
1  New York  Los Angeles  2016-03-28 04:00:00
2    Boston  Los Angeles  2016-03-28 06:00:00
----------
key 2:
        Origin  Destination                 Time
0     New York       Boston  2016-03-28 02:00:00
1     New York  Los Angeles  2016-03-28 04:00:00
2  Los Angeles       Boston  2016-03-28 06:00:00
----------

版本

import sys
import pandas as pd
print sys.version
print pd.__version__

2.7.11 |Anaconda custom (x86_64)| (default, Dec  6 2015, 18:57:58) 
[GCC 4.2.1 (Apple Inc. build 5577)]
0.18.1

【讨论】：

@piRSquared：字典没有键排序，这将返回可重现的结果...正在对键进行排序，然后测试重复以获得可重现的结果。 OP 看起来想要 dup 的第一个键。
@Provisional.Modulation “多种类型的重复数据帧”是什么意思？这使用pd.DataFrame.drop_duplicates() 方法，该方法包含keep 参数。 keep 可以设置为first，其中保留第一个看到的内容，并丢弃所有其他内容。或 last 保留最后一次看到的内容，并删除所有其他内容。或者，False，如果它被复制，它就会被丢弃。
OP 将 df 用于 dicts 是错误的形式。非pythonic
我把 'df' 改成了 'dd 有问题，令人困惑！
@Provisional.Modulation 您引用了我无权访问的'website.txt'。请包含原始文本。