创建具有分组键列的两个熊猫数据框的“差异”结果集

Question

我会尽量保持简短和重点（使用简化的数据）。我有一个包含四列的数据表（请记住，以后可能会添加更多列），它们本身都不是唯一的，但是这三列一起'ID'，'ID2'，'DO'必须是唯一的作为一个团队。我会将这张表放入一个数据框中，并将更新后的表放入另一个数据框中。

如果 df 是“原始数据”而 df2 是“更新数据”，这是查找原始数据发生了什么变化的最准确/最有效的方法吗？

import pandas as pd
#Sample Data:
df  = pd.DataFrame({'ID':[546,107,478,546,478], 'ID2':['AUSER','BUSER','CUSER','AUSER','EUSER'], 'DO':[3,6,8,4,6], 'DATA':['ORIG','ORIG','ORIG','ORIG','ORIG']})
df2 = pd.DataFrame({'ID':[107,546,123,546,123], 'ID2':['BUSER','AUSER','DUSER','AUSER','FUSER'], 'DO':[6,3,2,4,3], 'DATA':['CHANGE','CHANGE','CHANGE','ORIG','CHANGE']})
>>> df
   DATA  DO   ID    ID2
0  ORIG   3  546  AUSER
1  ORIG   6  107  BUSER
2  ORIG   8  478  CUSER
3  ORIG   4  546  AUSER
4  ORIG   6  478  EUSER

>>> df2
     DATA  DO   ID    ID2
0  CHANGE   6  107  BUSER
1  CHANGE   3  546  AUSER
2  CHANGE   2  123  DUSER
3    ORIG   4  546  AUSER
4  CHANGE   3  123  FUSER

#Compare Dataframes
merged = df2.merge(df, indicator=True, how='outer')

#Split the merged comparison into:
# - original records that will be updated or deleted 
# - new records that will be inserted or update the original record.
df_original = merged.loc[merged['_merge'] == 'right_only'].drop(columns=['_merge']).copy()
df_new = merged.loc[merged['_merge'] == 'left_only'].drop(columns=['_merge']).copy()

#Create another merge to determine if the new records will either be updates or inserts
check = pd.merge(df_new,df_original, how='left', left_on=['ID','ID2','DO'], right_on = ['ID','ID2','DO'], indicator=True)
in_temp  = check[['ID','ID2','DO']].loc[check['_merge']=='left_only']
upd_temp = check[['ID','ID2','DO']].loc[check['_merge']=='both']

#Create dataframes for each Transaction:
# - removals: Remove records based on provided key values
# - updates:  Update entire record based on key values
# - inserts:  Insert entire record
removals = pd.concat([df_original[['ID','ID2','DO']],df_new[['ID','ID2','DO']],df_new[['ID','ID2','DO']]]).drop_duplicates(keep=False)
updates  = df2.loc[(df2['ID'].isin(upd_temp['ID']))&(df2['ID2'].isin(upd_temp['ID2']))&(df2['DO'].isin(upd_temp['DO']))].copy()
inserts  = df2.loc[(df2['ID'].isin(in_temp['ID']))&(df2['ID2'].isin(in_temp['ID2']))&(df2['DO'].isin(in_temp['DO']))].copy()

结果：

>>> removals
    ID    ID2  DO
6  478  CUSER   8
8  478  EUSER   6

>>> updates
     DATA  DO   ID    ID2
0  CHANGE   6  107  BUSER
1  CHANGE   3  546  AUSER

>>> inserts
     DATA  DO   ID    ID2
2  CHANGE   2  123  DUSER
4  CHANGE   3  123  FUSER

重述问题。此逻辑是否会一致且正确地识别具有指定键列的两个数据帧之间的差异？有没有更有效或 Pythonic 的方法来解决这个问题？

更新了更多记录的样本数据和相应的结果。

Answer 1

import pandas as pd
#Sample Data:
df  = pd.DataFrame({'ID':[546,107,478,546], 'ID2':['AUSER','BUSER','CUSER','AUSER'], 'DO':[3,6,8,4], 'DATA':['ORIG','ORIG','ORIG','ORIG']})
df2 = pd.DataFrame({'ID':[107,546,123,546], 'ID2':['BUSER','AUSER','DUSER','AUSER'], 'DO':[6,3,2,4], 'DATA':['CHANGE','CHANGE','CHANGE','ORIG']})

对于更改：

#Concat both df and df2 together, and whenever there is two of the same, drop them both
df3 =  pd.concat([df, df2]).drop_duplicates(keep = False)

#Whenever the size of this following group by is 2 or more there was a change.
#Change
df3 = df3.groupby(['ID', 'ID2', 'DO'])['DATA']\
         .size()\
         .reset_index()\
         .query('DATA == 2')

df3.loc[:, 'DATA'] = 'CHANGE'

     ID  ID2    DO    DATA
0   107 BUSER    6   CHANGE
3   546 AUSER    3   CHANGE

对于插入：

#We can compare the ID comlumn for df and df2 and see whats new in df2

#Inserts
df2[(np.logical_not(df2['ID'].isin(df['ID'])))&
    (np.logical_not(df2['ID2'].isin(df['ID2'])))&
    (np.logical_not(df2['DO'].isin(df['DO'])))]

     ID  ID2    DO   DATA
2   123 DUSER   2   CHANGE

删除：

#Similar logic as above but flipped.

#Removals
df[(np.logical_not(df2['ID'].isin(df['ID'])))&
   (np.logical_not(df2['ID2'].isin(df['ID2'])))&
   (np.logical_not(df2['DO'].isin(df['DO'])))]

     ID  ID2    DO  DATA
2   478 CUSER   8   ORIG

编辑

df  = pd.DataFrame({'ID':[546,107,478,546,478], 'ID2':['AUSER','BUSER','CUSER','AUSER','EUSER'], 'DO':[3,6,8,4,6], 'DATA':['ORIG','ORIG','ORIG','ORIG','ORIG']})
df2 = pd.DataFrame({'ID':[107,546,123,546,123], 'ID2':['BUSER','AUSER','DUSER','AUSER','FUSER'], 'DO':[6,3,2,4,3], 'DATA':['CHANGE','CHANGE','CHANGE','ORIG','CHANGE']})

新的数据框。对于更改，我们将以完全相同的方式进行操作：

df3 =  pd.concat([df, df2]).drop_duplicates(keep = False)

#Change
Change = df3.groupby(['ID', 'ID2', 'DO'])['DATA']\
         .size()\
         .reset_index()\
         .query('DATA == 2')

Change.loc[:, 'DATA'] = 'CHANGE'

    ID   ID2    DO   DATA
0   107 BUSER   6   CHANGE
5   546 AUSER   3   CHANGE

对于插入/删除，我们将执行与上述相同的 groupby，但查询仅出现一次的 groupby。然后我们将跟进 df 和 df2 的内部连接，以查看添加/删除的内容。

InsertRemove = df3.groupby(['ID', 'ID2', 'DO'])['DATA']\
                  .size()\
                  .reset_index()\
                  .query('DATA == 1')

#Inserts
Inserts = InsertRemove.merge(df2, how = 'inner',  left_on= ['ID', 'ID2', 'DO'], right_on = ['ID', 'ID2', 'DO'])\
                      .drop('DATA_x', axis = 1)\
                      .rename({'DATA_y':'DATA'}, axis = 1)

     ID  ID2    DO   DATA
0   123 DUSER   2   CHANGE
1   123 FUSER   3   CHANGE

#Removals
Remove  = InsertRemove.merge(df, how = 'inner',  left_on= ['ID', 'ID2', 'DO'], right_on = ['ID', 'ID2', 'DO'])\
                      .drop('DATA_x', axis = 1)\
                      .rename({'DATA_y':'DATA'}, axis = 1)

     ID  ID2    DO  DATA
0   478 CUSER   8   ORIG
1   478 EUSER   6   ORIG