在两个字典之间对齐 DataFrame 的行和列答案

【问题标题】：Align the rows and columns of DataFrames between two Dictionaries在两个字典之间对齐 DataFrame 的行和列
【发布时间】：2022-01-17 14:45:21
【问题描述】：

我试图弄清楚如何对齐存储在两个单独的dictionaries 中的许多DataFrames 的行和列。

为了说明这一点，我创建了四个DataFramesdf1、df2、df3和df4，我将它们存储在dictionaries中：

df1:
    Dates       ID1  ID2  ID3   ID4  ID6
0   2021-01-01  0.0  0.0  0.1   0.0  0.0
1   2021-01-03  0.0  0.0  0.0   0.0  0.0
2   2021-01-04  0.1  0.1  0.0   0.1  0.1
3   2021-01-05  0.2  0.2  0.2   0.2  0.2
4   2021-01-06  0.1  0.1  0.4   0.1  0.1

df2:
    Dates       ID1    ID2   ID3   ID4   ID6
0   2021-01-01  0.00   0.0   0.1   0.0   0.0
1   2021-01-03  0.20   0.3   0.0   0.0   0.0
2   2021-01-04  0.15   -0.1  0.4   -0.1  -0.1
3   2021-01-05  0.20   -0.2  -0.2   0.3  0.2
4   2021-01-06  0.10   0.1   0.2   -0.1  0.1

df3:
    Dates       ID2   ID4    ID5   ID6
0   2021-01-01  3     1.0    4     2.0
1   2021-01-02  6     2.4    3     1.5
2   2021-01-03  3     -2.0   3     2.0
3   2021-01-04  -1    1.0    -3    -2.0
4   2021-01-05  3     4.0    4     3.0

df4:
    Dates       ID2   ID4   ID5    ID6
0   2021-01-02  6     2.4   3     1.5
1   2021-01-03  3     -2.0  3     2.0
2   2021-01-04  -1    1.0   -3    -2.0
3   2021-01-07  3     4.0   4     3.0

为了重现性：

import pandas as pd
df1 = pd.DataFrame({
    'Dates':['2021-01-01', '2021-01-03', '2021-01-04', '2021-01-05', '2021-01-06'],
    'ID1':[0,0,0.1,0.2,0.1],
    'ID2':[0,0,0.1,0.2,0.1],
    'ID3':[0.1,0,0,0.2,0.4],
    'ID4':[0,0,0.1,0.2,0.1], 
    'ID6':[0,0,0.1,0.2,0.1]})

df2 = pd.DataFrame({
    'Dates':['2021-01-01', '2021-01-03', '2021-01-04', '2021-01-05', '2021-01-06'],
    'ID1':[0,0.2,0.15,0.2,0.1],
    'ID2':[0,0.3,-0.1,-0.2,0.1],
    'ID3':[0.1,0,0.4,-0.2,0.2],
    'ID4':[0,0,-0.1,0.3,-0.1], 
    'ID6':[0,0,-0.1,0.2,0.1]})

df3 = pd.DataFrame({
    'Dates':['2021-01-01', '2021-01-02', '2021-01-03', '2021-01-04', '2021-01-05'], 
    'ID2':[3,6,3,-1,3],
    'ID4':[1,2.4,-2,1,4],
    'ID5':[4,3,3,-3,4],
    'ID6':[2,1.5,2,-2,3]})

df4 = pd.DataFrame({
    'Dates':['2021-01-02', '2021-01-03', '2021-01-04', '2021-01-07'], 
    'ID2':[6,3,-1,3],
    'ID4':[2.4,-2,1,4],
    'ID5':[3,3,-3,4],
    'ID6':[1.5,2,-2,3]})

dic1 = dict()
dic1['df1'] = df1
dic1['df2'] = df2

dic2 = dict()
dic2['df3'] = df3
dic2['df4'] = df4

我想得到的是：

dic1['df1']:
    Dates       ID2  ID4    ID6
0   2021-01-03  0.0  0.0    0.0
1   2021-01-04  0.1  0.1    0.1

dic1['df2']:
    Dates       ID2   ID4   ID6
0   2021-01-03  0.3   0.0   0.0
1   2021-01-04  -0.1  -0.1  -0.1

dic2['df3']:
    Dates       ID2  ID4   ID6
0   2021-01-03  3    -2    2
1   2021-01-04  -1   1     -2

dic2['df4']:
    Dates       ID2  ID4   ID6
0   2021-01-03  3    -2    2
1   2021-01-04  -1   1     -2

我尝试对齐两个dictionaries 的行和列，这样我只包括不同dictionaries 的所有DataFrames 中包含的列。 dictionaries 中是否有有效的命令来处理这个问题？

【问题讨论】：

标签： python pandas dictionary

【解决方案1】：

在DataFrames 中使用DataFrame.align 和创建DatetmeIndex：

dic1['df1'], dic2['df2'] = (dic1['df1'].set_index('Dates')
                              .align(dic2['df2'].set_index('Dates'),
                                     fill_value=0, 
                                     join='inner'))
print (dic1['df1'])
            ID2  ID4  ID6
Dates                    
2021-01-01  0.0  0.0  0.0
2021-01-03  0.0  0.0  0.0
2021-01-04  0.1  0.1  0.1
2021-01-05  0.2  0.2  0.2

print (dic2['df2'])
            ID2  ID4  ID6
Dates                    
2021-01-01    3  1.0  2.0
2021-01-03    3 -2.0  2.0
2021-01-04   -1  1.0 -2.0
2021-01-05    3  4.0  3.0

a, b = (dic1['df1'].set_index('Dates')
               .align(dic2['df2'].set_index('Dates'), 
                      fill_value=0,
                      join='inner'))

dic1['df1'], dic2['df2'] = a.reset_index(), b.reset_index()
print (dic1['df1'])
        Dates  ID2  ID4  ID6
0  2021-01-01  0.0  0.0  0.0
1  2021-01-03  0.0  0.0  0.0
2  2021-01-04  0.1  0.1  0.1
3  2021-01-05  0.2  0.2  0.2

print (dic2['df2'])
        Dates  ID2  ID4  ID6
0  2021-01-01    3  1.0  2.0
1  2021-01-03    3 -2.0  2.0
2  2021-01-04   -1  1.0 -2.0
3  2021-01-05    3  4.0  3.0

编辑：首先将所有字典合并到一个大字典中，然后将 Dates 和列名提取到嵌套列表中，最后按 DataFrame.reindex 过滤值：

idx, cols = [],[]
for k, v in {**dic1, **dic2}.items():
    cols.append(v.drop('Dates', axis=1).columns)
    idx.append(v['Dates'])
    
from functools import reduce
idx = reduce(lambda x, y: set(x) & set(y), idx)
print (idx)
{'2021-01-04', '2021-01-03'}

cols = reduce(lambda x, y: set(x) & set(y), cols)
print (cols)
{'ID4', 'ID2', 'ID6'}

for k, v in dic1.items():
    dic1[k] = dic1[k].set_index('Dates').reindex(index=idx, columns=cols, fill_value=0).reset_index()

for k, v in dic2.items():
    dic2[k] = dic2[k].set_index('Dates').reindex(index=idx, columns=cols, fill_value=0).reset_index()

【讨论】：

非常感谢您提供这个优雅的解决方案！
如果我每个字典都有几个DataFrame，会不会有一个简单的调整？
@fjurt - 你能说得更具体一些吗？添加示例数据？
在所有字典里一起
非常感谢！

【解决方案2】：

您可以计算 df1 和 df2 列和日期的交集并只保留这些：

df1 = pd.DataFrame({
    'Dates':['2021-01-01', '2021-01-03', '2021-01-04', '2021-01-05', '2021-01-06'],
    'ID1':[0,0,0.1,0.2,0.1],
    'ID2':[0,0,0.1,0.2,0.1],
    'ID3':[0.1,0,0,0.2,0.4],
    'ID4':[0,0,0.1,0.2,0.1], 
    'ID6':[0,0,0.1,0.2,0.1]})

df2 = pd.DataFrame({
    'Dates':['2021-01-01', '2021-01-02', '2021-01-03', '2021-01-04', '2021-01-05'], 
    'ID2':[3,6,3,-1,3],
    'ID4':[1,2.4,-2,1,4],
    'ID5':[4,3,3,-3,4],
    'ID6':[2,1.5,2,-2,3]})

cols = set(df1.columns).intersection(set(df2.columns))
dates = set(df1["Dates"]).intersection(set(df2["Dates"]))

dic1 = dict()
dic1['df1'] = df1[df1["Dates"].isin(dates)][cols].reset_index()

dic2 = dict()
dic2['df2'] = df2[df2["Dates"].isin(dates)][cols].reset_index()

print(dic1['df1'])
print(dic2['df2'])

输出：

   index       Dates  ID6  ID2  ID4
0      0  2021-01-01  0.0  0.0  0.0
1      1  2021-01-03  0.0  0.0  0.0
2      2  2021-01-04  0.1  0.1  0.1
3      3  2021-01-05  0.2  0.2  0.2
   index       Dates  ID6  ID2  ID4
0      0  2021-01-01  2.0    3  1.0
1      2  2021-01-03  2.0    3 -2.0
2      3  2021-01-04 -2.0   -1  1.0
3      4  2021-01-05  3.0    3  4.0

【讨论】：

非常感谢，效果很好