【问题标题】:Merge dataframes and remove duplicate columns合并数据框并删除重复的列
【发布时间】:2020-08-16 04:10:43
【问题描述】:

我一整天都在研究这个问题,但找不到更好的解决方案。我有数据框列表

ls=[df1,df2,df3,df4] 并尝试了几种合并方法。

df= reduce(lambda left,right: pd.merge(left,right,on='Time',how='outer'), ls)

我阅读的大多数示例只是删除 col_y 列并更改 col_x 列名称。我怎样才能实现所需的数据框,在此先感谢。

df1

Time col1 col2
 1  a   11
 2  b   12
 3  c   13
 4  d   14

df2

Time col3 col1
1   11    na
2   15    na
3   66    na
4   78    na
5   33    f
6   22    g

合并df3

 Time   col1_x  col2    col3    col1_y
 1      a      11     11           na
 2      b      12     15           na
 3      c      13     66           na
 4      d      14     78           na
 5     na      na     33            f
 6     na      na     22            g

我希望最终的 df 是什么:

所需的合并df

 Time   col1    col2    col3    
 1      a      11     11           
 2      b      12     15           
 3      c      13     66           
 4      d      14     78           
 5      f      na     33            
 6      g      na     22            

【问题讨论】:

    标签: python pandas dataframe


    【解决方案1】:

    将combine_first 与drop 一起使用,如果您有超过2 个数据帧,请在合并并级联combine_first 与df3 和df4 后执行此操作:

    df = df1.merge(df2, how='outer', on=['Time'])
    df['col1'] = df['col1_x'].combine_first(df['col1_y'])
    df = df.drop(['col1_x', 'col1_y'], axis=1)
    #or alternative solution
    df1 = df1.set_index('time')
    df2 = df2.set_index('time')
    df = df1.combine_first(df2)
    
    print(df)
       Time  col2  col3 col1
    0     1  11.0    11    a
    1     2  12.0    15    b
    2     3  13.0    66    c
    3     4  14.0    78    d
    4     5   NaN    33    f
    5     6   NaN    22    g
    

    将time设置为索引后,您可以使用combine_first的数据帧操作:

    df1 = df1.set_index('time')
    df2 = df2.set_index('time')
    df3 = df3.set_index('time')
    df4 = df4.set_index('time')
    df = df1.combine_first(df2).combine_first(df3).combine_first(df4)
    

    使用reduce函数:

    df = reduce(lambda left,right: left.set_index('time').combine_first(right.set_index('time')), 
                ls)
    

    【讨论】:

      【解决方案2】:

      当数据帧共享索引时,您可以利用这一事实,分配和替换在操作之前对齐索引。

      df1 = pd.DataFrame.from_records(
          [
              {"Time": 1, "col1": "a", "col2": 11},
              {"Time": 2, "col1": "b", "col2": 12},
              {"Time": 3, "col1": "c", "col2": 13},
              {"Time": 4, "col1": "d", "col2": 14},
          ]
      ).set_index("Time")
      
      df2 = pd.DataFrame.from_records(
          [
              {"Time": 1, "col3": 11, "col1": np.nan},
              {"Time": 2, "col3": 15, "col1": np.nan},
              {"Time": 3, "col3": 66, "col1": np.nan},
              {"Time": 4, "col3": 78, "col1": np.nan},
              {"Time": 5, "col3": 33, "col1": "f"},
              {"Time": 6, "col3": 22, "col1": "g"},
          ]
      ).set_index("Time")
      
      dfs = [df1, df2]
      
      index = pd.Index(dfs[0].index)
      for this_df in dfs[1:]:
          index = index.union(this_df.index)
      
      df = pd.DataFrame(index=index)
      
      for this_df in dfs:
          for col in this_df.columns:
              if col not in df.columns:
                  df[col] = this_df[col]
              else:
                  df[col] = df[col].fillna(this_df[col])
      
      print(df)
      
           col1  col2  col3
      Time                 
      1       a  11.0    11
      2       b  12.0    15
      3       c  13.0    66
      4       d  14.0    78
      5       f   NaN    33
      6       g   NaN    22
      

      【讨论】:

        猜你喜欢
        • 2011-11-18
        • 2015-01-01
        • 1970-01-01
        • 2016-10-24
        • 2016-11-09
        • 2017-05-31
        • 2020-12-21
        • 1970-01-01
        • 2018-06-09
        相关资源
        最近更新 更多