合并部分重叠的 JSON 文件，用最新数据覆盖旧数据答案

【问题标题】：Merge partially overlapping JSON files overwriting old data with most recent data合并部分重叠的 JSON 文件，用最新数据覆盖旧数据
【发布时间】：2020-03-23 10:22:58
【问题描述】：

我有一系列 JSON 文件。所有后续文件都包含与前一个文件部分重叠的数据。如下例所示，第一个文件的最后 3 行与第二个表的前 3 行具有相同的时间戳。

第一个文件：

                wind_v    wind_u    dewpoint     temp       rh
hours
1555038000000 -1.412255 -0.023800  283.15000  284.15000  24.53871
1555048800000 -0.164115 -1.692985  284.32549  286.47823  21.82451
1555059600000  0.055155 -2.692985  285.04983  288.09226  21.82451
1555070400000 -2.412255 -0.857100  284.26031  286.47823  33.66288
1555081200000 -1.192985 -0.055155  284.15000  285.15000  30.98614
1555092000000 -0.857100  0.114030  283.71146  284.32549  37.11403

第二个文件：

               wind_v   wind_u   dewpoint    temp       rh
hours
1555070400000 -0.0572  3.4300  210.152144  292.03969  79.8188
1555081200000  0.4200  4.7622  207.006067  291.71146  83.1700
1555092000000  1.1578 -1.2322  205.239848  294.32549  73.7388
1555102800000  0.1750  0.9200  205.420127  297.86420  83.2532
1555113600000  0.2778  2.6106  206.944729  297.03969  82.2800
1555124400000 -2.4828  3.3722  208.115948  296.15000  83.7500

我想将所有文件合并到一个文件中，从而用最近的值覆盖这些值。

import pandas as pd
import json
import os

directory = './'

fileNames = [x for x in os.listdir(directory) if x.endswith(".json")]
fileNames = sorted(fileNames)

# Open first 2 files
with open(directory + fileNames[0]) as file1, open(directory + fileNames[1]) as file2:
    data2 = json.load(file2)
    df2 = pd.DataFrame(data2['data'])
    df2.set_index('hours', inplace=True, drop=True)

    data1 = json.load(file1)
    df1 = pd.DataFrame(data1['data'])
    df1.set_index('hours', inplace=True, drop=True)

print(df1)
print(df1)

# This does not work either
result = pd.merge(df1, df2, how='outer', on='hours')

print(result)

我的代码做了什么：

               wind_v_x  wind_u_x  dewpoint_x     temp_x      rh_x  wind_v_y  wind_u_y  dewpoint_y     temp_y     rh_y
hours
1555038000000 -1.412255 -0.023800   283.15000  284.15000  24.53871       NaN       NaN         NaN        NaN      NaN
1555048800000 -0.164115 -1.692985   284.32549  286.47823  21.82451       NaN       NaN         NaN        NaN      NaN
1555059600000  0.055155 -2.692985   285.04983  288.09226  21.82451       NaN       NaN         NaN        NaN      NaN
1555070400000 -2.412255 -0.857100   284.26031  286.47823  33.66288   -0.0572    3.4300  210.152144  292.03969  79.8188
1555081200000 -1.192985 -0.055155   284.15000  285.15000  30.98614    0.4200    4.7622  207.006067  291.71146  83.1700
1555092000000 -0.857100  0.114030   283.71146  284.32549  37.11403    1.1578   -1.2322  205.239848  294.32549  73.7388
1555102800000       NaN       NaN         NaN        NaN       NaN    0.1750    0.9200  205.420127  297.86420  83.2532
1555113600000       NaN       NaN         NaN        NaN       NaN    0.2778    2.6106  206.944729  297.03969  82.2800
1555124400000       NaN       NaN         NaN        NaN       NaN   -2.4828    3.3722  208.115948  296.15000  83.7500

我希望它是怎样的：

                wind_v    wind_u   dewpoint      temp        rh
hours
1555038000000 -1.412255 -0.023800  283.15000   284.15000  24.53871
1555048800000 -0.164115 -1.692985  284.32549   286.47823  21.82451
1555059600000  0.055155 -2.692985  285.04983   288.09226  21.82451
1555070400000 -0.0572    3.4300    210.152144  292.03969  79.8188
1555081200000  0.4200    4.7622    207.006067  291.71146  83.1700
1555092000000  1.1578   -1.2322    205.239848  294.32549  73.7388
1555102800000  0.1750    0.9200    205.420127  297.86420  83.2532
1555113600000  0.2778    2.6106    206.944729  297.03969  82.2800
1555124400000 -2.4828    3.3722    208.115948  296.15000  83.7500

任何提示都会有所帮助。

【问题讨论】：

标签： pandas dataframe join merge

【解决方案1】：

一种解决方案是使用额外的列来保持文件优先级。

步骤如下：

根据文件优先级添加一列（1为最低优先级，n最高优先级）
使用concat连接所有数据帧
通过仅保留具有最高优先级的行来删除重复项：
- 使用groupby按小时分组行
- 使用transform 选择最大的priority 值。

代码：

print(df_1)
#                  wind_v    wind_u   dewpoint       temp        rh  file
# hours                                                                  
# 1555038000000 -1.412255 -0.023800  283.15000  284.15000  24.53871     1
# 1555048800000 -0.164115 -1.692985  284.32549  286.47823  21.82451     1
# 1555059600000  0.055155 -2.692985  285.04983  288.09226  21.82451     1
# 1555070400000 -2.412255 -0.857100  284.26031  286.47823  33.66288     1
# 1555081200000 -1.192985 -0.055155  284.15000  285.15000  30.98614     1
# 1555092000000 -0.857100  0.114030  283.71146  284.32549  37.11403     1
print(df_2)
#                wind_v  wind_u    dewpoint       temp       rh  file
# hours                                                              
# 1555070400000 -0.0572  3.4300  210.152144  292.03969  79.8188     2
# 1555081200000  0.4200  4.7622  207.006067  291.71146  83.1700     2
# 1555092000000  1.1578 -1.2322  205.239848  294.32549  73.7388     2
# 1555102800000  0.1750  0.9200  205.420127  297.86420  83.2532     2
# 1555113600000  0.2778  2.6106  206.944729  297.03969  82.2800     2
# 1555124400000 -2.4828  3.3722  208.115948  296.15000  83.7500     2

df = pd.concat([df_1,df_2])
print(df)
#                 wind_v    wind_u    dewpoint       temp        rh  file
# hours                                                                   
# 1555038000000 -1.412255 -0.023800  283.150000  284.15000  24.53871     1
# 1555048800000 -0.164115 -1.692985  284.325490  286.47823  21.82451     1
# 1555059600000  0.055155 -2.692985  285.049830  288.09226  21.82451     1
# 1555070400000 -2.412255 -0.857100  284.260310  286.47823  33.66288     1
# 1555081200000 -1.192985 -0.055155  284.150000  285.15000  30.98614     1
# 1555092000000 -0.857100  0.114030  283.711460  284.32549  37.11403     1
# 1555070400000 -0.057200  3.430000  210.152144  292.03969  79.81880     2
# 1555081200000  0.420000  4.762200  207.006067  291.71146  83.17000     2
# 1555092000000  1.157800 -1.232200  205.239848  294.32549  73.73880     2
# 1555102800000  0.175000  0.920000  205.420127  297.86420  83.25320     2
# 1555113600000  0.277800  2.610600  206.944729  297.03969  82.28000     2
# 1555124400000 -2.482800  3.372200  208.115948  296.15000  83.75000     2

df = df[df['file'] == df.groupby("hours")['file'].transform('max')]
#                 wind_v    wind_u    dewpoint       temp        rh  file
# hours                                                                   
# 1555038000000 -1.412255 -0.023800  283.150000  284.15000  24.53871     1
# 1555048800000 -0.164115 -1.692985  284.325490  286.47823  21.82451     1
# 1555059600000  0.055155 -2.692985  285.049830  288.09226  21.82451     1
# 1555070400000 -2.412255 -0.857100  284.260310  286.47823  33.66288     1
# 1555081200000 -1.192985 -0.055155  284.150000  285.15000  30.98614     1
# 1555092000000 -0.857100  0.114030  283.711460  284.32549  37.11403     1
# 1555070400000 -0.057200  3.430000  210.152144  292.03969  79.81880     2
# 1555081200000  0.420000  4.762200  207.006067  291.71146  83.17000     2
# 1555092000000  1.157800 -1.232200  205.239848  294.32549  73.73880     2
# 1555102800000  0.175000  0.920000  205.420127  297.86420  83.25320     2
# 1555113600000  0.277800  2.610600  206.944729  297.03969  82.28000     2
# 1555124400000 -2.482800  3.372200  208.115948  296.15000  83.75000     2

【讨论】：

首先我将表与pd.merge(df1, df2, how='outer', on='hours') 合并，然后将_x 和_y 列与result['wind_v_y'].combine_first(result['wind_v_x']) 合并，删除不必要的_x 和_y 列与result.drop(['wind_v_x', 'wind_v_y'], axis=1, inplace=True)。不确定这是否是正确的方法，但有效。
应该可以！然而，outermerge 的问题是数据框的大小会随着使用多个文件而急剧增加（行和列）。所有没有重复的记录都会导致空单元格。