根据大型数据集的几个条件合并 pandas 数据帧

Question

我想根据特定条件合并两个数据框。首先，我只想匹配全名，然后对于不匹配的条目，我想将名字和姓氏视为匹配条件。我有两个数据框如下：

df1
first_name  last_name   full_name
   John       Shoeb     John Shoeb
   John      Shumon   John Md Shumon
   Abu        Babu      Abu A Babu
  William     Curl      William Curl   

df2
givenName    surName     displayName
John          Shoeb      John Shoeb
John         Shumon     John M Shumon
Abu           Babu        Abu Babu
Raju          Kaju        Raju Kaju
Bill          Curl        Bill Curl

我先根据全名合并：

df3 = pd.merge(df1, df2, left_on=df1['full_name'].str.lower(), right_on=df2['displayName'].str.lower(), how='left')

并添加status 和log 列：

df3.loc[ (df3.full_name.str.lower()==df3.displayName.str.lower()), 'status'] = True
df3.loc[ (df3.full_name.str.lower()==df3.displayName.str.lower()), 'log'] = 'Full Name Matching'

所以生成的数据框 df3 现在看起来像：

first_name   last_name     full_name    givenName   surName   displayName   status  log
John          Shoeb        John Shoeb      John      Shoeb    John Shoeb    True    Full Name Matching
  John        Shumon     John Md Shumon     NaN       NaN        NaN         NaN    NaN
  Abu          Babu        Abu A Babu       NaN       NaN        NaN         NaN    NaN
William        Curl       William Curl      NaN       NaN        NaN        False   NaN

预期结果 现在我想应用基于 df1（名字和姓氏）和 df2（givenName 和 surName）的匹配条件。最终的数据框应如下所示：

  first_name     last_name     full_name    givenName   surName   displayName   status  log
    John          Shoeb        John Shoeb      John      Shoeb    John Shoeb    True    Full Name Matching
      John        Shumon     John Md Shumon    John     Shumon    John Shumon   True    FN LN Matching
      Abu          Babu        Abu A Babu       Abu       Babu      Abu Babu    True    FN LN Matching
    William        Curl       William Curl      NaN       NaN        NaN        False   NaN

问题对于第二部分，即名字和姓氏匹配，我能够使用数据框的itertuples() 完成它。但是，当将相同的操作应用于庞大的数据集时，它会一直运行下去。我正在寻找有效的方法，以便将其应用于大量数据。

Answer 1

您可以在合并中使用indicator=True。然后比较第一次合并和第二次合并是否为"both"（例如np.where）：

df3 = (
    pd.merge(
        df1,
        df2,
        left_on=df1["full_name"].str.lower(),
        right_on=df2["displayName"].str.lower(),
        how="left",
        indicator=True,
    )
    .drop(columns="key_0")
    .rename(columns={"_merge": "first_merge"})
)

df3 = pd.merge(
    df3,
    df2,
    left_on=df1["first_name"].str.lower() + " " + df1["last_name"].str.lower(),
    right_on=df2["givenName"].str.lower() + " " + df2["surName"].str.lower(),
    how="left",
    indicator=True,
)

df3["log"] = np.where(
    (df3["first_merge"] == "both"),
    "Full Name Matching",
    np.where(df3["_merge"] == "both", "FN LN Matching", None),
)
df3["status"] = df3["log"].notna()

df3 = df3[
    [
        "first_name",
        "last_name",
        "full_name",
        "givenName_y",
        "surName_y",
        "displayName_y",
        "status",
        "log",
    ]
].rename(
    columns={
        "givenName_y": "givenName",
        "surName_y": "surName",
        "displayName_y": "displayName",
    }
)
print(df3)

打印：

  first_name last_name       full_name givenName surName    displayName  status                 log
0       John     Shoeb      John Shoeb      John   Shoeb     John Shoeb    True  Full Name Matching
1       John    Shumon  John Md Shumon      John  Shumon  John M Shumon    True      FN LN Matching
2        Abu      Babu      Abu A Babu       Abu    Babu       Abu Babu    True      FN LN Matching
3    William      Curl    William Curl       NaN     NaN            NaN   False                None