使用熊猫有条件地合并两个数据框答案

【问题标题】：Conditional merging of two dataframes using pandas使用熊猫有条件地合并两个数据框
【发布时间】：2021-12-13 23:03:20
【问题描述】：

我有两个数据框 - df1 具有 ISIN、Name、Weight 等列，而 df2 具有 Short Name、ISIN 等列。

df1 =

ISIN    Name            Weight
        Enbridge Inc    0.1
        UDR Inc         1.1
        Tyson Foods Inc 1.9

和 df2=

Short Name            ISIN
Enbridge Inc.         bvefj154
UDR Group             iuhb38g7
Tyson Foods Pvt Ltd.  hruidf12

我开发了一个模糊逻辑，它将匹配 df1 和 df2 中的 Name 和 Short Name。因此它使用逻辑它会知道来自两个数据帧的 Enbridge Inc 是相同的。对于 UDR Group 和 UDR Inc，它们也是相同的，因为名称是匹配的，不是全部而是几乎全部。

我正在寻找一种在 df1 中填充 ISIN column 的方法，其逻辑是如果名称匹配（Enbridge Inc 全部匹配），然后从 df2 中选择相应 Short Name 的 ISIN 并将其添加到 ISIN df1 中存在相关名称的列。

所以我期待的输出应该是这样的： df1=

ISIN            Name            Weight
bvefj154        Enbridge Inc    0.1
iuhb38g7        UDR Inc         1.1
hruidf12        Tyson Foods Inc 1.9

使用 pandas 的 merge 函数我尝试完成任务但收到如下错误：

KeywordError:'Name'

这是相同的代码。

import pandas as pd
df1 = pd.merge(df1, df2, on=['Name', '% Weight'], how='right')

我该怎么做？请帮忙。

编辑：这是使用fuzzywuzzy模块进行模糊逻辑匹配的代码

def fuzzy_merge(df_1, df_2, key1, key2, threshold=90, limit=1):
    """
    :param df_1: the left table to join
    :param df_2: the right table to join
    :param key1: key column of the left table
    :param key2: key column of the right table
    :param threshold: how close the matches should be to return a match, based on Levenshtein distance
    :param limit: the amount of matches that will get returned, these are sorted high to low
    :return: dataframe with boths keys and matches
    """
    s = df_2[key2].tolist()

    m = df_1[key1].apply(lambda x: process.extract(x, s, limit=limit))
    df_1['matches'] = m

    m2 = df_1['matches'].apply(lambda x: ', '.join([i[0] for i in x if i[1] >= threshold]))
    df_1['matches'] = m2

    print(df_1)
    df_1.to_csv('fuzzy-1390-match.csv')
    #return df_1


fuzzy_merge(df1, df2, 'Name', 'Short Name', threshold=90)

输出：

5,Enbridge Inc Flt 07/15/80 Sr:20-A,0.0127,ENBRIDGE INC
6,Enbridge Inc. 6.25% 03/01/78,0.0122,ENBRIDGE INC
7,Emera 6.75% 6/15/76-26,0.0113,MERA
8,Scentre Group Trust 2 Flt 09/24/80 Sr:144A,0.011,SCENTRE GROUP
9,Credit Suisse Group AG 7.5 Perp,0.0106,
10,Aegon Funding Corp Ii 5.100% 12/15/49,0.0101,
11,Dte Energy Co 5.250% 12/01/77 Sr:E,0.01,DTE ENERGY CO
12,Dai-Ichi Life Insurance 4%,0.0099,
13,Southern Co Flt 09/15/51 Sr:21-A,0.0098,SOUTHERN CO

EDIT2：

这是数据框（df1 和 df2）： df1=

0             Transcanada Trust 5.875 08/15/76    0.0176
1              Bp Capital Markets Plc Flt Perp    0.0169
2               Transcanada Trust Flt 09/15/79    0.0169
3              Bp Capital Markets Plc Flt Perp    0.0155
4          Prudential Financial 5.375% 5/15/45    0.0150
5            Enbridge Inc Flt 07/15/80 Sr:20-A    0.0127
6                 Enbridge Inc. 6.25% 03/01/78    0.0122
7                       Emera 6.75% 6/15/76-26    0.0113
8   Scentre Group Trust 2 Flt 09/24/80 Sr:144A    0.0110
9              Credit Suisse Group AG 7.5 Perp    0.0106
10       Aegon Funding Corp Ii 5.100% 12/15/49    0.0101
11          Dte Energy Co 5.250% 12/01/77 Sr:E    0.0100
12                  Dai-Ichi Life Insurance 4%    0.0099
13            Southern Co Flt 09/15/51 Sr:21-A    0.0098
14         Prudential Financial 5.625% 6/15/43    0.0097
15         Southern Co 4.950% 01/30/80 Sr:2020    0.0093
16  Scentre Group Trust 2 Flt 09/24/80 Sr:144A    0.0093
17             Metlife Inc 9.25% 4/8/2038 144A    0.0089
18          American Intl Group 8.175% 5/15/58    0.0086
19               Southern Co Flt 01/15/51 Sr:B    0.0079

df2=

          Short Name          ISIN
0   ABU DHABI COMMER  AEA000201011
1   ABU DHABI NATION  AEA002401015
2   ABU DHABI NATION  AEA006101017
3   ADNOC DRILLING C  AEA007301012
4   ALPHA DHABI HOLD  AEA007601015
5      DUBAI ISLAMIC  AED000201015
6    EMAAR PROP PJSC  AEE000301011
7           ETISALAT  AEE000401019
8   EMIRATES NBD PJS  AEE000801010
9    INTL HOLDING CO  AEI000201014
10   FIRST ABU DHABI  AEN000101016
11  SCHLUMBERGER LTD  AN8068571086
12  ERSTE GROUP BANK  AT0000652011
13            OMV AG  AT0000743059
14        VERBUND AG  AT0000746409
15  ARISTOCRAT LEISU  AU000000ALL7
16  AUST AND NZ BANK  AU000000ANZ3
17      AFTERPAY LTD  AU000000APT1
18           ASX LTD  AU000000ASX7
19     BHP GROUP LTD  AU000000BHP4

【问题讨论】：

您的问题是Name 在两个dfs 中都不存在。您应该合并left_on 和right_on，在您的情况下为Name 和short_name。但既然你想使用模糊合并，你应该先这样做
@JoshFriedlander 是的，我先进行模糊匹配，然后进行合并，但是我不确定在 left_on 和 right_on 上添加哪些列名。你能举个例子吗？
模糊匹配是什么样的？它是一个功能吗？它输出什么？ here 是一个可能适用于您的模糊合并示例
@JoshFriedlander 我使用了相同的代码，在编辑中添加了输出和代码

标签： python pandas dataframe

【解决方案1】：

您可以使用split 来获取结果。我搜索 df1.Name 中的第一个单词是否在 df2.Short Name 中

import pandas as pd

df1 = pd.DataFrame({'ISIN': ['', '', ''], 'Name': ['Enbridge Inc', 'UDR Inc', 'Tyson Foods Inc'], 'Weight': ['0.1', '1.1', '1.9']})
df2 = pd.DataFrame({'Short Name': ['Enbridge Inc.', 'UDR Group', 'Tyson Foods Pvt Ltd.'], 'ISIN': ['bvefj154', 'iuhb38g7', 'hruidf12']})

def strMergeData(strColumnDf1):
    strColumnDf1 = strColumnDf1.split()[0]
    for strColumnDf2 in df2['Short Name']:
        if strColumnDf1 in strColumnDf2:
            return df2[df2['Short Name'] == strColumnDf2]['ISIN'].values[0]
            break
        else:
            pass
        
df1['ISIN'] = df1.apply(lambda x: strMergeData(x['Name']),axis=1)
print(df1)

输出：

       ISIN             Name Weight
0  bvefj154     Enbridge Inc    0.1
1  iuhb38g7          UDR Inc    1.1
2  hruidf12  Tyson Foods Inc    1.9

Demo

您可以在下面提供的代码中找到您的示例测试。我只是在 ALPHA DHABI HOLD 中添加 TRANSCANADA 以匹配一些示例。

import pandas as pd

df1 = pd.DataFrame({'ISIN': ['', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', ''],
                    'Name': ['Transcanada Trust 5.875 08/15/76',
                             'Bp Capital Markets Plc Flt Perp',
                             'Transcanada Trust Flt 09/15/79',
                             'Bp Capital Markets Plc Flt Perp',
                             'Prudential Financial 5.375% 5/15/45',
                             'Enbridge Inc Flt 07/15/80 Sr:20-A',
                             'Enbridge Inc. 6.25% 03/01/78',
                             'Emera 6.75% 6/15/76-26',
                             'Scentre Group Trust 2 Flt 09/24/80 Sr:144A',
                             'Credit Suisse Group AG 7.5 Perp',
                             'Aegon Funding Corp Ii 5.100% 12/15/49',
                             'Dte Energy Co 5.250% 12/01/77 Sr:E',
                             'Dai-Ichi Life Insurance 4%',
                             'Southern Co Flt 09/15/51 Sr:21-A',
                             'Prudential Financial 5.625% 6/15/43',
                             'Southern Co 4.950% 01/30/80 Sr:2020',
                             'Scentre Group Trust 2 Flt 09/24/80 Sr:144A',
                             'Metlife Inc 9.25% 4/8/2038 144A',
                             'American Intl Group 8.175% 5/15/58',
                             'Southern Co Flt 01/15/51 Sr:B',
                             19.5],
                    'Weight': [0.0176, 0.0169, 0.0169, 0.0155,0.0150,0.0127,0.0122,0.0113,0.0110,0.0106,0.0101,0.0100
                               ,0.0099,0.0098,0.0097,0.0093,0.0093,0.0089,0.0086,0.0079,0.0091]})

df2 = pd.DataFrame({'Short Name': ['ABU DHABI COMMER', 'ABU DHABI NATION', 'ABU DHABI NATION',
                                   'ADNOC DRILLING C','TRANSCANADA ALPHA DHABI HOLD','DUBAI ISLAMIC' ,
                                   'EMAAR PROP PJSC','ETISALAT','EMIRATES NBD PJS','INTL HOLDING CO' ,
                                   'FIRST ABU DHABI'  ,'SCHLUMBERGER LTD'  ,'ERSTE GROUP BANK'  ,'OMV AG',
                                   'VERBUND AG',  'ARISTOCRAT LEISU',  'AUST AND NZ BANK',  'AFTERPAY LTD',
                                   'ASX LTD',  'BHP GROUP LTD',19.5],
                    'ISIN': [ 'AEA000201011','AEA002401015','AEA006101017','AEA007301012','AEA007601015',
                              'AED000201015','AEE000301011','AEE000401019','AEE000801010','AEI000201014',
                              'AEN000101016','AN8068571086','AT0000652011','AT0000743059','AT0000746409',
                              'AU000000ALL7','AU000000ANZ3','AU000000APT1','AU000000ASX7','AU000000BHP4','FLOAT_TEST'] })

def strMergeData(strColumnDf1):
    strColumnDf1 = str(strColumnDf1).split()[0]
    for strColumnDf2 in df2['Short Name']:
        if str(strColumnDf1).upper() in str(strColumnDf2).upper():
            return df2[df2['Short Name'] == strColumnDf2]['ISIN'].values[0]
            break
        else:
            pass
        
df1['ISIN'] = df1.apply(lambda x: strMergeData(x['Name']),axis=1)
print(df1)

输出：

            ISIN                                        Name  Weight
0   AEA007601015            Transcanada Trust 5.875 08/15/76  0.0176
1           None             Bp Capital Markets Plc Flt Perp  0.0169
2   AEA007601015              Transcanada Trust Flt 09/15/79  0.0169
3           None             Bp Capital Markets Plc Flt Perp  0.0155
4           None         Prudential Financial 5.375% 5/15/45  0.0150
5           None           Enbridge Inc Flt 07/15/80 Sr:20-A  0.0127
6           None                Enbridge Inc. 6.25% 03/01/78  0.0122
7           None                      Emera 6.75% 6/15/76-26  0.0113
8           None  Scentre Group Trust 2 Flt 09/24/80 Sr:144A  0.0110
9           None             Credit Suisse Group AG 7.5 Perp  0.0106
10          None       Aegon Funding Corp Ii 5.100% 12/15/49  0.0101
11          None          Dte Energy Co 5.250% 12/01/77 Sr:E  0.0100
12          None                  Dai-Ichi Life Insurance 4%  0.0099
13          None            Southern Co Flt 09/15/51 Sr:21-A  0.0098
14          None         Prudential Financial 5.625% 6/15/43  0.0097
15          None         Southern Co 4.950% 01/30/80 Sr:2020  0.0093
16          None  Scentre Group Trust 2 Flt 09/24/80 Sr:144A  0.0093
17          None             Metlife Inc 9.25% 4/8/2038 144A  0.0089
18          None          American Intl Group 8.175% 5/15/58  0.0086
19          None               Southern Co Flt 01/15/51 Sr:B  0.0079
20    FLOAT_TEST                                        19.5  0.0091

Demo

【讨论】：

查看演示链接。你会看到它工作正常
如果您没有在问题中给出所有示例，我不知道是否存在浮动。你能分享一个我可以检查的完整数据框的例子吗？
共享。请在编辑中查看
检查我的更新
我在数据框中添加了一个浮点数。你可以再检查一下