查找两个不同数据框列之间的部分匹配，并在找到匹配时分配值

Question

我想用df2 数据框的“类别”列中的正确值填充df1 数据框的“类别”列。

import pandas as pd

df1 = pd.DataFrame({"Receiver": ["Insurance company", "Shop", "Pizza place", "Library", "Gas station 24/7", "Something else", "Whatever receiver"], "Category": ["","","","","","",""]}) 
df2 = pd.DataFrame({"Category": ["Insurances", "Groceries", "Groceries", "Fastfood", "Fastfood", "Car"], "Searchterm": ["Insurance", "Shop", "Market", "Pizza", "Burger", "Gas"]})

输出：

df1
Receiver                Category
0   Insurance company   
1   Shop    
2   Pizza place 
3   Library 
4   Gas station 24/7    
5   Something else  
6   Whatever receiver   

df2
    Category    Searchterm
0   Insurances  Insur
1   Groceries   Shop
2   Groceries   Market
3   Fastfood    Pizza
4   Fastfood    Burger
5   Car         Gas

我想逐行比较df1["Receiver"] 和df2["Searchterm"]，如果后者甚至部分匹配前者，则将该行的df2["Category"] 分配给df1["Category"]。

例如，df2["Searchterm"] 中的“Pizza”部分匹配 df1["Receiver"] 中的“Pizza place”，所以我想将“Fastfood”（这是 df2["Category"] 中的 Pizza 类别）分配给“Pizza place” df1["Category"] 中的类别。

期望的输出是：

df1
Receiver                Category
0   Insurance company   Insurances
1   Shop                Groceries
2   Pizza place         Fastfood
3   Library             
4   Gas station 24/7    Car
5   Something else      
6   Whatever receiver   

那么我怎样才能用正确的类别填写df1["Category"]？谢谢。

Answer 1

迭代类别

在类别数量相对于接收者数量较少的假设下，一种策略是迭代类别。使用此解决方案，请注意 last 匹配仅会出现在找到多个类别的位置。

for tup in df2.itertuples(index=False):
    mask = df1['Receiver'].str.contains(tup.Searchterm, regex=False)
    df1.loc[mask, 'Category'] = tup.Category

print(df1)

#      Category           Receiver
# 0  Insurances  Insurance company
# 1   Groceries               Shop
# 2    Fastfood        Pizza place
# 3                        Library
# 4         Car   Gas station 24/7
# 5                 Something else
# 6              Whatever receiver

性能基准测试

如前所述，与df2 中的类别相比，此解决方案与df1 中的行相比更适合扩展。为了说明这一点，请考虑以下不同大小的输入数据帧的性能。

def jpp(df1, df2):
    for tup in df2.itertuples(index=False):
        df1.loc[df1['Receiver'].str.contains(tup.Searchterm, regex=False), 'Category'] = tup.Category
    return df1

def user347(df1, df2):
    df1['Category'] = df1['Receiver'].replace((df2['Searchterm'] + r'.*').values,
                                              df2['Category'].values,
                                              regex=True)
    df1.loc[df1['Receiver'].isin(df1['Category']), 'Category'] = ''
    return df1

df1 = pd.concat([df1]*10**4, ignore_index=True)
df2 = pd.concat([df2], ignore_index=True)

%timeit jpp(df1, df2)      # 145 ms per loop
%timeit user347(df1, df2)  # 364 ms per loop

df1 = pd.concat([df1], ignore_index=True)
df2 = pd.concat([df2]*100, ignore_index=True)

%timeit jpp(df1, df2)      # 666 ms per loop
%timeit user347(df1, df2)  # 88 ms per loop

Answer 2

您可以将Series.replace 与regex 一起用于矢量化方法：

df1['Category'] = df1['Receiver'].replace(
    (df2['Searchterm'] + r'.*').values,
    df2['Category'].values,
    regex=True
)

df1.loc[df1['Receiver'].isin(df1['Category']), 'Category'] = ''

print(df1)

     Category           Receiver
0  Insurances  Insurance company
1   Groceries               Shop
2    Fastfood        Pizza place
3                        Library
4         Car   Gas station 24/7
5                 Something else
6              Whatever receiver

请注意，这假定每个Searchterm 字符串都将在每个Receiver 字符串的开头找到。如果不是这样，请相应地调整正则表达式。

Answer 3

使用 str.extract 的另一种解决方案

pat = '('+'|'.join(df2['Searchterm'])+')'
df1["Category"] = df1['Receiver'].str.extract(pat)[0].map(df2.set_index('Searchterm')['Category'].to_dict()).fillna('')

    Receiver            Category
0   Insurance company   Insurances
1   Shop                Groceries
2   Pizza place         Fastfood
3   Library 
4   Gas station 24/7    Car
5   Something else  
6   Whatever receiver   

性能基准测试

def jpp(df1, df2):
    for tup in df2.itertuples(index=False):
        df1.loc[df1['Receiver'].str.contains(tup.Searchterm, regex=False), 'Category'] = tup.Category
    return df1

def user347(df1, df2):
    df1['Category'] = df1['Receiver'].replace((df2['Searchterm'] + r'.*').values,
                                              df2['Category'].values,
                                              regex=True)
    df1.loc[df1['Receiver'].isin(df1['Category']), 'Category'] = ''
    return df1

def vai(df1, df2):
    pat = '('+'|'.join(df2['Searchterm'])+')'
    df1["Category"] = df1['Receiver'].str.extract(pat)[0].map(df2.set_index('Searchterm')['Category'].to_dict()).fillna('')

df1 = pd.concat([df1]*10**4, ignore_index=True)
df2 = pd.concat([df2], ignore_index=True)

%timeit jpp(df1, df2)    
%timeit user347(df1, df2)
%timeit vai(df1, df2)


120 ms ± 2.26 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
221 ms ± 4.74 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
78.2 ms ± 1.56 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

df1 = pd.concat([df1], ignore_index=True)
df2 = pd.concat([df2]*100, ignore_index=True)

%timeit jpp(df1, df2)
%timeit user347(df1, df2)
%timeit vai(df1, df2)

11.4 s ± 276 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
20.4 s ± 296 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
98.3 ms ± 408 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)