是否可以在此用例中创建 Pandas 矢量化？

Question

我正在尝试制作一个与下面的.apply 方法具有相同目的的 Pandas 矢量化脚本：

concised_df['email_account_letter_or_number_only'] = concised_df.apply(lambda x: x['email_account'] if (str(x['email_account']).isdigit() or not bool(re.search('[a-zA-Z]', str(x['email_account'])))) else (re.sub('[^A-Za-z]+', '', x['email_account'])), axis=1) 

代码的逻辑是这样的：如果email_account 是全数字或者不包含字母，那么只需将email_account 存储为email_account_letter_or_number_only。否则，执行re.sub 以仅保留email_account_letter_or_number_only 的字母（换句话说，删除所有数字和特殊字符）。

我尝试使 Pandas 矢量化的原因是为了使我的方法更加优化。如本博客https://towardsdatascience.com/apply-function-to-pandas-dataframe-rows-76df74165ee4（方法6.矢量化）所述，Pandas 矢量化比.apply 快得多。

这是我想要的输入和输出示例：

email_account	email_account_letter_or_number_only
0018889	0018889
nacho.taro	nachotaro
nachth45678	nachth
nacikita	nacikita
nacia_art	naciaart

我尝试过 Goole 搜索“带有 if else 条件的 Pandas 矢量化”，但到目前为止我能够找到的结果都引用了其他方法，例如 numpy.where 或 pd.DataFrame.loc (How to iterate a vectorized if/else statement over additional columns?) 而不是 Pandas 矢量化。

Answer 1

import io
import pandas as pd

df_str = '''
email_account   email_account_letter_or_number_only
0018889 0018889
nacho.taro  nachotaro
nachth45678 nachth
nacikita    nacikita
nacia_art   naciaart
'''
df = pd.read_csv(io.StringIO(df_str.strip()), sep='\s+', index_col=False)



# 1.replace the char that is not letter or number to ''
obj = df['email_account'].fillna('').str.replace('\W|_', '')
# 0        0018889
# 1      nachotaro
# 2    nachth45678
# 3       nacikita
# 4       naciaart
# Name: email_account, dtype: object



# 2.replace the digit to '' when the cell contains letters, regexp
cond = obj.str.contains('\D')
obj[cond] = obj[cond].str.replace('\d', '')
print(obj)

# 0      0018889
# 1    nachotaro
# 2       nachth
# 3     nacikita
# 4     naciaart
# Name: email_account, dtype: object


# result
df['tag'] = obj
df['email_account_letter_or_number_only'] == df['tag']
# 0    True
# 1    True
# 2    True
# 3    True
# 4    True
# dtype: bool

您还可以定义一个函数来处理数据，并映射或应用：

# def a function and apply, use pandas Series map
def data_clean(email):
    if pd.isna(email):
        return '' 
    else:
        email = re.sub('\W|_', '', email)
        if email.isdigit():
            pass
        else:
            email = re.sub('\d+', '', email)
        return email   


df['email_account'].map(data_clean) == df['tag']

我觉得这个方法比python原生for iteration更有效。

---

添加性能测试：

In [12]: import re
    ...: obj_r = df['email_account'].copy()
    ...: def fun1(obj_r):
    ...:     obj = obj_r.fillna('').str.replace('\W|_', '')
    ...:     cond = obj.str.contains('\D')
    ...:     obj[cond] = obj[cond].str.replace('\d', '')
    ...:     return obj
    ...:
    ...: def fun2(obj_r):
    ...:     obj = obj_r.map(data_clean)
    ...:     return obj
    ...:

In [13]: %timeit fun1(obj_r)
2.4 ms ± 197 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

In [14]: %timeit fun2(obj_r)
187 µs ± 7.56 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)