计算匹配词并将计数分配给 Python 中的每个匹配词列

Question

我了解如何使用预定义的单词列表计算 pandas 列中匹配的单词并将计数分配到另一列（类似于this post here）。但我想知道是否有一种方法或函数可以将计数分配给按列样式匹配的单词列。

index | text
1     | "I have a pen and ipod, but I lost it today."
2     | "I have pineapple and pen, but I lost it today."

long_list = ['pen', 'pineapple', 'ipod']

index | text                                             | pen | pineapple | ipod |
1     | "I have a pen and ipod, but I lost it today."    | 1   |    0      |   1  |
2     | "I have pineapple and pen, but I lost it today." | 1   |    1      |   0  |

Answer 1

这是使用extract 和named capturing groups 的简洁解决方案：

regex = '|'.join(map(lambda i: f'(?P<{i}>{i})', long_list))
df.join(df['text'].str.extract(regex).notnull().astype(int))

输出：

index                                            text  pen  pineapple  ipod
    1     I have a pen and ipod, but I lost it today.    1          0     0
    2  I have pineapple and pen, but I lost it today.    0          1     0

如果单词包含无效字符，也可以使用未命名的捕获组（它们将编号为 0/1/2/3），然后重命名列：

long_list = ['pen', 'pineapple', 'ipod', 'cheese cake']
regex = '|'.join(map(lambda x: f'({x})', long_list))
df.join(df['text'].str.extract(regex)
                  .notnull().astype(int)
                  .rename(columns=dict(enumerate(long_list)))
        )

输出：

index              text  pen  pineapple  ipod  cheese cake
    1  I have a pen ...    1          0     0            0
    2  I have pineap...    0          1     0            0

它是如何工作的

extract 将为每个捕获组创建一个列，其中组名作为列名，单元格中匹配的字符串，否则为 NaN。然后我们使用notnull+astype(int)将这个输出转换为整数

正则表达式注释

注意。正则表达式的形式为'(?P<pen>pen)|(?P<pineapple>pineapple)|(?P<ipod>ipod)'

为了确保整个单词都匹配（即铅笔不应该与钢笔匹配），让我们添加单词边界 (\b)：

regex = '|'.join(map(lambda i: fr'(?P<{i}>\b{i}\b)', long_list))

给出：'(?P<pen>\\bpen\\b)|(?P<pineapple>\\bpineapple\\b)|(?P<ipod>\\bipod\\b)'

如果使用的单词包含空格（或在 python 变量中无效的字符），则应替换/删除这些单词：

regex = '|'.join(map(lambda i: fr'(?P<{i.replace(" ", "_")}>\b{i}\b)', long_list))

计数出现次数的变体

df.join(df['text']
          .str.extractall(regex)
          .notnull().astype(int)
          .groupby(level=0).sum()
       )

输出（我将输入修改为在第一行有两个“笔”）：

index                                               text  pen  pineapple  ipod
    1  I have a pen and another pen an ipod, but I lo...    2          0     1
    2     I have pineapple and pen, but I lost it today.    1          1     0

Answer 2

试试pd.get_dummies 和str.findall：

>>> df.join(pd.get_dummies(df['text'].str.findall(f'({"|".join(long_list)})').explode()).groupby(level=0).sum())
   index                                            text  ipod  pen  pineapple
0      1     I have a pen and ipod, but I lost it today.     1    1          0
1      2  I have pineapple and pen, but I lost it today.     0    1          1
>>> 

这不需要 for 循环。

Answer 3

您可以尝试使用str.contains

for i in long_list:
    df.loc[df.text.str.contains(i), i] = 1