Pandas 中的条件词频计数

Question

我有一个如下的数据框：

data = {'speaker':['Adam','Ben','Clair'],
        'speech': ['Thank you very much and good afternoon.',
                   'Let me clarify that because I want to make sure we have got everything right',
                   'By now you should have some good rest']}
df = pd.DataFrame(data)

我想计算语音列中的单词数，但只计算预定义列表中的单词。例如，列表是：

wordlist = ['much', 'good','right']

我想生成一个新列，显示这三个词在每一行中出现的频率。因此，我的预期输出是：

     speaker                   speech                               words
0   Adam          Thank you very much and good afternoon.             2
1   Ben        Let me clarify that because I want to make sur...      1
2   Clair        By now you should have received a copy of our ...    1

我试过了：

df['total'] = 0
for word in df['speech'].str.split():
    if word in wordlist: 
        df['total'] += 1

但我运行它后，total 列始终为零。我想知道我的代码有什么问题？

Answer 1

您可以使用以下矢量化方法：

data = {'speaker':['Adam','Ben','Clair'],
        'speech': ['Thank you very much and good afternoon.',
                   'Let me clarify that because I want to make sure we have got everything right',
                   'By now you should have some good rest']}
df = pd.DataFrame(data)

wordlist = ['much', 'good','right']

df['total'] = df['speech'].str.count(r'\b|\b'.join(wordlist))

这给出了：

>>> df
  speaker                                             speech  total
0    Adam            Thank you very much and good afternoon.      2
1     Ben  Let me clarify that because I want to make sur...      1
2   Clair              By now you should have some good rest      1

Answer 2

import pandas as pd

data = {'speaker': ['Adam', 'Ben', 'Clair'],
        'speech': ['Thank you very much and good afternoon.',
                   'Let me clarify that because I want to make sure we have got everything right',
                   'By now you should have some good rest']}
df = pd.DataFrame(data)

wordlist = ['much', 'good', 'right']

df["speech"] = df["speech"].str.split()
df = df.explode("speech")
counts = df[df.speech.isin(wordlist)].groupby("speaker").size()
print(counts)

Answer 3

如果您有一个非常大的列表和一个大数据框要搜索，这是一个更快（运行时）的解决方案。

我猜这是因为它利用了字典（需要 O(N) 来构建和 O(1) 来搜索）。性能方面，正则表达式搜索速度较慢。

import pandas as pd
from collections import Counter

def occurrence_counter(target_string, search_list):
    data = dict(Counter(target_string.split()))
    count = 0
    for key in search_list:
        if key in data:
            count+=data[key]
    return count

data = {'speaker':['Adam','Ben','Clair'],
        'speech': ['Thank you very much and good afternoon.',
                   'Let me clarify that because I want to make sure we have got everything right',
                   'By now you should have some good rest']}
df = pd.DataFrame(data)

wordlist = ['much', 'good','right']

df['speech'].apply(lambda x: occurrence_counter(x, wordlist))