如何遍历包含字符串列表的熊猫行以检查每个单词是否为英文？

Question

我有一个包含评论文本的 pandas 数据框。在文本预处理之后，我最终得到了每行中的字符串列表。现在我想遍历这些字符串列表的每一行来检查每个字符串是否是英文的。我想计算非英语单词的出现次数以创建另一列“出现次数”。

对于英语语言检查，我将使用 pyenchant 库。

类似于下面的代码

review_text sentiment   error_related
0   [simple, effective, way, new, word, kid]    1   NaN
1   [fh, fcfatgv]   1   NaN
2   [son, loved, easy, even, though, son, first, g...   1   NaN

english_dict = enchant.Dict("en_US")

def enlgish_counter(df, df_text_column):
    number_of_non_english_words = []
    for review in df_text_column:
        for word in review:
            a=0
        if english_dict.check(i)==False:
            a=a+1 
    non_english_words.append(a)

Answer 1

您没有包含示例数据，因此我手动构建了它。请注意，我的数据框格式可能与您的不同。

import pandas as pd
import enchant

english_dict = enchant.Dict("en_US")

# Construct the dataframe
words = ['up and vote', 'wet 0001f914 turtle 0001f602', 'thumbnailшщуй',
       'lobby', 'mods saffron deleted iâ', 'â', 'itâ donâ edit', 'thatâ',
       'didnâ canâ youâ'] 

df = pd.DataFrame()

for word in words:
    record = {'text': word}
    df = df.append(record, ignore_index=True)

# Get texts column
for text in df['text']:
    # Counters
    eng_words = 0
    non_eng_words = 0
    # For every word in text
    for word in text.split(' '):
        # Check if it is english
        if english_dict.check(word) == True:
            eng_words += 1
        else:
            non_eng_words += 1
    # Print the result
    # NOTE that these results are discarded each new text
    print('EN: {}; NON-EN: {}'.format(eng_words, non_eng_words))

---

如果你想修改你的数据集，你应该把这段代码包装成一个函数：

def create_occurences(df):
    eng_words_list = []
    non_eng_words_list = []
    for text in df['text']:
        eng_words = 0
        non_eng_words = 0
        for word in text.split(' '):
            if english_dict.check(word) == True:
                eng_words += 1
            else:
                non_eng_words += 1
        eng_words_list.append(eng_words)
        non_eng_words_list.append(non_eng_words)
    df['eng_words'] = pd.Series(eng_words_list, index=df.index)
    df['non_eng_words'] = pd.Series(non_eng_words_list, index=df.index)

create_occurences(df)
df