在python中计算没有停用词的tfidf矩阵答案

【问题标题】：calculate tfidf matrix without stop words in python在python中计算没有停用词的tfidf矩阵
【发布时间】：2021-01-03 15:47:54
【问题描述】：

我正在尝试计算没有停用词的tfidf 矩阵。这是我的代码：

def removeStopWords(documents):
    stop_words = set(stopwords.words('italian'))

    english_stop_words = set(stopwords.words('english'))

    stop_words.update(list(set(english_stop_words)))

    for d in documents:
        document = d['document']

        word_tokens = word_tokenize(document)

         filtered_sentence = ''

        for w in word_tokens:
            if not inStopwords(w, stop_words):
                 filtered_sentence = w + ' ' + filtered_sentence

        d['document'] = filtered_sentence[:-1]

    return calculateTFIDF(documents)


def calculateTFIDF(corpus):

    tfidf = TfidfVectorizer()
    x = tfidf.fit_transform(corpus)
    df_tfidf = pd.DataFrame(x.toarray(), columns=tfidf.get_feature_names())

    return {c: s[s > 0] for c, s in zip(df_tfidf, df_tfidf.T.values)}

但是当我返回矩阵时（格式为{word:value}），它还包含一些停用词，如when 或il。我该如何解决？谢谢

stopwords

【问题讨论】：

我用这个https://stackoverflow.com/questions/34612023/install-nltk-in-python-2-7-for-64-bit-machine在python 2.7上安装nltk，它可以工作

标签： python pandas python-2.7 numpy nltk

【解决方案1】：

有更好的方法可以去除 TF-IDF 计算的停用词。 TfidfVectorizer 有一个参数 stop_words，您可以在其中传递要排除的单词集合。

from nltk.corpus import stopwords
from sklearn.feature_extraction.text import TfidfVectorizer
import pandas as pd

documents = ['I went to the barbershop when my hair was long.', 'The barbershop was closed.']

# create set of stopwords to remove
stop_words = set(stopwords.words('italian'))
english_stop_words = set(stopwords.words('english'))
stop_words.update(english_stop_words)

# check if word in stop words
print('when' in stop_words)  # True
print('il' in stop_words)  # True

# else add word to the set
print('went' in stop_words)  # False
stop_words.add('went')

# create tf-idf from original documents
tfidf = TfidfVectorizer(stop_words=stop_words)
x = tfidf.fit_transform(documents)
df_tfidf = pd.DataFrame(x.toarray(), columns=tfidf.get_feature_names())

print({c: s[s > 0] for c, s in zip(df_tfidf, df_tfidf.T.values)})
# {'barbershop': array([0.44943642, 0.57973867]), 'closed': array([0.81480247]), 'hair': array([0.6316672]), 'long': array([0.6316672])}

【讨论】：