NLTK 停用词删除问题答案

【问题标题】：NLTK stopword removal issueNLTK 停用词删除问题
【发布时间】：2013-12-23 09:10:37
【问题描述】：

我正在尝试创建 document classification, as described in NLTK Chapter 6，但在删除停用词时遇到了问题。当我添加

all_words = (w for w in all_words if w not in nltk.corpus.stopwords.words('english'))

Traceback (most recent call last):
  File "fiction.py", line 8, in <module>
    word_features = all_words.keys()[:100]
AttributeError: 'generator' object has no attribute 'keys'

我猜测停用词代码改变了用于“all_words”的对象类型，导致它们的 .key() 函数无用。如何在使用键功能之前删除停用词而不更改其类型？完整代码如下：

import nltk 
from nltk.corpus import PlaintextCorpusReader

corpus_root = './nltk_data/corpora/fiction'
fiction = PlaintextCorpusReader(corpus_root, '.*')
all_words=nltk.FreqDist(w.lower() for w in fiction.words())
all_words = (w for w in all_words if w not in nltk.corpus.stopwords.words('english'))
word_features = all_words.keys()[:100]

def document_features(document): # [_document-classify-extractor]
    document_words = set(document) # [_document-classify-set]
    features = {}
    for word in word_features:
        features['contains(%s)' % word] = (word in document_words)
    return features

print document_features(fiction.words('fic/11.txt'))

【问题讨论】：

标签： python nltk

【解决方案1】：

我会首先避免将它们添加到FreqDist 实例中：

all_words=nltk.FreqDist(w.lower() for w in fiction.words() if w.lower() not in nltk.corpus.stopwords.words('english'))

根据您的语料库的大小，我认为在执行此操作之前为停用词创建一组可能会提高性能：

stopword_set = frozenset(ntlk.corpus.stopwords.words('english'))

如果这不适合您的情况，看起来您可以利用 FreqDist 继承自 dict 的事实：

for stopword in nltk.corpus.stopwords.words('english'):
    if stopword in all_words:
        del all_words[stopword]

【讨论】：