AttributeError：未找到下限；从 Sklearn CountVectorizer 中删除不常见的功能？

Question

制作语料库和词汇

K = 10
XYtr['description'] = XYtr['description'].fillna("nan")
Xte['description'] = Xte['description'].fillna("nan")
corpus = list(XYtr['description'])+list(Xte['description'])
vectorizer = CountVectorizer()
corpus = vectorizer.fit_transform(corpus)
lda = LatentDirichletAllocation(n_components = K)
lda.fit(corpus)
#There are no problems until here

# Create a list of (term, frequency) tuples sorted by their frequency
sum_words = corpus.sum(axis=0) 
words_freq = [(word, sum_words[0, idx]) for word, idx in vectorizer.vocabulary_.items()]
words_freq = sorted(words_freq, key = lambda x: x[1])

# Keep only the terms in a list
vocabulary, _ = zip(*words_freq[:int(total_features * 0.2)])
vocabulary = list(vocabulary)

#Finally, we use the vocabulary to limit the model to the less frequent terms.
bottom_vect = CountVectorizer(vocabulary=vocabulary)
topics = bottom_vect.fit_transform(corpus)

这在代码的最后一行返回了“AttributeError: lower not found”。因此，我无法获得“主题”。

如果您能提出一些建议，我们将不胜感激。

这是我的数据集的几行

XYtr：

Xte：

Answer 1

您收到该错误是因为您使用 CountVectorizer() 的结果覆盖了 corpus 。举个例子：

corpus = ['This is the first document.',
'This document is the second document.',
'And this is the third one.',
'Is this the first document?']

将CountVectorizer() 的结果分配给另一个对象X：

vectorizer = CountVectorizer()
X = vectorizer.fit_transform(corpus)
lda = LatentDirichletAllocation(n_components = 2)
lda.fit(X)

sum_words = X.sum(axis=0) 
words_freq = [(word, sum_words[0, idx]) for word, idx in vectorizer.vocabulary_.items()]
words_freq = sorted(words_freq, key = lambda x: x[1])

total_features = len(words_freq)
vocabulary, _ = zip(*words_freq[:int(total_features * 0.2)])
vocabulary = list(vocabulary)

然后重新运行您的 CountVectorizer ：

bottom_vect = CountVectorizer(vocabulary=vocabulary)
topics = bottom_vect.fit_transform(corpus)