将 Sklearn CountVectorizer 词汇设置为短语字典

Question

您好，我一直在尝试使用 scikit-learn 进行文本分析，并且想到了使用 CountVectorizer 来检测文档是否包含一组关键字和短语。

我知道我们可以这样做：

words = ['cat', 'dog', 'walking']
example = ['I was walking my dog and cat in the park']
vect = CountVectorizer(vocabulary=words)
dtm = vect.fit_transform(example)
>>> pd.DataFrame(dtm.toarray(), columns=vect.get_feature_names()) 

...

   cat  dog  walking
    1    1        1

我想知道是否可以调整一些东西，以便我可以使用单词短语而不是单个单词

从上面的例子：

phrases = ['cat in the park', 'walking my dog']
example = ['I was walking my dog and cat in the park']
vect = CountVectorizer(vocabulary=phrases)
dtm = vect.fit_transform(example)
>>> pd.DataFrame(dtm.toarray(), columns=vect.get_feature_names()) 
... 

       cat in the park   walking my dog
            1                   1

现在使用短语的代码只是输出

cat in the park   walking my dog
     0                   0

提前谢谢你！

Answer 1

试试这个：

In [104]: lens = [len(x.split()) for x in phrases]

In [105]: mn, mx = min(lens), max(lens)

In [106]: vect = CountVectorizer(vocabulary=phrases, ngram_range=(mn, mx))

In [107]: dtm = vect.fit_transform(example)

In [108]: pd.DataFrame(dtm.toarray(), columns=vect.get_feature_names())
Out[108]:
   cat in the park  walking my dog
0                1               1

In [109]: print(mn, mx)
3 4