nltk：如何获取包含特定单词的二元组

Question

我是 nltk 的新手，想获取特定单词（例如“man”）的搭配，以便稍后按频率过滤它们并按 PMI 分数对其进行排序。

这是我检索包含“man”的二元组的试用代码，但它返回一个空列表：

>>> text = "hello, yesterday I have seen a man walking. On the other side there was another man yelling \"who are you, man?\""
>>> tokens = word_tokenize(text)
>>> finder = BigramCollocationFinder.from_words(tokens, window_size=5)
>>> filter_man = lambda w: "man" not in w
>>> finder.apply_word_filter(filter_man)
>>> finder.ngram_fd.items()
[(('have', 'seen'), 1), ((',', 'yesterday'), 1), (('on', 'the'), 1), (('I', 'have'), 1), (('of', 'another'), 1), (('walking', 'on'), 1), (('seen', 'a'), 1), (('hello', ','), 1), (('man', 'walking'), 1), (('side', 'of'), 1), (('the', 'opposite'), 1), (('a', 'man'), 1), (('opposite', 'side'), 1), (('another', 'man'), 1), (('yesterday', 'I'), 1)]
>>> finder.ngram_fd.items()
[]
>>> 

我做错了什么？

Answer 1

finder = BigramCollocationFinder.from_words(text.split())
word_filter = lambda w1, w2: "man" not in (w1, w2)
finder.apply_ngram_filter(word_filter)

bigram_measures = nltk.collocations.BigramAssocMeasures()
raw_freq_ranking = finder.nbest(bigram_measures.raw_freq, 10) #top-10