从三元组列表生成一元组和二元组

Question

我正在研究将三元组频率存储在内存中并通过以下方式即时计算一元组和二元组频率的潜在方法：

给定一个三元组 u , v , w ：

count(v, w) = sum (.,v,w) 即所有 u 的总和

同样，count(w) = sum(.,w)

这确实会导致缺少一些单字母组合，例如句子开始标记，但这听起来像是生成单字母组合和双字母组合的有效方法吗？

Answer 1

是的。那可行。您可以通过自己制作一个小型语料库并手动进行计数以确保结果相同来检查它。

from collections import Counter

corpus = [['the','dog','walks'], ['the','dog','runs'], ['the','cat','runs']]
corpus_with_ends = [['<s>','<s>'] + s + ['<e>'] for s in corpus]

trigram_counts = Counter(trigram for s in corpus_with_ends for trigram in zip(s,s[1:],s[2:]))

unique_bigrams = set((b,c) for a,b,c in trigram_counts)
bigram_counts = dict((bigram,sum(count for trigram,count in trigram_counts.iteritems() if trigram[1:] == bigram)) for bigram in unique_bigrams)

unique_unigrams = set((c,) for a,b,c in trigram_counts if c != '<e>')
unigram_counts = dict((unigram,sum(count for trigram,count in trigram_counts.iteritems() if trigram[2:] == unigram)) for unigram in unique_unigrams)

现在你可以检查一下了：

>>> true_bigrams = [bigram for s in corpus_with_ends for bigram in zip(s[1:],s[2:])]
>>> true_bigram_counts = Counter(true_bigrams)
>>> bigram_counts == true_bigram_counts
True

>>> true_unigrams = [(unigram,) for s in corpus_with_ends for unigram in s[2:-1]]
>>> true_unigram_counts = Counter(true_unigrams)
>>> unigram_counts == true_unigram_counts
True