试图用 gensim 模仿 Scikit ngram答案

【问题标题】：Trying to mimick Scikit ngram with gensim试图用 gensim 模仿 Scikit ngram
【发布时间】：2017-10-10 15:24:54
【问题描述】：

我正在尝试使用 gensim 模仿 CountVectorizer() 中的 n_gram 参数。我的目标是能够将 LDA 与 Scikit 或 Gensim 一起使用，并找到非常相似的二元组。

例如，我们可以使用 scikit 找到以下二元组：“abc computer”、“binary unordered”以及使用 gensim “A survey”、“Graph minors”...

我在下面附上了我的代码，以便在双元组/单元组方面比较 Gensim 和 Scikit。

感谢您的帮助

documents = [["Human" ,"machine" ,"interface" ,"for" ,"lab", "abc" ,"computer" ,"applications"],
      ["A", "survey", "of", "user", "opinion", "of", "computer", "system", "response", "time"],
      ["The", "EPS", "user", "interface", "management", "system"],
      ["System", "and", "human", "system", "engineering", "testing", "of", "EPS"],
      ["Relation", "of", "user", "perceived", "response", "time", "to", "error", "measurement"],
      ["The", "generation", "of", "random", "binary", "unordered", "trees"],
      ["The", "intersection", "graph", "of", "paths", "in", "trees"],
      ["Graph", "minors", "IV", "Widths", "of", "trees", "and", "well", "quasi", "ordering"],
      ["Graph", "minors", "A", "survey"]]

使用 gensim 模型，我们找到 48 个唯一标记，我们可以使用 print(dictionary.token2id) 打印 unigram/bigrams

# 1. Gensim
from gensim.models import Phrases

# Add bigrams and trigrams to docs (only ones that appear 20 times or more).
bigram = Phrases(documents, min_count=1)
for idx in range(len(documents)):
    for token in bigram[documents[idx]]:
        if '_' in token:
            # Token is a bigram, add to document.
            documents[idx].append(token)

documents = [[doc.replace("_", " ") for doc in docs] for docs in documents]
print(documents)

dictionary = corpora.Dictionary(documents)
print(dictionary.token2id)

并且使用 scikit 96 唯一标记，我们可以使用 print(vocab) 打印 scikit 的词汇

# 2. Scikit
import re
token_pattern = re.compile(r"\b\w\w+\b", re.U)

def custom_tokenizer( s, min_term_length = 1 ):
    """
    Tokenizer to split text based on any whitespace, keeping only terms of at least a certain length which start with an alphabetic character.
    """
    return [x.lower() for x in token_pattern.findall(s) if (len(x) >= min_term_length and x[0].isalpha() ) ]

from sklearn.feature_extraction.text import CountVectorizer

def preprocess(docs, min_df = 1, min_term_length = 1, ngram_range = (1,1), tokenizer=custom_tokenizer ):
    """
    Preprocess a list containing text documents stored as strings.
    doc : list de string (pas tokenizé)
    """
    # Build the Vector Space Model, apply TF-IDF and normalize lines to unit length all in one call
    vec = CountVectorizer(lowercase=True,
                      strip_accents="unicode",
                      tokenizer=tokenizer,
                      min_df = min_df,
                      ngram_range = ngram_range,
                      stop_words = None
                     ) 
    X = vec.fit_transform(docs)
    vocab = vec.get_feature_names()

    return (X,vocab)

docs_join = list()

for i in documents:
    docs_join.append(' '.join(i))

(X, vocab) = preprocess(docs_join, ngram_range = (1,2))

print(vocab)

【问题讨论】：

标签： python scikit-learn gensim

【解决方案1】：

gensim Phrases 类旨在“从句子流中自动检测常用短语（多词表达）”。因此，它只会为您提供“比预期更频繁地出现”的二元组。这就是为什么使用 gensim 包你只能得到一些二元组，如：'response time'、'Graph minors'、'A survey'。

如果您查看bigram.vocab，您会发现这些二元组出现 2 次，而所有其他二元组只出现一次。

scikit-learn 的 CountVectorizer 类为您提供所有二元组。

【讨论】：