bigrams python的CountVectorizer词汇规范

Question

我正在尝试获取大量（~160.000）个文档的术语计数的稀疏矩阵。

我清理了文本并希望循环遍历所有文档（即，一次计数向量化一个并附加生成的 1xN 数组。以下代码适用于逐字的情况，但不适用于二元组：

cv1 = sklearn.feature_extraction.text.CountVectorizer(stop_words=None,vocabulary=dictionary1)
cv2 = sklearn.feature_extraction.text.CountVectorizer(stop_words=None,vocabulary=dictionary2)

for row in range(start,end+1):
    report_name = fund_reports_table.loc[row, "report_names"]
    raw_report = open("F:/EDGAR_ShareholderReports/" + report_name, 'r', encoding="utf8").read()

    ## word for word
    temp = cv1.fit_transform([raw_report]).toarray()
    res1 = np.concatenate((res1,temp),axis=0)

    ## big grams
    bigram=set()
    sentences = raw_report.split(".")
    for line in sentences:
        token = nltk.word_tokenize(line)
        bigram = bigram.union(set(list(ngrams(token, 2)))  )

    temp = cv2.fit_transform(list(bigram)).toarray()
    res2=np.concatenate((res2,temp),axis=0)

Python 返回

"AttributeError: 'tuple' object has no attribute 'lower'" 

大概是因为我将数据输入二元向量化计数器的方式无效。

“raw_report”是一个字符串。单字词典是：

dictionary1 =['word1', 'words2',...]

dictionary2 类似，但基于通过合并所有文档的所有二元组构建的二元组（并保持唯一值，在前面完成），因此生成的结构是

dictionary2 =[('word1','word2'),('wordn','wordm'),...]

文档二元组具有相同的结构，这就是为什么我很困惑为什么 python 不接受输入。有没有办法解决这个问题，或者我的整个方法不是很pythonic并且开始适得其反？

提前感谢您的帮助！

备注：我知道我可以在更精细的 CountVectorize 命令中完成整个过程（即清理和标记化以及一步计数），但我更希望自己也能够做到这一点（以便查看和存储中间输出）。此外，鉴于我使用的大量文本，我担心我会遇到内存问题。

Answer 1

您的问题来自您的 dictionary2 基于元组这一事实。这是一个极简主义的例子，它表明当二元组是字符串时这很有效。如果要单独处理每个文件，可以将其作为列表传递给 vectorizer.transform()。

from sklearn.feature_extraction.text import CountVectorizer

Doc1 = 'Wimbledon is one of the four Grand Slam tennis tournaments, the others being the Australian Open, the French Open and the US Open.'
Doc2 = 'Since the Australian Open shifted to hardcourt in 1988, Wimbledon is the only major still played on grass'
doc_set = [Doc1, Doc2]

my_vocabulary= ['Grand Slam', 'Australian Open', 'French Open', 'US Open']

vectorizer = CountVectorizer(ngram_range=(2, 2))
vectorizer.fit_transform(my_vocabulary)
term_count = vectorizer.transform(doc_set)

# Show the index key for each bigram
vectorizer.vocabulary_
Out[11]: {'grand slam': 2, 'australian open': 0, 'french open': 1, 'us open': 3}

# Sparse matrix of bigram counts - each row corresponds to a document
term_count.toarray()
Out[12]: 
array([[1, 1, 1, 1],
       [1, 0, 0, 0]], dtype=int64)

您可以使用列表推导来修改您的字典2。

dictionary2 = [('Grand', 'Slam'), ('Australian', 'Open'), ('French', 'Open'), ('US', 'Open')]
dictionary2 = [' '.join(tup) for tup in dictionary2]

dictionary2
Out[26]: ['Grand Slam', 'Australian Open', 'French Open', 'US Open']

编辑：基于以上我认为您可以使用以下代码：

from sklearn.feature_extraction.text import CountVectorizer

# Modify dictionary2 to be compatible with CountVectorizer
dictionary2_cv = [' '.join(tup) for tup in dictionary2]

# Initialize and train CountVectorizer
cv2 = CountVectorizer(ngram_range=(2, 2))
cv2.fit_transform(dictionary2_cv)

for row in range(start,end+1):
    report_name = fund_reports_table.loc[row, "report_names"]
    raw_report = open("F:/EDGAR_ShareholderReports/" + report_name, 'r', encoding="utf8").read()

    ## word for word
    temp = cv1.fit_transform([raw_report]).toarray()
    res1 = np.concatenate((res1,temp),axis=0)

    ## big grams
    bigram=set()
    sentences = raw_report.split(".")
    for line in sentences:
        token = nltk.word_tokenize(line)
        bigram = bigram.union(set(list(ngrams(token, 2)))  )

    # Modify bigram to be compatible with CountVectorizer
    bigram = [' '.join(tup) for tup in bigram]

    # Note you must not fit_transform here - only transform using the trained cv2
    temp = cv2.transform(list(bigram)).toarray()
    res2=np.concatenate((res2,temp),axis=0)