Scikit学习朴素贝叶斯ValueError：维度不匹配答案

【问题标题】：Scikit learn Naive Bayes ValueError: dimension mismatchScikit学习朴素贝叶斯ValueError：维度不匹配
【发布时间】：2015-10-28 10:09:22
【问题描述】：

我正在 Scikit-learn 中研究朴素贝叶斯分类器。

在训练和预测阶段我都使用以下代码从元组列表中获取 csr_matrix：

def convert_to_csr_matrix(vectors):
    """
    convert list of tuples representation to scipy csr_matrix that is needed
    for scikit learner
    """
    logger.info("building the csr_sparse matrix representing tf-idf")
    row = [[i] * len(v) for i, v in enumerate(vectors)]
    row = list(chain(*row))
    column = [j for j, _ in chain(*vectors)]
    data = [d for _, d in chain(*vectors)]
    return csr_matrix((data, (row, column)))

我主要基于scipy csr_matrix from several vectors represented as list of sets实现的

不幸的是，现在在预测阶段我收到以下错误：

File "/Users/zikes/project/taxonomy_data_preprocessing/single_classification.py", line 93, in predict
top_predictions = self.top.predict(item)
File "/Users/zikes/project/taxonomy_data_preprocessing/single_classification.py", line 124, in predict
category, res = model.predict(item)
File "/Users/zikes/project/taxonomy_data_preprocessing/single_classification.py", line 176, in predict
prediction = self.clf.predict(item)
File "/Users/zikes/.virtualenvs/taxonomy/lib/python2.7/site-packages/sklearn/naive_bayes.py", line 64, in predict
jll = self._joint_log_likelihood(X)
File "/Users/zikes/.virtualenvs/taxonomy/lib/python2.7/site-packages/sklearn/naive_bayes.py", line 615, in _joint_log_likelihood
return (safe_sparse_dot(X, self.feature_log_prob_.T)
File "/Users/zikes/.virtualenvs/taxonomy/lib/python2.7/site-packages/sklearn/utils/extmath.py", line 178, in safe_sparse_dot
ret = a * b
File "/Users/zikes/.virtualenvs/taxonomy/lib/python2.7/site-packages/scipy/sparse/base.py", line 354, in __mul__
raise ValueError('dimension mismatch')
ValueError: dimension mismatch

有没有人知道可能出了什么问题？我猜想某种稀疏向量的维度是错误的。但我不明白为什么？

在调试过程中，我在朴素贝叶斯模型中提到的feature_log_prob_ 的日志中打印出来，它看起来像：

[[-11.82052115 -12.51735721 -12.51735721 ..., -12.51735721 -11.60489688
-12.2132116 ]
[-12.21403023 -12.51130295 -12.51130295 ..., -11.84156341 -12.51130295
-12.51130295]]

还有shape：(2, 53961)

我的预测csr_matrix = (0, 7637) 0.770238101052 (0, 21849) 0.637756432886

并表示为元组列表，它看起来像：[(7637, 0.7702381010520318), (21849, 0.6377564328862234)]

【问题讨论】：

标签： python numpy scipy scikit-learn

【解决方案1】：

所以在对问题进行了一些调查后，我意识到可能的修复方法可能是：

def convert_to_csr_matrix(vectors):
   """
   convert list of tuples representation to scipy csr_matrix that is needed
   for scikit learner
   """
   logger.info("building the csr_sparse matrix representing tf-idf")
   row = [[i] * len(v) for i, v in enumerate(vectors)]
   row = list(chain(*row))
   column = [j for j, _ in chain(*vectors)]
   data = [d for _, d in chain(*vectors)]
   return csr_matrix((data, (row, column)))

return csr_matrix((data, (row, column))) 行应替换为 return csr_matrix((data, (row, column)), shape=(len(vectors), dimension))

【讨论】：