ValueError：发现样本数量不一致的数组：[4 16149]

Question

您好，我是 scikit 学习和数据科学的新手。我在尝试从矢量化器中检索信息最多的功能时遇到了上述问题。我的代码（经过编辑以反映@Gang 的评论）：

values = dataset.data
word_vectorizer = CountVectorizer(analyzer='word', stop_words=custom_stop_words)
trainset = word_vectorizer.fit_transform(values)
tags = ['dem','rep','dem','rep']
tags = np.array(tags)
trainset = trainset.toarray()

word_svm = svm.LinearSVC()
word_svm.fit(trainset, tags)


def most_informative_feature_for_binary_classification(vectorizer, classifier, n=10):
class_labels = classifier.classes_
feature_names = vectorizer.get_feature_names()
topn_class1 = sorted(zip(classifier.coef_[0], feature_names))[:n]
topn_class2 = sorted(zip(classifier.coef_[0], feature_names))[-n:]

for coef, feat in topn_class1:
    print class_labels[0], coef, feat

print

for coef, feat in reversed(topn_class2):
    print class_labels[1], coef, feat


most_informative_feature_for_binary_classification(word_vectorizer, word_svm)

终端输出：

Traceback (most recent call last):
File "classification.py", line 251, in <module>
word_svm.fit(trainset, tags)
File "/usr/local/lib/python2.7/site-packages/sklearn/svm/classes.py", line 205, in fit
dtype=np.float64, order="C")
File "/usr/local/lib/python2.7/site-packages/sklearn/utils/validation.py", line 520, in check_X_y
check_consistent_length(X, y)
File "/usr/local/lib/python2.7/site-packages/sklearn/utils/validation.py", line 176, in check_consistent_length
"%s" % str(uniques))
ValueError: Found arrays with inconsistent numbers of samples: [    4 16149]

对于此事，我将不胜感激。如果我没有提供足够的信息，请告诉我。提前感谢您的宝贵时间！

Answer 1

这里是失败的地方——两个参数应该是相同的类型——数组

word_svm.fit(trainset, tags)

tags 不是数组，应该转换成数组

tags = ['dem','rep','dem','rep']

你可以使用print来查看它们是否是同一个类型

print type(tags)
print type(trainset)

下面的代码是用文本编辑器写的，没有运行，不保证可以工作，但是你明白了，我转换为数组可能是错误的，List很好。

您的 trainset 正确包含无效数据，请替换

word_svm.fit(trainset, tags)

用这个：

trainset_good, trainset_bad = trainset_check(trainset, tags)
print 'Bad data\n'
print trainset_bad
if len(trainset_good)==0:
   print 'No good valid data found, exit'
   sys.exit(1)

# use good data
word_svm.fit(trainset_good, tags)

将此函数添加到代码中

def trainset_check(trainset, tags):
    trainset_good = []
    trainset_bad = []
    if not trainset:
        print 'Err trainset is empty'
        return trainset_good, trainset_bad
    if not tags:
        print 'Err - tags empty'
        return trainset_good, trainset_bad
    if len(trainset)==0:
        print 'Err trainset is empty'
        return trainset_good, trainset_bad
    if len(tags)==0:
        print 'Err tags empty'
        return trainset_good, trainset_bad
    for item in trainset:
        if len(item) != len(tags):
            print 'Error - trainset item is not the same length as tags'
            print item
            trainset_bad.append(item)
            # skip to next
            continue
        # filter out None type
        item_new = filter(None, item)
        if len(item_new) != len(tags):
            print 'Error - trainset item is not the same length as tags'
            # bad trainset data, skip to next
            print item
            trainset_bad.append(item)
            continue
         trainset_good.append(item)
    return trainset_good, trainset_bad