cross_val_score 和 StratifiedKFold 给出不同的结果

Question

这是StratifiedKFold 的代码，带有循环

kfold = StratifiedKFold(n_splits=5, shuffle=True, random_state=2020)
for train_idx, val_idx in kfold.split(train,labels):
  x_train,y_train=train[train_idx],labels[train_idx]
  x_val,y_val=train[val_idx],labels[val_idx]

  count_vectorizer = CountVectorizer()
  count_vectorizer.fit(x_train)
  X_train_cv = count_vectorizer.transform(x_train)
  X_val_cv  = count_vectorizer.transform(x_val)

  cv_classifier = LogisticRegression(solver='lbfgs',C=25,max_iter=500)
  cv_classifier.fit(X_train_cv, y_train)
  y_pred = cv_classifier.predict(X_val_cv)
  f1=f1_score(y_val, y_pred,average='macro')
  print(f1)

我得到的结果是

0.49
0.46
0.48
0.48
0.50

遵循cross_val_score 代码

from sklearn.model_selection import cross_val_score
cv_classifier = LogisticRegression(solver='lbfgs',C=25,max_iter=500,class_weight='balance')
count_vectorizer = CountVectorizer()
count_vectorizer.fit(train)
train_cv = count_vectorizer.transform(train)
print(cross_val_score(cv_classifier,train_cv, labels, cv=StratifiedKFold(5,shuffle = True),scoring='f1_macro'))

我得到的结果是

0.70 0.74 0.70 0.734 0.679

EIDT 我加了pipeline

cv_classifier = LogisticRegression(solver='lbfgs',C=25,max_iter=500,class_weight='balance')
classifier_pipeline = make_pipeline(CountVectorizer(), cv_classifier)

print(cross_val_score(classifier_pipeline,train, labels, cv=StratifiedKFold(5,shuffle = True),scoring='f1_macro'))

Answer 1

第二种情况下结果更好的原因是train_cv数据集已经被count_vectorizer拟合和转换了。

在前一种情况下，在每个 CV 折叠中，您将矢量化器拟合到训练数据上并转换验证数据。这是正确的方法，因为向量化器在拟合期间看不到验证数据。

为了对cross_val_score() 执行相同的操作，您应该创建一个包含矢量化器和逻辑回归模型的pipeline。然后，将此管道传递给cross_val_score()，而数据应该是初始train 数据集（不是train_cv 数据集）。

Answer 2

您还需要在cv=StratifiedKFold(5,shuffle = True) 此处设置随机种子，或者为cross_val_score 提供相同的kfold。

我为您的工作流程创建了一个玩具示例。

from sklearn.pipeline import make_pipeline
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.datasets import fetch_20newsgroups
from sklearn.metrics import f1_score
import numpy as np

categories = ['alt.atheism', 'talk.religion.misc']
newsgroups_train = fetch_20newsgroups(subset='train',
                                      categories=categories)

from sklearn.linear_model import LogisticRegression

train, labels = np.array(newsgroups_train.data), newsgroups_train.target


kfold = StratifiedKFold(n_splits=5, shuffle=True, random_state=2020)
for train_idx, val_idx in kfold.split(train, labels):
    x_train, y_train = train[train_idx], labels[train_idx]
    x_val, y_val = train[val_idx], labels[val_idx]

    count_vectorizer = CountVectorizer()
    count_vectorizer.fit(x_train)
    X_train_cv = count_vectorizer.transform(x_train)
    X_val_cv = count_vectorizer.transform(x_val)

    cv_classifier = LogisticRegression(solver='lbfgs', C=25, max_iter=500)
    cv_classifier.fit(X_train_cv, y_train)
    y_pred = cv_classifier.predict(X_val_cv)
    f1 = f1_score(y_val, y_pred, average='macro')
    print(f1)

cv_classifier = LogisticRegression(solver='lbfgs',C=25,max_iter=500,class_weight='balance') 
classifier_pipeline = make_pipeline(CountVectorizer(), cv_classifier) 
print(cross_val_score(classifier_pipeline,train, labels, cv=kfold,scoring='f1_macro'))

输出：

0.9059466848940533
0.9527147766323024
0.9174937965260546
0.9336297237218165
0.9526315789473685
[0.90594668 0.95271478 0.9174938  0.93362972 0.95263158]