当我尝试将 tf-idf 应用于测试集时尺寸不匹配

Question

我正在尝试按照以下答案将新的预处理算法应用于我的数据集： Encoding text in ML classifier

我现在尝试的如下：

def test_tfidf(data, ngrams = 1):

    df_temp = data.copy(deep = True)
    df_temp = basic_preprocessing(df_temp)
    
    tfidf_vectorizer = TfidfVectorizer(ngram_range=(1, ngrams))
    tfidf_vectorizer.fit(df_temp['Text'])

    list_corpus = df_temp["Text"].tolist()
    list_labels = df_temp["Label"].tolist()

    X = tfidf_vectorizer.transform(list_corpus)
    
    return X, list_labels

（我建议参考我上面提到的所有代码的链接）。 当我尝试将后两个函数应用于我的数据集时：

train_x, train_y, count_vectorizer  = tfidf(undersample_train, ngrams = 1)
testing_set = pd.concat([X_test, y_test], axis=1)
test_x, test_y = test_tfidf(testing_set, ngrams = 1)

full_result = full_result.append(training_naive(train_x, test_x, train_y, test_y), ignore_index = True)

我收到此错误：

---> 12 full_result = full_result.append(training_naive(train_x, test_x, train_y, test_y, ), ignore_index = True) 
---> 14     y_pred = clf.predict(X_test_naive)

ValueError: dimension mismatch

错误中提到的函数是：

def training_naive(X_train_naive, X_test_naive, y_train_naive, y_test_naive, preproc):
    
    clf = MultinomialNB() 
    clf.fit(X_train_naive, y_train_naive)
    y_pred = clf.predict(X_test_naive)
        
    return 

任何有助于理解我的新定义和/或将 tf-idf 应用于我的数据集的任何帮助（请参阅此处以获取相关部分：Encoding text in ML classifier），我们将不胜感激。

更新：我认为这个问题/答案可能对帮助我解决问题也很有用：scikit-learn ValueError: dimension mismatch

如果我将test_x, test_y = test_tfidf(testing_set, ngrams = 1) 替换为test_x, test_y = test_tfidf(undersample_train, ngrams = 1)，它不会返回任何错误。但是，我认为这是不对的，因为我得到的值非常非常高（所有统计数据中的 99%）

Answer 1

当使用转换时（在这种情况下为TfidfVectorizer），您必须使用相同的对象来转换训练和测试数据。转换器通常仅使用训练数据进行拟合，然后重新用于转换测试数据。

在您的情况下执行此操作的正确方法：

def tfidf(data, ngrams = 1):

    df_temp = data.copy(deep = True)
    df_temp = basic_preprocessing(df_temp)
    
    tfidf_vectorizer = TfidfVectorizer(ngram_range=(1, ngrams))
    tfidf_vectorizer.fit(df_temp['Text'])

    list_corpus = df_temp["Text"].tolist()
    list_labels = df_temp["Label"].tolist()

    X = tfidf_vectorizer.transform(list_corpus)
    
    return X, list_labels, tfidf_vectorizer


def test_tfidf(data, vectorizer, ngrams = 1):

    df_temp = data.copy(deep = True)
    df_temp = basic_preprocessing(df_temp)

    # No need to create a new TfidfVectorizer here!

    list_corpus = df_temp["Text"].tolist()
    list_labels = df_temp["Label"].tolist()

    X = vectorizer.transform(list_corpus)
    
    return X, list_labels

# this method is copied from the other SO question
def training_naive(X_train_naive, X_test_naive, y_train_naive, y_test_naive, preproc):
    
    clf = MultinomialNB() # Gaussian Naive Bayes
    clf.fit(X_train_naive, y_train_naive)

    res = pd.DataFrame(columns = ['Preprocessing', 'Model', 'Precision', 'Recall', 'F1-score', 'Accuracy'])
    
    y_pred = clf.predict(X_test_naive)
    
    f1 = f1_score(y_pred, y_test_naive, average = 'weighted')
    pres = precision_score(y_pred, y_test_naive, average = 'weighted')
    rec = recall_score(y_pred, y_test_naive, average = 'weighted')
    acc = accuracy_score(y_pred, y_test_naive)
    
    res = res.append({'Preprocessing': preproc, 'Model': 'Naive Bayes', 'Precision': pres, 
                     'Recall': rec, 'F1-score': f1, 'Accuracy': acc}, ignore_index = True)

    return res 

train_x, train_y, count_vectorizer  = tfidf(undersample_train, ngrams = 1)
testing_set = pd.concat([X_test, y_test], axis=1)
test_x, test_y = test_tfidf(testing_set, count_vectorizer, ngrams = 1)

full_result = full_result.append(training_naive(train_x, test_x, train_y, test_y, count_vectorizer), ignore_index = True)