为什么 train_test_split 对回归结果的影响如此之大？

Question

我正在对 Kaggle 的房价预测竞赛进行一些测试。

为方便起见，请在下方找到下载、预处理和开始使用简单线性回归模型进行预测的完整过程：

下载数据

from kaggle.api.kaggle_api_extended import KaggleApi

api = KaggleApi()
api.authenticate()
saveDir = "data"
if not os.path.exists("data"):
    os.makedirs(saveDir)
api.competition_download_files("house-prices-advanced-regression-techniques","data")

print("the following files have been downloaded \n" + '\n'.join('{}'.format(item) for item in os.listdir("data")))
print("they are located in " + saveDir)

获取训练和测试数据

train = pd.read_csv(saveDir + r"\train.csv")
test = pd.read_csv(saveDir + r"\test.csv")

xTrain = train.iloc[:,1:-1] # remove id & SalePrice
yTrain = train.iloc[:,-1] # SalePrice
xTest = test.iloc[:,1:] # remove id

拆分数值和测试数据

catData = xTrain.columns[xTrain.dtypes == object]
numData = list(set(xTrain.columns) - set(catData))
print("The number of columns in the original dataframe is " + str(len(xTrain.columns)))
print("The number of columns in the categorical and numerical data dds up to " + str(len(catData)+len(numData)))

定义一个清理函数来处理 NaN / None

def cleanData(data, catData, numData) : 
dataClean = data.copy()

# Let's deal with NaN ...

# check where there are NaN in categorical
dataClean[catData].columns[dataClean[catData].isna().any(axis=0)]

# take care that some categorical could be numerics so
# differentiate the two cases
dataTypes = [dataClean.loc[dataClean.loc[:,col].notnull(),col].apply(type).iloc[0] for col in catData] # get the data type for each column
                                                                                                         # have to be carefull to not take a data that is NaN or None
                                                                                                         # when evaluating its type
from itertools import compress
catDataNum = [True if ((col == float) | (col == int)) else False for col in dataTypes ] # if data type is numeric (float/int), register it
catDataNum = list(compress(catData, catDataNum))
catDataNotNum = list(set(catData)-set(catDataNum))

print("The number of columns in the dataframe is " + str(len(dataClean.columns)))
print("The number of columns in the categorical and numerical data dds up to " + 
      str(len(catDataNum) + len(catDataNotNum)+len(numData)))

# Check what NA means for each feature ...
# BsmtQual : NA means no basement
# GarageType : NA means no garage
# BsmtExposure : NA means no basement
# Alley : NA means no alley access
# BsmtFinType2 : NA means no basement
# GarageFinish : NA means no garage
# did not check the rest ... I will just replace with a category "No"

# For categorical, NaN values mean the considered feature
# do not exist (this requires dataset analysis as performed above)
dataClean[catDataNotNum] = dataClean[catDataNotNum].fillna(value = 'No')
mean = dataClean[catDataNum].mean()
dataClean[catDataNum] = dataClean[catDataNum].fillna(value = mean)

# for numerical, replace with mean
mean = dataClean[numData].mean()
dataClean[numData] = dataClean[numData].fillna(value = mean)

return dataClean

进行清洁

xTrainClean = cleanData(xTrain, catData, numData)

# check if no NaN or None anymore
if xTrainClean.isna().sum().sum() != 0:
    print(xTrainClean.iloc[:,xTrainClean.isna().any(axis=0).values])
else :
    print("All good! No more NaN or None in training data!")

# same with test data
# perform the cleaning
xTestClean = cleanData(xTest, catData, numData)

# check if no NaN or None anymore
if xTestClean.isna().sum().sum() != 0:
    print(xTestClean.iloc[:,xTestClean.isna().any(axis=0).values])
else :
    print("All good! No more NaN or None in test data!")

预处理数据，即one-hot对分类特征进行编码

import sklearn as sk
import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder

# We would like to perform a linear regression on all data
# but some data are categorical ... 
# so first, perform a one-hot encoding on categorical variables
ct = ColumnTransformer(transformers = [("OneHotEncoder", OneHotEncoder(categories='auto', drop=None, 
                                      sparse=False, n_values='auto',
                                      handle_unknown = "error"),
                                      catData)],
                      remainder = "passthrough")
ct = ct.fit(pd.concat([xTrainClean, xTestClean])) # fit on both xTrain & xTest to be sure to have all possible categorical values
                                        # test it separately (.fit(xTrain) / .fit(xTest) and analyze to understand)
                                        # resulting categories and values can be obtained through
                                        # ct.named_transformers_ ['OneHotEncoder'].categories_
xTrainOneHot = ct.transform(xTrainClean)

将训练数据拆分为“内部”训练和测试集

xTestOneHotKaggle = xTestOneHot.copy()

from sklearn.model_selection import train_test_split
xTrainInternalOneHot, xTestInternalOneHot, yTrainInternal, yTestInternal = train_test_split(xTrainOneHot, yTrain, test_size=0.5, random_state=42, shuffle = False)

print("The training data now contains " + str(xTrainInternalOneHot.shape[0]) + " samples")
print("The training data now contains " + str(yTrainInternal.shape[0]) + " labels")
print("The test data now contains " + str(xTestInternalOneHot.shape[0]) + " samples")
print("The test data now contains " + str(yTestInternal.shape[0]) + " labels")

火车...这就是有趣的地方

reg = LinearRegression().fit(xTrainInternalOneHot,yTrainInternal)
yTrainInternalPredict = reg.predict(xTrainInternalOneHot)
yTestInternalPredict = reg.predict(xTestInternalOneHot)
print("The R2 score on training data is equal to " + str(reg.score(xTrainInternalOneHot,yTrainInternal)))
print("The R2 score on the internal test data is equal to " + str(reg.score(xTestInternalOneHot, yTestInternal)))

from sklearn.metrics import mean_squared_log_error
print("Tke Kaggle metric score (RMSLE) on internal training data is equal to " + 
      str(np.sqrt(mean_squared_log_error(yTrainInternal, yTrainInternalPredict))))
print("Tke Kaggle metric score (RMSLE) on internal test data is equal to " + 
      str(np.sqrt(mean_squared_log_error(yTestInternal, yTestInternalPredict))))

问题

因此，通过上述过程，在计算 Kaggle 指标（即 RMLSE）时会出现错误，因为有些值是负数。 f不寻常的事情是，如果我将 test_size 参数从 0.5 更改为 0.2，则不再有负值。人们可以理解，随着越来越多的数据被用于训练，因此模型表现得更好。但是如果我将它从 0.2 移动到 0.3（不那么剧烈的变化，即大约 100 个训练样本），那么模型预测负值的问题就会再次出现。

两个问题：

1. 这是预期的，即 模型对 训练数据 ?这更清楚，因为如果 test_size = 0.2 与 shuffle = False 一起使用然后它可以工作。如果在 shuffle 时使用 = 是的，然后模型开始预测负值。

2. 如何处理这种行为？显然，这是一个非常简单的 模型（没有标准化，没有缩放，没有正则化......）但我 相信真正了解正在发生的事情很有趣 这个非常简单的模型。

Answer 1

这是预期的，即模型对训练数据如此敏感吗？这更清楚，因为如果 test_size = 0.2 与 shuffle = False 一起使用，那么它可以工作。如果在 shuffle = True 时使用，则模型开始预测负值。

对于您的问题，是的，这种拆分很重要！

如何处理这种行为？显然，这是一个非常简单的模型（没有标准化、没有缩放、没有正则化......），但我相信真正理解这个非常简单的模型中发生的事情很有趣。

您听说过交叉验证吗？

https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.KFold.html

这个概念是用几个数据拆分来训练你的分类器/回归，它们总是有不同的训练/测试拆分来避免你正在解释的这种行为，然后你可以真正判断你的预测质量，因为新数据也可能有几个不同的结构。 所以你运行了几次迭代，然后判断结果。