在两个相似数据集上拟合的缩放器会产生不同的结果

Question

我正在使用来自以下链接的测试和训练数据集：

https://www.kaggle.com/emrearslan123/house-price-prediction/data

import pandas as pd
from sklearn.preprocessin import StandardScaler
import numpy as np


df = pd.read_csv('./train.csv')
test = pd.read_csv('./test.csv')


important_num_cols = list(df.corr()["SalePrice"][(df.corr()["SalePrice"]>0.50) |   
          (df.corr()["SalePrice"]<-0.50)].index)

cat_cols = ["MSZoning", "Utilities","BldgType","Heating","KitchenQual","SaleCondition","LandSlope"]

important_cols = important_num_cols + cat_cols


df = df[important_cols]
important_cols.remove('SalePrice')
test = test[important_cols]

#the following lines correspond to na values in the test dataset 
#while df has no null values 

test['TotalBsmtSF'] = test['TotalBsmtSF'].fillna(test['TotalBsmtSF'].mode()[0])
test['GarageCars'] = test['GarageCars'].fillna(test['GarageCars'].mode()[0])
test['GarageArea'] = test['GarageArea'].fillna(test['GarageArea'].mean())
test['MSZoning'] = test['MSZoning'].fillna(test['MSZoning'].mode()[0])
test['Utilities'] = test['Utilities'].fillna(test['Utilities'].mode()[0])
test['KitchenQual'] = test['KitchenQual'].fillna(test['KitchenQual'].mode()[0])



y = df['SalePrice']
X = df.drop('SalePrice', axis=1)

所以基本上我的训练数据集 X 与我的测试数据集具有相同数量的特征。但是，当我在两个数据集上应用一个热编码 pd.get_dummies 时，我的训练数据比我的测试数据多 3 个特征/列。

X = pd.get_dummies(X, columns=cat_cols)
test = pd.get_dummies(test, columns=cat_cols)

这怎么可能？

Answer 1

您的分类列在训练和测试集中有不同数量的类别，下面我们使用nunique() 来告诉使用每个分类列中唯一条目的数量：

import pandas as pd
train_df = pd.read_csv("train.csv")
test_df = pd.read_csv("test.csv")

cat_cols = ["MSZoning", "Utilities","BldgType","Heating","KitchenQual",
"SaleCondition","LandSlope"]

train_df[cat_cols].nunique()

MSZoning         5
Utilities        2
BldgType         5
Heating          6
KitchenQual      4
SaleCondition    6
LandSlope        3

test_df[cat_cols].nunique()
MSZoning         5
Utilities        1
BldgType         5
Heating          4
KitchenQual      4
SaleCondition    6
LandSlope        3

您可以做的是收集测试和训练的类别，然后将您的列分类，明确指定类别：

categories = pd.concat([train_df,test_df])[cat_cols].dropna().apply(lambda x:x.unique())
for cat in cat_cols:
    train_df[cat] = pd.Categorical(train_df[cat],categories=categories[cat])
    test_df[cat] = pd.Categorical(test_df[cat],categories=categories[cat])

还有：

pd.get_dummies(train_df[cat_cols]).shape
(1460, 31)

pd.get_dummies(test_df[cat_cols]).shape
(1459, 31)