statsmodels 的形状未对齐错误 GLM 预测与交叉验证

Question

我正在做一个问题，我在 statsmodels 中构建一个阶跃函数，同时首先使用交叉验证来确定理想的削减量。但是我遇到了一个我无法理解如何解决的问题。

使用 Sklearn 中的 KFold 函数添加交叉验证循环后，我开始收到错误：

ValueError: shapes (480,2) and (1,) not aligned: 2 (dim 1) != 1 (dim 0)

我不确定为什么现在会发生这种情况，因为在我开始使用交叉验证循环之前它工作得非常好，没有任何问题。

如果有人可以查看我的代码块并指出此问题的根源，我将不胜感激。

进去前X_train和y_train的形状：

X_train: (2400,)  y_train: (2400,)

代码：

import statsmodels.api as sm
from sklearn.model_selection import KFold

kf = KFold(n_splits=5,shuffle=True, random_state=1)

cuts = []
RMSE = []

for i in range(1,11):
  cuts.append(i)
  cross_val_rms = []
  for train_index, test_index in kf.split(X_train):
    train_x,test_x= X_train.iloc[train_index], X_train.iloc[test_index]
    train_y,test_y= y_train.iloc[train_index], y_train.iloc[test_index]
    
    df_cut, bins = pd.cut(train_x, i, retbins=True, right=True)
    df_steps = pd.concat([train_x, df_cut, train_y],
                        keys=['age','age_cuts','wage'], axis = 1)
    df_steps_dummies = pd.get_dummies(df_cut)
    GLM_fitted = sm.GLM(df_steps.wage, df_steps_dummies).fit()
    bin_mapping = np.digitize(test_x, bins)
    X_valid = pd.get_dummies(bin_mapping)
    pred = GLM_fitted.predict(X_valid)
    rms = np.sqrt(mean_squared_error(test_y, pred))
    cross_val_rms.append(rms)
  mean_rms = sum(cross_vall_rms)/len(cross_vall_rms)
  RMSE.append(mean_rms)

cuts_df = pd.DataFrame()
cuts_df['Cuts'] = cuts
cuts_df['RMSE'] = RMSE

print('Cuts with lowest Root Mean Squared Error:',cuts_df.loc[cuts_df['RMSE'].idxmin], sep='\n')

错误：

---------------------------------------------------------------------------

ValueError                                Traceback (most recent call last)

<ipython-input-166-a9794538c3e5> in <module>()
     21     bin_mapping = np.digitize(test_x, bins)
     22     X_valid = pd.get_dummies(bin_mapping)
---> 23     pred = GLM_fitted.predict(X_valid)
     24     rms = np.sqrt(mean_squared_error(test_y, pred))
     25     cross_val_rms.append(rms)

1 frames

/usr/local/lib/python3.7/dist-packages/statsmodels/genmod/generalized_linear_model.py in predict(self, params, exog, exposure, offset, linear)
    870             exog = self.exog
    871 
--> 872         linpred = np.dot(exog, params) + offset + exposure
    873         if linear:
    874             return linpred

<__array_function__ internals> in dot(*args, **kwargs)

ValueError: shapes (480,2) and (1,) not aligned: 2 (dim 1) != 1 (dim 0)

Answer 1

我认为，如果您解释一下您在回归中尝试做什么，这会有所帮助。您会收到错误，因为如果您从训练折叠中获得 3 个 bin，这并不意味着您从测试折叠中获得了 3 个 bin，您可能会获得 2 个折叠，因为 1 个 bin 中没有值。

据我所知，您可以先简单地将值离散化，然后使用示例数据进行训练：

import numpy as np
import pandas as pd
import statsmodels.api as sm
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import KFold


X_train = pd.Series(np.random.uniform(0,1,2400))
y_train = pd.Series(np.random.uniform(0,1,2400))

然后

for i in range(2,11):

  cross_val_rms = []
  df_steps_dummies = pd.get_dummies(pd.cut(X_train,i))
  
  for train_index, test_index in kf.split(X_train):
    train_x,test_x= df_steps_dummies.iloc[train_index,:], df_steps_dummies.iloc[test_index,:]
    train_y,test_y= y_train[train_index], y_train[test_index]
    
    GLM_fitted = sm.GLM(train_y, train_x).fit()
    pred = GLM_fitted.predict(test_x)
    rms = np.sqrt(mean_squared_error(test_y, pred))
    cross_val_rms.append(rms)

RMSE.append(np.array(cross_val_rms).mean())