如何在 Pandas 和 sklearn 中将预测值合并回原始 DataFrame

Question

首先提示将 sklearn 与 pandas 一起使用，如果这可能是一个基本问题，我们深表歉意。这是我的代码：

import pandas as pd
from sklearn.linear_model import LogisticRegression

X = df[predictors]
y = df['Plc']

X_train = X[:int(X.shape[0]*0.7)]
X_test = X[int(X.shape[0]*0.7):]
y_train = y[:int(X.shape[0]*0.7)]
y_test = y[int(X.shape[0]*0.7):]


model = LogisticRegression(max_iter=1000)
model.fit(X_train, y_train)
result = model.score(X_test, y_test)
print("Accuracy: %.3f%%" % (result*100.0))

现在我希望将预测值恢复为原始df，这样我就可以查看实际df['Plc'] 列与y_test 的预测值之间的差异。

我已经尝试过了，但感觉它 a) 可能不是最好的方法，并且 b) 索引号没有按预期排列。

y_pred = pd.DataFrame()
y_pred['preds'] = model.predict(X_test)
y_test = pd.DataFrame(y_test)
y_test['index1'] = y_test.index
y_test = y_test.reset_index()
y_test = pd.concat([y_test,y_pred],axis=1)
y_test.set_index('index1')
df = df.reset_index()
df_out = pd.merge(df,y_test,how = 'inner',left_index = True, right_index = True)

关于我应该做什么的任何想法？谢谢！

Answer 1

因为您的X_test 对应于X_test = X[int(X.shape[0]*0.7):]，这是您样本的最后 30%，您可以将预测结果添加到原始数据帧的较低 30% 部分：

Z=model.predict(X_test)
df.loc[int(X.shape[0]*0.7):,'predictions']=Z

这里我们在df 中有一个名为“预测”的新列。 如果您的数据框是一个示例：

df=pd.DataFrame({'predictor1':[0.1,0.3,0.3,0.3,0.5,0.9,0.02,0.8,0.8,0.75],
             'predictor2':[0.1,0.4,0.4,0.5,0.5,0.9,0.02,0.8,0.8,0.75],
        'Plc':np.array([0,1,1,1,1,1,1,0,1,1])})
predictor=['predictor1','predictor2']

它会给你结果：

   predictor1  predictor2  Plc  predictions
0        0.10        0.10    0          NaN
1        0.30        0.40    1          NaN
2        0.30        0.40    1          NaN
3        0.30        0.50    1          NaN
4        0.50        0.50    1          NaN
5        0.90        0.90    1          NaN
6        0.02        0.02    1          NaN
7        0.80        0.80    0          1.0
8        0.80        0.80    1          1.0
9        0.75        0.75    1          1.0

Z=[1,1,1] 被添加到最后 3 个样本中。

Answer 2

您可以“即时”定义df 中的preds 列，而无需创建其他数据框：

import pandas as pd
import numpy as np
from sklearn.linear_model import LogisticRegression

# Generate fake data
df = pd.DataFrame(np.random.rand(1000, 4),
                  columns = list('abcd'))
df['Plc'] = np.random.randint(0,2,1000)

# Split X and y
predictors = list('abcd')
X = df[predictors]
y = df['Plc']

# Split train and test
train_size = int(X.shape[0]*0.7)
X_train = X[:train_size]
X_test = X[train_size:]
y_train = y[:train_size]
y_test = y[train_size:]

# Train the model
model = LogisticRegression(max_iter=1000)
model.fit(X_train, y_train)

# Predict train and test
y_pred_train = model.predict(X_train)
y_pred_test = model.predict(X_test)

现在您至少有两个选择：

• 堆叠预测并根据堆叠数组创建列：

df['preds'] = np.hstack([y_pred_train, y_pred_test])

• 初始化列，然后分配预测：

df['preds'] = np.nan
df.loc[:train_size-1, 'pred'] = y_pred_train
df.loc[train_size:, 'pred'] = y_pred_test

它们产生相同的结果。

Answer 3

我相信您想要在这里将 X_test、y_test 和 y_pred 合并到同一个数据帧中（因为没有 X_train 没有用）。我认为很容易将 train_test_split 与 Pandas 一起使用来保留索引（尽管也有一种方法可以使用 numpy Scikit-learn train_test_split with indices）。我将在这里使用虹膜作为玩具数据，但你明白了。

from sklearn.datasets import load_iris
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
X, y = load_iris(return_X_y=True)
X = pd.DataFrame(X)
y = pd.Series(y)
### you can use shuffle = False instead of random if it's needed
X_train,X_test,y_train,y_test=train_test_split(X,y,test_size=0.3,random_state=42)

model = LogisticRegression(max_iter=1000)
model.fit(X_train, y_train)
df = X_test.copy()
df['Plc']= y_test
df.reset_index(inplace=True)
df['pred'] = model.predict(X_test)

## then print df, you can remove the index of the original df if you like

如果你真的想合并 X_train,y_train 并在 pred 列中有 NaN，你可以以相同的方式合并 X_train 和 y_train 并使用 pd.concat 制作单个数据帧

df2 = X_train.copy()
df2['Plc'] = y_train
df2.reset_index(inplace=True)
pd.concat([df,df2])

index   0   1   2   3   Plc pred
0   73  6.1 2.8 4.7 1.2 1   1.0
1   18  5.7 3.8 1.7 0.3 0   0.0
2   118 7.7 2.6 6.9 2.3 2   2.0
3   78  6.0 2.9 4.5 1.5 1   1.0
4   76  6.8 2.8 4.8 1.4 1   1.0
... ... ... ... ... ... ... ...
100 71  6.1 2.8 4.0 1.3 1   NaN
101 106 4.9 2.5 4.5 1.7 2   NaN
102 14  5.8 4.0 1.2 0.2 0   NaN
103 92  5.8 2.6 4.0 1.2 1   NaN
104 102 7.1 3.0 5.9 2.1 2   NaN
150 rows × 7 columns