keras：评估多类 CNN 的 ROC AUC

Question

我正在使用kerasSequential() API 为 5 类问题构建我的 CNN 模型。由于准确度不是多类问题的好指标，我必须评估其他指标来评估我的模型。目前，我使用sklearn的confusion_matrix和classification_report，但我想研究更多指标，所以我决定评估ROC AUC，但我不确定这是如何使用keras完成的，有哪些修改我应该对我的代码等做些什么。

目前，这是我构建模型的方式：

model = Sequential()
activ = 'relu'
model.add(Conv2D(32, (1, 3), strides=(1, 1), padding='same', activation=activ, input_shape=(1, 100, 4)))
model.add(Conv2D(32, (1, 3), strides=(1, 1), padding='same', activation=activ ))
model.add(MaxPooling2D(pool_size=(1, 2) ))

model.add(Conv2D(64, (1, 3), strides=(1, 1), padding='same', activation=activ))
model.add(Conv2D(64, (1, 3), strides=(1, 1), padding='same', activation=activ))
model.add(MaxPooling2D(pool_size=(1, 2)))

model.add(Flatten())
A = model.output_shape
model.add(Dense(int(A[1] * 1/4.), activation=activ)) 

model.add(Dense(5, activation='softmax'))

model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])

model.fit(x_train, y_train, epochs=5, batch_size=64, shuffle=True, callbacks=callbacks,
                          validation_split=0.2)

测试评估：

Pred = model.predict(x_test, batch_size=32)
Pred_Label = np.argmax(Pred, axis=1)

test_acc = accuracy_score(y_test, Pred_Label)
ConfusionM = confusion_matrix(list(y_test), Pred_Label, labels=[0, 1, 2, 3, 4])
class_report = classification_report(list(y_test), Pred_Label, labels=[0, 1, 2, 3, 4])

要得到结果，如下所示：

Confusion Matrix:   
[[ 2514  1040  2584  6690  1773]
 [  208   359    37   668   126]
 [ 1445  1156  1172  3438  1106]
 [ 3158  2014  2993 10185  1951]
 [  154    77    29   493   151]]

Classification Report: 
              precision    recall  f1-score   support

     Class:0       0.34      0.17      0.23     14601
     Class:1       0.08      0.26      0.12      1398
     Class:2       0.17      0.14      0.15      8317
     Class:3       0.47      0.50      0.49     20301
     Class:4       0.03      0.17      0.05       904

    accuracy                           0.32     45521
   macro avg       0.22      0.25      0.21     45521
weighted avg       0.35      0.32      0.32     45521

如何将 ROC AUC 添加到我的模型指标中？

Answer 1

您可以在编译模型之前添加以下代码。您可以使用keras library metrics 探索各种指标。

METRICS = [keras.metrics.AUC(name='auc')]
model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=METRICS)

此外，您可以在使用混淆矩阵和sklearn library function 的这些类似行中设计一个函数，并绘制结果。

def plot_roc(Pred, Pred_Label):
  fp, tp, _ = sklearn.metrics.roc_curve(labels, predictions).

你一定要看看这个out

干杯。

Answer 2

绘制多类分类器的ROC曲线的另一种方法如下所示。让我们来看一个玩具问题，CIFAR10，一个多类数据集，由 10 个不同的类组成。

import tensorflow as tf
import numpy as np 

(x_train, y_train), (_, _) = tf.keras.datasets.cifar10.load_data()

# train set / data 
x_train = x_train.astype('float32') / 255
y_train = tf.keras.utils.to_categorical(y_train , num_classes=10)

print(x_train.shape, y_train.shape)
# (50000, 32, 32, 3) (50000, 10)

型号

input = tf.keras.Input(shape=(32,32,3))
efnet = tf.keras.layers.Conv2D(filters=64, kernel_size=(3, 3),
                               strides=(1, 1), input_shape=(32, 32, 3),
                               activation='relu')(input)

# Now that we apply global max pooling.
gap = tf.keras.layers.GlobalMaxPooling2D()(efnet)

# Finally, we add a classification layer.
output = tf.keras.layers.Dense(10, activation='softmax')(gap)

# bind all
func_model = tf.keras.Model(input, output)

编译运行

func_model.compile(
          loss      = tf.keras.losses.CategoricalCrossentropy(),
          metrics   = tf.keras.metrics.CategoricalAccuracy(),
          optimizer = tf.keras.optimizers.Adam())
# fit 
func_model.fit(x_train, y_train, batch_size=128, epochs=20, verbose = 1)

获取预测标签和真实标签

ypred = func_model.predict(x_train)
ypred = ypred.argmax(axis=-1)
ypred
array([7, 9, 7, ..., 9, 1, 1])

ytrain = y_train.argmax(axis=-1)
ytrain
array([6, 9, 9, ..., 9, 1, 1])

绘制单个目标的 ROC 曲线。

import matplotlib.pyplot as plt 
from sklearn.preprocessing import LabelBinarizer
from sklearn.metrics import roc_curve, auc, roc_auc_score


target= ['airplane', 'automobile', 'bird', 'cat', 'deer',
          'dog', 'frog', 'horse', 'ship', 'truck']

# set plot figure size
fig, c_ax = plt.subplots(1,1, figsize = (12, 8))

# function for scoring roc auc score for multi-class
def multiclass_roc_auc_score(y_test, y_pred, average="macro"):
    lb = LabelBinarizer()
    lb.fit(y_test)
    y_test = lb.transform(y_test)
    y_pred = lb.transform(y_pred)

    for (idx, c_label) in enumerate(target):
        fpr, tpr, thresholds = roc_curve(y_test[:,idx].astype(int), y_pred[:,idx])
        c_ax.plot(fpr, tpr, label = '%s (AUC:%0.2f)'  % (c_label, auc(fpr, tpr)))
    c_ax.plot(fpr, fpr, 'b-', label = 'Random Guessing')
    return roc_auc_score(y_test, y_pred, average=average)


print('ROC AUC score:', multiclass_roc_auc_score(ytrain, ypred))

c_ax.legend()
c_ax.set_xlabel('False Positive Rate')
c_ax.set_ylabel('True Positive Rate')
plt.show()

ROC AUC score: 0.6868888888888889

Answer 3

    TypeError                                 Traceback (most recent call last)
    ~\AppData\Local\Temp/ipykernel_6176/1137608719.py in <module>
         17     return roc_auc_score(y_test, y_pred, average=average)
         18 
    ---> 19 print('ROC AUC score:', multiclass_roc_auc_score(ytest, ypred))
         20 
         21 c_ax.legend()
    
    ~\AppData\Local\Temp/ipykernel_6176/1137608719.py in multiclass_roc_auc_score(y_test, y_pred, average)

     13     for (idx, c_label) in enumerate(target):
     14         fpr, tpr, thresholds = roc_curve(y_test[:,idx].astype(int), y_pred[:,idx])
---> 15         c_ax.plot(fpr, tpr, label = '%s (AUC:%0.2f)'  % (c_label, auc(fpr, tpr)))
     16     c_ax.plot(fpr, fpr, 'b-', label = 'Random Guessing')
     17     return roc_auc_score(y_test, y_pred, average=average)

TypeError: 'list' object is not callable

对于此代码：-

def multiclass_roc_auc_score(y_test, y_pred, average="macro"):
    lb = LabelBinarizer()
    lb.fit(y_test)
    y_test = lb.transform(y_test)
    y_pred = lb.transform(y_pred)

    for (idx, c_label) in enumerate(target):
        fpr, tpr, thresholds = roc_curve(y_test[:,idx].astype(int), y_pred[:,idx])
        c_ax.plot(fpr, tpr, label = '%s (AUC:%0.2f)'  % (c_label, auc(fpr, tpr)))
    c_ax.plot(fpr, fpr, 'b-', label = 'Random Guessing')
    return roc_auc_score(y_test, y_pred, average=average)

print('ROC AUC score:', multiclass_roc_auc_score(ytest, ypred))
    
c_ax.legend()
c_ax.set_xlabel('False Positive Rate')
c_ax.set_ylabel('True Positive Rate')
plt.show()