如何在 tensorflow.keras 模型指标中使用 sklearn AUC？答案

【问题标题】：how to use sklearn AUC in tensorflow.keras model metrics?如何在 tensorflow.keras 模型指标中使用 sklearn AUC？
【发布时间】：2019-12-25 17:19:23
【问题描述】：

我正在尝试在 tf.keras 中使用 sklearn AUC 作为模型指标，为此我使用了来自此链接 AUC 的自定义函数

下面是我的模型：

def auc(y_true, y_pred):
    return tf.py_func(roc_auc_score, (y_true, y_pred), tf.double)

model = Model(inputs= [text_x,state_x,grade_x,cat_x,subcat_x,teach_x,num_x],outputs = [output_layer])
model.compile(optimizer = 'Adam', loss= 'binary_crossentropy', metrics=[auc])

history = model.fit(x = input_data , y= y_train,batch_size = 180, epochs = 15, callbacks = [es, mc], validation_data = (val_data, y_val))

Train on 69918 samples, validate on 17480 samples
Epoch 1/15
69918/69918 [==============================] - 278s 4ms/sample - loss: 0.3086 - auc: 0.8516 - val_loss: 0.4711 - val_auc: 0.6896
Epoch 2/15
69918/69918 [==============================] - 275s 4ms/sample - loss: 0.1417 - auc: 0.9738 - val_loss: 0.6638 - val_auc: 0.6692
Epoch 3/15
69918/69918 [==============================] - 275s 4ms/sample - loss: 0.0506 - auc: 0.9964 - val_loss: 0.9611 - val_auc: 0.6824
Epoch 4/15
69918/69918 [==============================] - 276s 4ms/sample - loss: 0.0329 - auc: 0.9983 - val_loss: 0.9462 - val_auc: 0.6719

评估模型时出现此错误，ValueError：

test_input_data = [text_test_1,state_test,grade_test,cat_test,subcat_test,teach_test,num_test]
score = model.evaluate(test_input_data, y_test,verbose = 1)
print('test_loss: ',score[0])
print('test_acc: ',score[1])

<ipython-input-103-336c032c70f4> in <module>()
      1 test_input_data = [text_test_1,state_test,grade_test,cat_test,subcat_test,teach_test,num_test]
----> 2 score = model.evaluate(test_input_data, y_test,verbose = 1)
      3 print('test_loss: ',score[0])
      4 print('test_acc: ',score[1])

3 frames
/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py in __call__(self, *args, **kwargs)
   1456         ret = tf_session.TF_SessionRunCallable(self._session._session,
   1457                                                self._handle, args,
-> 1458                                                run_metadata_ptr)
   1459         if run_metadata:
   1460           proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

InvalidArgumentError: 2 root error(s) found.
  (0) Invalid argument: ValueError: Only one class present in y_true. ROC AUC score is not defined in that case.
Traceback (most recent call last):

  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/script_ops.py", line 209, in __call__
    ret = func(*args)

  File "/usr/local/lib/python3.6/dist-packages/sklearn/metrics/ranking.py", line 355, in roc_auc_score
    sample_weight=sample_weight)

  File "/usr/local/lib/python3.6/dist-packages/sklearn/metrics/base.py", line 76, in _average_binary_score
    return binary_metric(y_true, y_score, sample_weight=sample_weight)

  File "/usr/local/lib/python3.6/dist-packages/sklearn/metrics/ranking.py", line 323, in _binary_roc_auc_score
    raise ValueError("Only one class present in y_true. ROC AUC score "

ValueError: Only one class present in y_true. ROC AUC score is not defined in that case.


     [[{{node metrics_15/auc/PyFunc}}]]
  (1) Invalid argument: ValueError: Only one class present in y_true. ROC AUC score is not defined in that case.
Traceback (most recent call last):

  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/script_ops.py", line 209, in __call__
    ret = func(*args)

  File "/usr/local/lib/python3.6/dist-packages/sklearn/metrics/ranking.py", line 355, in roc_auc_score
    sample_weight=sample_weight)

  File "/usr/local/lib/python3.6/dist-packages/sklearn/metrics/base.py", line 76, in _average_binary_score
    return binary_metric(y_true, y_score, sample_weight=sample_weight)

  File "/usr/local/lib/python3.6/dist-packages/sklearn/metrics/ranking.py", line 323, in _binary_roc_auc_score
    raise ValueError("Only one class present in y_true. ROC AUC score "

ValueError: Only one class present in y_true. ROC AUC score is not defined in that case.


     [[{{node metrics_15/auc/PyFunc}}]]
     [[metrics_15/auc/PyFunc/_1683]]
0 successful operations.
0 derived errors ignored.

我尝试了 tf.keras.metrics.AUC，然后它工作正常，但使用 sklearn AUC 时出现此错误。如何在 tf.keras.model 度量函数中设置 sklearn 的 AUC。任何帮助将不胜感激..谢谢。

【问题讨论】：

问题是 y_test 只有一个类的标签，你应该有两个类的标签，否则无法计算 AUC。
@MatiasValdenegro 你是对的，我的数据是不平衡的数据，我有 2 个标签，并且在 keras 模型中将数据分成批次，可能在一批中只包含 1 个标签。那么如何摆脱它
@MatiasValdenegro 即使我在尝试使用它时也遇到了类似的问题。有什么方法可以编写一个函数来跳过只存在一个类标签的批次。我找不到任何帮助（或）解决此问题。
这能回答你的问题吗？ How to compute Receiving Operating Characteristic (ROC) and AUC in keras?

标签： python tensorflow keras scikit-learn auc

【解决方案1】：

我遇到了同样的问题，但在 Github 上找到了这段代码：pranaya-mathur account 你也可以关注

from sklearn.metrics import roc_auc_score
def auc_score(y_true, y_pred):
    if len(np.unique(y_true[:,1])) == 1:
        return 0.5
    else:
        return roc_auc_score(y_true, y_pred)
def auc(y_true, y_pred):
    return tf.py_func(auc1, (y_true, y_pred), tf.double)
#in model.compile you can use auc function name
model.compile(optimizer=optimizer,loss='categorical_crossentropy',metrics=[auc])

【讨论】：