如何根据 AUC 指标在 Keras 中保存最佳模型？

Question

我想保存基于 auc 的 Keras 中最好的模型，我有这个代码：

def MyMetric(yTrue, yPred):
    auc = tf.metrics.auc(yTrue, yPred)
    return auc

best_model = [ModelCheckpoint(filepath='best_model.h5', monitor='MyMetric', save_best_only=True)]

train_history = model.fit([train_x], 
          [train_y], batch_size=batch_size, epochs=epochs, validation_split=0.05, 
                          callbacks=best_model, verbose = 2)

所以我的模型运行疯了，我收到了这个警告：

RuntimeWarning: Can save best model only with MyMetric available, skipping.
  'skipping.' % (self.monitor), RuntimeWarning)

如果有人能告诉我这是正确的做法，那将是很好的，如果不是，我应该怎么做？

Answer 1

您必须将要监控的 Metric 传递给 model.compile。

https://keras.io/metrics/#custom-metrics

model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=[MyMetric])

此外，tf.metrics.auc 返回一个包含张量和 update_op 的元组。 Keras 期望自定义度量函数只返回一个张量。

def MyMetric(yTrue, yPred):
    import tensorflow as tf
    auc = tf.metrics.auc(yTrue, yPred)
    return auc[0]

执行此步骤后，您将收到有关未初始化值的错误。请查看以下主题：

https://github.com/keras-team/keras/issues/3230

How to compute Receiving Operating Characteristic (ROC) and AUC in keras?

Answer 2

您可以通过以下方式定义调用 tensorflow 来计算 AUROC 的自定义指标：

def as_keras_metric(method):
    import functools
    from keras import backend as K
    import tensorflow as tf
    @functools.wraps(method)
    def wrapper(self, args, **kwargs):
        """ Wrapper for turning tensorflow metrics into keras metrics """
        value, update_op = method(self, args, **kwargs)
        K.get_session().run(tf.local_variables_initializer())
        with tf.control_dependencies([update_op]):
            value = tf.identity(value)
        return value
    return wrapper

@as_keras_metric
def AUROC(y_true, y_pred, curve='ROC'):
    return tf.metrics.auc(y_true, y_pred, curve=curve)

然后您需要使用此指标编译您的模型：

model.compile(loss=train_loss, optimizer='adam', metrics=['accuracy',AUROC])

最后：通过以下方式检查模型：

model_checkpoint = keras.callbacks.ModelCheckpoint(path_to_save_model, monitor='val_AUROC', 
                                                   verbose=0, save_best_only=True, 
                                                   save_weights_only=False, mode='auto', period=1)

但要小心：我相信验证 AUROC 是按批次计算并取平均值的；所以可能会给检查点带来一些错误。一个好主意可能是在模型训练完成后验证训练模型的预测（使用 sklearn.metrics 计算）的 AUROC 是否与训练和检查点时 Tensorflow 报告的相匹配

Answer 3

假设你使用TensorBoard，那么你有一个历史记录——以tfevents文件的形式——你所有的度量计算，你所有的时期；那么tf.keras.callbacks.Callback 就是你想要的。

我使用tf.keras.callbacks.ModelCheckpoint 和save_freq: 'epoch' 将每个时期的权重保存为h5 文件or tf file。

为避免用模型文件填充硬盘驱动器，请编写一个新的 Callback——或扩展 ModelCheckpoint 类的——on_epoch_end 实现：

def on_epoch_end(self, epoch, logs=None):
    super(DropWorseModels, self).on_epoch_end(epoch, logs)
    if epoch < self._keep_best:
        return

    model_files = frozenset(
        filter(lambda filename: path.splitext(filename)[1] == SAVE_FORMAT_WITH_SEP,
               listdir(self._model_dir)))

    if len(model_files) < self._keep_best:
        return

    tf_events_logs = tuple(islice(log_parser(tfevents=path.join(self._log_dir,
                                                                self._split),
                                             tag=self.monitor),
                                  0,
                                  self._keep_best))
    keep_models = frozenset(map(self._filename.format,
                                map(itemgetter(0), tf_events_logs)))

    if len(keep_models) < self._keep_best:
        return

    it_consumes(map(lambda filename: remove(path.join(self._model_dir, filename)),
                    model_files - keep_models))

附录（导入和实用功能实现）：

from itertools import islice
from operator import itemgetter
from os import path, listdir, remove
from collections import deque

import tensorflow as tf
from tensorflow.core.util import event_pb2


def log_parser(tfevents, tag):
    values = []
    for record in tf.data.TFRecordDataset(tfevents):
        event = event_pb2.Event.FromString(tf.get_static_value(record))
        if event.HasField('summary'):
            value = event.summary.value.pop(0)
            if value.tag == tag:
                values.append(value.simple_value)

    return tuple(sorted(enumerate(values), key=itemgetter(1), reverse=True))

it_consumes = lambda it, n=None: deque(it, maxlen=0) if n is None \
                                 else next(islice(it, n, n), None)

SAVE_FORMAT = 'h5'
SAVE_FORMAT_WITH_SEP = '{}{}'.format(path.extsep, SAVE_FORMAT)

为了完整起见，课程的其余部分：

class DropWorseModels(tf.keras.callbacks.Callback):
    """
    Designed around making `save_best_only` work for arbitrary metrics
             and thresholds between metrics
    """

    def __init__(self, model_dir, monitor, log_dir, keep_best=2, split='validation'):
        """
        Args:
            model_dir: directory to save weights. Files will have format
                        '{model_dir}/{epoch:04d}.h5'.
            split: dataset split to analyse, e.g., one of 'train', 'test', 'validation'
            monitor: quantity to monitor.
            log_dir: the path of the directory where to save the log files to be
                        parsed by TensorBoard.
            keep_best: number of models to keep, sorted by monitor value
        """
        super(DropWorseModels, self).__init__()
        self._model_dir = model_dir
        self._split = split
        self._filename = 'model-{:04d}' + SAVE_FORMAT_WITH_SEP
        self._log_dir = log_dir
        self._keep_best = keep_best
        self.monitor = monitor

这具有能够在单个回调中保存和删除多个模型文件的额外优势。您可以使用不同的阈值支持轻松扩展，例如，将所有模型文件的 AUC 保持在阈值或 TP、FP、TN、FN 在阈值内。