基于 Keras 中的自定义指标的早期停止和学习率计划

Question

我在 Keras 中有一个对象检测模型，并希望根据在验证集上计算的平均精度 (mAP) 来监控和控制我的训练。

我已将代码从 tensorflow-models 移植到我的脚本中，该脚本使用提供的模型和数据运行评估。它不是作为 Keras 指标实现的，而是作为一个独立的类实现的：

evaluation = SSDEvaluation(model, data, data_size)
mAP = evaluation.evaluate()

我完全可以拥有这样的东西。确实，我不希望为训练批次计算它，因为它会减慢训练速度。

我的问题是：如何根据每个 epoch 之后计算的这个指标来重用 ReduceLROnPlateau 和 EarlyStopping 回调？

Answer 1

您可以使用更新您的logs 对象的LambdaCallback 来做到这一点：

假设你的 evaluation.evaluate() 返回一个像 {'val/mAP': value} 这样的字典，你可以这样做：

eval_callback = LambdaCallback(
     on_epoch_end=lambda epoch, logs: logs.update(evaluation.evaluate())
) 

这里的诀窍是logs 会被进一步传递给其他回调，所以他们可以直接访问该值：

early_stopping = EarlyStopping(monitor='val/mAP', min_delta=0.0, patience=10, verbose=1, mode='max') 

它将自动出现在CSVLogger 和任何其他回调中。但请注意，eval_callback 必须在使用回调列表中的值的任何回调之前：

callbacks = [eval_callback, early_stopping]

Answer 2

我不确定SSDEvaluation 是什么，但如果可以接受任何没有开销的平均精度计算，我建议使用 keras callbacks 进行以下方法。

您希望 oto 使用两个 callbacl 的主要思想 - EarlyStopping 和 ReduceLROnPlateau - 都在 epoch 结束时起作用并监视 loss 或 metric 值。他们从method 的logs 参数中获取此值

 def on_epoch_end(self, epoch, logs=None):
     """Called at the end of an epoch.
     ...
     """

- 将实际的 map 发送到日志值，我们强制此方法和所有从日志中获取准确度值的回调使用它。 Callbcaks 从这里选择值（thisine int 代码 - 提前停止，this 一个用于 Reduce LR）。
因此，我们应该为这两个回调“伪造”日志。我想这并不理想，但可行的解决方案。

这些类从回调继承并计算 map 值，也避免了通过共享对象 Hub 重新计算 map。

from sklearn.metrics import average_precision_score

import keras
from keras.callbacks import Callback, EarlyStopping, ReduceLROnPlateau


class MAPHub:
    def __init__(self):
        self.map_value = None

- 它只是共享 map 值的中心。可能会引起一些副作用。你可以尽量避免使用它。

def on_epoch_end(self, epoch, logs):
    """self just a callbcak instance"""
    if self.last_metric_for_epoch == epoch:
        map_ = self.hub.map_value
    else:
        prediction = self.model.predict(self._data, verbose=1)
        map_ = average_precision_score(self._target, prediction)
        self.hub.map_value = map_
        self.last_metric_for_epoch = epoch

- 此函数计算并共享 地图

class EarlyStoppingByMAP(EarlyStopping):
    def __init__(self, data, target, hub, *args, **kwargs):
        """
        data, target - values and target for the map calculation
        hub - shared object to store _map_ value 
        *args, **kwargs for the super __init__
        """
        # I've set monitor to 'acc' here, because you're interested in metric, not loss
        super(EarlyStoppingByMAP, self).__init__(monitor='acc', *args, **kwargs)
        self._target = target
        self._data = data 
        self.last_metric_for_epoch = -1
        self.hub = hub

    def on_epoch_end(self, epoch, logs):
        """
        epoch is the number of epoch, logs is a dict logs with 'loss' value 
        and metric 'acc' values
        """
        on_epoch_end(self, epoch, logs)      
        logs['acc'] = self.hub.map_value  # "fake" metric with calculated value
        print('Go callback from the {}, logs: \n{}'.format(EarlyStoppingByMAP.__name__, logs))
        super(EarlyStoppingByMAP, self).on_epoch_end(epoch, logs)  # works as a callback fn


class ReduceLROnPlateauByMAP(ReduceLROnPlateau):
    def __init__(self, data, target, hub, *args, **kwargs):
        # the same as in previous
        # I've set monitor to 'acc' here, because you're interested in metric, not loss
        super(ReduceLROnPlateauByMAP, self).__init__(monitor='acc', *args, **kwargs)
        self._target = target
        self._data = data 
        self.last_metric_for_epoch = -1
        self.hub = hub


    def on_epoch_end(self, epoch, logs):
        on_epoch_end(self, epoch, logs)
        logs['acc'] = self.hub.map_value   # "fake" metric with calculated value
        print('Go callback from the {}, logs: \n{}'.format(ReduceLROnPlateau.__name__, logs))
        super(ReduceLROnPlateauByMAP, self).on_epoch_end(epoch, logs)  # works as a callback fn

- NB 不要在构造函数中使用monitor 参数！您应该使用'acc'，参数已经设置为正确的值。

一些测试：

from keras.datasets import mnist
from keras.models import Model
from keras.layers import Dense, Input
import numpy as np

(X_tr, y_tr), (X_te, y_te) = mnist.load_data()
X_tr = (X_tr / 255.).reshape((60000, 784))
X_te = (X_te / 255.).reshape((10000, 784))


def binarize_labels(y):
    y_bin = np.zeros((len(y), len(np.unique(y)))) 
    y_bin[range(len(y)), y] = 1
    return y_bin

y_train_bin, y_test_bin = binarize_labels(y_tr), binarize_labels(y_te)


inp = Input(shape=(784,))
x = Dense(784, activation='relu')(inp)
x = Dense(256, activation='relu')(x)
out = Dense(10, activation='softmax')(x)

model = Model(inp, out)
model.compile(loss='categorical_crossentropy', optimizer='adam')

- 一个简单的“测试套件”。现在去适应它：

hub = MAPHub()  # instentiate a hub
# I will use default params except patience as example, set it to 1 and 5
early_stop = EarlyStoppingByMAP(X_te, y_test_bin, hub, patience=1)  # Patience is EarlyStopping's param
reduce_lt = ReduceLROnPlateauByMAP(X_te, y_test_bin, hub, patience=5)  # Patience is ReduceLR's param

history = model.fit(X_tr, y_train_bin, epochs=10, callbacks=[early_stop, reduce_lt])
Out:
Epoch 1/10
60000/60000 [==============================] - 12s 207us/step - loss: 0.1815
10000/10000 [==============================] - 1s 59us/step
Go callback from the EarlyStoppingByMAP, logs: 
{'loss': 0.18147853660446903, 'acc': 0.9934216252519924}
10000/10000 [==============================] - 0s 40us/step
Go callback from the ReduceLROnPlateau, logs: 
{'loss': 0.18147853660446903, 'acc': 0.9934216252519924}
Epoch 2/10
60000/60000 [==============================] - 12s 197us/step - loss: 0.0784
10000/10000 [==============================] - 0s 40us/step
Go callback from the EarlyStoppingByMAP, logs: 
{'loss': 0.07844233275586739, 'acc': 0.9962269038764738}
10000/10000 [==============================] - 0s 41us/step
Go callback from the ReduceLROnPlateau, logs: 
{'loss': 0.07844233275586739, 'acc': 0.9962269038764738}
Epoch 3/10
60000/60000 [==============================] - 12s 197us/step - loss: 0.0556
10000/10000 [==============================] - 0s 40us/step
Go callback from the EarlyStoppingByMAP, logs: 
{'loss': 0.05562876497630107, 'acc': 0.9972085346550085}
10000/10000 [==============================] - 0s 40us/step
Go callback from the ReduceLROnPlateau, logs: 
{'loss': 0.05562876497630107, 'acc': 0.9972085346550085}
Epoch 4/10
60000/60000 [==============================] - 12s 198us/step - loss: 0.0389
10000/10000 [==============================] - 0s 41us/step
Go callback from the EarlyStoppingByMAP, logs: 
{'loss': 0.0388911374788188, 'acc': 0.9972696414934574}
10000/10000 [==============================] - 0s 41us/step
Go callback from the ReduceLROnPlateau, logs: 
{'loss': 0.0388911374788188, 'acc': 0.9972696414934574}
Epoch 5/10
60000/60000 [==============================] - 12s 197us/step - loss: 0.0330
10000/10000 [==============================] - 0s 39us/step
Go callback from the EarlyStoppingByMAP, logs: 
{'loss': 0.03298293751536124, 'acc': 0.9959456176387349}
10000/10000 [==============================] - 0s 39us/step
Go callback from the ReduceLROnPlateau, logs: 
{'loss': 0.03298293751536124, 'acc': 0.9959456176387349}

好的，看起来至少是一个提前停止的作品。我猜是ReduceLROnPlateau，因为它们使用相同的日志和相似的逻辑——如果设置了适当的参数的话。

如果你不想使用 sklearn 函数，但是 SSDEvaluation（我只是找不到它是什么） - 你可以很容易地采用 on_epoch_method 函数来处理这个评估函数。

希望对你有帮助。