Tensorflow 2：如何拟合调用方法中返回多个值的子类模型？

Question

我通过 TensorFlow 2 中的模型子类化构建了以下模型：

from tensorflow.keras import Model, Input
from tensorflow.keras.applications import DenseNet201
from tensorflow.keras.applications.densenet import preprocess_input
from tensorflow.keras.layers import Flatten, Dense

class Detector(Model):
    
    def __init__(self, num_classes=3, name="DenseNet201"):
        super(Detector, self).__init__(name=name)
        self.feature_extractor = DenseNet201(
            include_top=False,
            weights="imagenet",
        )
        self.feature_extractor.trainable = False
        self.flatten_layer = Flatten()
        self.prediction_layer = Dense(num_classes, activation=None)

    def call(self, inputs):
        x = preprocess_input(inputs)
        extracted_feature = self.feature_extractor(x, training=False)
        x = self.flatten_layer(extracted_feature)
        y_hat = self.prediction_layer(x)
        return extracted_feature, y_hat

后续步骤是编译和拟合模型。模型编译正常，但是在拟合我的图像生成器（从ImageDataGenerator 构建）时，我遇到了错误：InvalidArgumentError: Incompatible shapes: [64,18,18] vs. [64,1] [[node Equal (defined at :19) ]] [Op:__inference_train_function_32187] 函数调用堆栈：train_function –。

history = detector.fit(
    train_generator,
    epochs=1,
    validation_data=val_generator,
    callbacks=callbacks
)

这很明显，因为 TensorFlow 不知道预测是 y_hat 还是 extracted_featureduring detector.fit()，因此抛出了错误。那么，对于我的情况，detector.fit 的正确实现是什么？

Answer 1

按照这个question-answer1，您应该首先使用（假设）一个输入和一个输出来训练您的模型。稍后，如果您想计算 grad-cam，您将选择基础模型的一些 中间层（不是基础模型的最终输出），在这种情况下，您需要构建特征提取器分别地。例如

# (let's say: one input and one output)
# use for training 
base_model = keras.application(...)
x = base_model(..)
dese_drop_bn_[whatever] = x
out = dese_drop_bn_[whatever]
model = Model(base_model.input, out) 

# inference / we need to compute grad cam 
new_model = tf.keras.models.Model(model.input, 
                       [model.layers[15].output, model.output])

在上面，model 用于训练，然后在推理时间后期，如果您需要基于层（例如第 15 层）计算 grad-cam，则需要使用适当的输出构建 new_model。希望这能让事情变得清楚。有关特征提取的更多信息，请参阅官方文档Extract and reuse nodes in the graph of layers2。仅供参考，正如我之前告诉你的那样，here 正在发生完全相同的事情。另外，请查看official code example，您会在那里看到完全相同的内容。

---

但是，我认为还有另一种方法可能很容易为您服务。也就是说，当您使用自定义模型时，我们可以在 call() 方法中获取特权 training 参数。通常在训练时间是True，在推理时间是False。因此，基于此，我们可以相应地返回所需的输出。这是完整的代码示例：

import tensorflow as tf 

# get some data
data_dir = tf.keras.utils.get_file(
    'flower_photos',
   'https://storage.googleapis.com/download.tensorflow.org/example_images/flower_photos.tgz',
    untar=True)

datagen_kwargs = dict(rescale=1./255, validation_split=.20)
dataflow_kwargs = dict(target_size=(64, 64),
                       batch_size=16,
                       interpolation="bilinear")

train_datagen = tf.keras.preprocessing.image.ImageDataGenerator(
      rotation_range=40,
      horizontal_flip=True,
      width_shift_range=0.2, height_shift_range=0.2,
      shear_range=0.2, zoom_range=0.2,
      **datagen_kwargs)

train_generator = train_datagen.flow_from_directory(
    data_dir, subset="training", shuffle=True, **dataflow_kwargs)

for image, label in train_generator:
    print(image.shape, image.dtype)
    print(label.shape, label.dtype)
    print(label[:4])
    break

(16, 64, 64, 3) float32
(16, 5) float32
[[0. 0. 0. 0. 1.]
 [0. 0. 0. 1. 0.]
 [0. 0. 0. 1. 0.]
 [0. 0. 0. 0. 1.]]

这里我们根据call 方法中training 的布尔值来实现这个技巧。

class Detector(Model):
    def __init__(self, num_classes=5, name="DenseNet201"):
        super(Detector, self).__init__(name=name)
        self.feature_extractor = DenseNet201(
            include_top=False,
            weights="imagenet",
        )

        self.feature_extractor.trainable = False
        self.flatten_layer = Flatten()
        self.prediction_layer = Dense(num_classes, activation='softmax')

    def call(self, inputs, training):
        x = preprocess_input(inputs)
        extracted_feature = self.feature_extractor(x, training=False)
        x = self.flatten_layer(extracted_feature)
        y_hat = self.prediction_layer(x)

        if training:
            return y_hat 
        else:
            return [y_hat, extracted_feature]

训练

det = Detector()
det.compile(loss='categorical_crossentropy', 
            optimizer='adam', metrics=['acc'])

train_step = train_generator.samples // train_generator.batch_size

det.fit(train_generator, 
      steps_per_epoch=train_step,
      validation_data=train_generator, 
      validation_steps=train_step,
      epochs=2, verbose=2)

Epoch 1/2
37s 139ms/step - loss: 1.7543 - acc: 0.2650 - val_loss: 1.5310 - val_acc: 0.3764
Epoch 2/2
21s 115ms/step - loss: 1.4913 - acc: 0.3915 - val_loss: 1.3066 - val_acc: 0.4667
<tensorflow.python.keras.callbacks.History at 0x7fa2890b1790>

评估

det.evaluate(train_generator, 
      steps=train_step)

4s 76ms/step - loss: 1.3066 - acc: 0.4667
[1.3065541982650757, 0.46666666865348816]

推理

在这里，我们将获得此模型的两个输出（与我们在训练时获得的 1 个输出不同）。

y_hat, base_feature = det.predict(train_generator, 
                        steps=train_step)

y_hat.shape, base_feature.shape
((720, 5), (720, 2, 2, 1920))

现在，您可以进行 grad-cam 或任何需要此类特征图的操作。