变分自动编码器损失未显示正确？

Question

我以 Keras 实现为例实现了一个变分自动编码器 (https://keras.io/examples/generative/vae/)。在绘制训练损失时，我注意到这些与控制台中显示的不同。我还看到 Keras 示例中控制台中显示的损失是不正确的，考虑到 total_loss =reconstruction_loss + kl_loss。

控制台显示的loss不是total_loss吗？

我的 VAE 代码：

class Sampling(layers.Layer):
    """Uses (z_mean, z_log_var) to sample z, the vector encoding a digit."""

    def call(self, inputs):
        z_mean, z_log_var = inputs
        batch = tf.shape(z_mean)[0]
        dim = tf.shape(z_mean)[1]
        epsilon = tf.keras.backend.random_normal(shape=(batch, dim))
        return z_mean + tf.exp(0.5 * z_log_var) * epsilon

latent_dim = 100 

encoder_inputs = keras.Input(shape=(64, 64, 3)) #eigentlich 160
x = layers.Conv2D(32, 4, strides=2, padding="same")(encoder_inputs)   
x = layers.LeakyReLU(alpha=0.2)(x)
x = layers.Conv2D(32, 3, strides=1, padding="same")(x)   
x = layers.LeakyReLU(alpha=0.2)(x)
x = layers.Conv2D(64, 4,strides=2, padding="same")(x)
x = layers.LeakyReLU(alpha=0.2)(x)
x = layers.Conv2D(64, 3,strides=1, padding="same")(x)
x = layers.LeakyReLU(alpha=0.2)(x)
x = layers.Conv2D(128, 4,strides=2, padding="same")(x)  
x = layers.LeakyReLU(alpha=0.2)(x)
x = layers.Conv2D(64, 3,strides=1, padding="same")(x)
x = layers.LeakyReLU(alpha=0.2)(x)
x = layers.Conv2D(32, 3,strides=1, padding="same")(x)
x = layers.LeakyReLU(alpha=0.2)(x)
x = layers.Conv2D(100, 8,strides=1, padding="valid")(x)        
x = layers.LeakyReLU(alpha=0.2)(x)
x = layers.Flatten()(x)
z_mean = layers.Dense(latent_dim, name="z_mean")(x)
z_log_var = layers.Dense(latent_dim, name="z_log_var")(x)
z = Sampling()([z_mean, z_log_var])
encoder = keras.Model(encoder_inputs, [z_mean, z_log_var, z], name="encoder")
encoder.summary()

latent_inputs = keras.Input(shape=(latent_dim,))
x = layers.Reshape((1, 1, 100))(latent_inputs)
x = layers.Conv2DTranspose(100, 8, strides=1, padding="valid")(x)
x = layers.LeakyReLU(alpha=0.2)(x)
x = layers.Conv2DTranspose(32, 3, strides=1, padding="same")(x)
x = layers.LeakyReLU(alpha=0.2)(x)
x = layers.Conv2DTranspose(64, 3, strides=1, padding="same")(x)
x = layers.LeakyReLU(alpha=0.2)(x)
x = layers.Conv2DTranspose(128, 4, strides=2, padding="same")(x)
x = layers.LeakyReLU(alpha=0.2)(x)
x = layers.Conv2DTranspose(64, 3, strides=1, padding="same")(x)
x = layers.LeakyReLU(alpha=0.2)(x)
x = layers.Conv2DTranspose(64, 4, strides=2, padding="same")(x)
x = layers.LeakyReLU(alpha=0.2)(x)
x = layers.Conv2DTranspose(32, 3, strides=1, padding="same")(x)
x = layers.LeakyReLU(alpha=0.2)(x)
x = layers.Conv2DTranspose(32, 4, strides=2, padding="same")(x)
x = layers.LeakyReLU(alpha=0.2)(x)

decoder_outputs = layers.Conv2DTranspose(3, 3, activation="sigmoid", padding="same")(x)
decoder = keras.Model(latent_inputs, decoder_outputs, name="decoder")
decoder.summary()

class VAE(keras.Model):
    def __init__(self, encoder, decoder, encoder_t1, encoder_t2, encoder_t3, encoder_t4, **kwargs):
        super(VAE, self).__init__(**kwargs)
        self.encoder = encoder
        self.decoder = decoder

    def train_step(self, data):
        if isinstance(data, tuple):
            data = data[0]
        with tf.GradientTape() as tape:
            z_mean, z_log_var, z = encoder(data)
            reconstruction = decoder(z)
            reconstruction_loss = tf.reduce_mean(    #mean
                keras.losses.mse(data, reconstruction)      #binary_crossentropy
            )
            reconstruction_loss *= 64 * 64                                    #entspricht bildgröße
            kl_loss = 1 + z_log_var - tf.square(z_mean) - tf.exp(z_log_var)
            kl_loss = tf.reduce_mean(kl_loss)      #mean
            kl_loss *= -0.5 
            total_loss = reconstruction_loss + kl_loss
        grads = tape.gradient(total_loss, self.trainable_weights)
        self.optimizer.apply_gradients(zip(grads, self.trainable_weights))
        return {
            "loss": total_loss,
            "reconstruction_loss": reconstruction_loss,
            "kl_loss": kl_loss,
        }
    def call(self, inputs):
      z_mean, z_log_var, z = encoder(inputs)
      reconstruction = decoder(z)
      reconstruction_loss = tf.reduce_mean(
          keras.losses.mse(inputs, reconstruction)
      )
      reconstruction_loss *= 64 * 64
      kl_loss = 1 + z_log_var - tf.square(z_mean) - tf.exp(z_log_var)
      kl_loss = tf.reduce_mean(kl_loss)
      kl_loss *= -0.5 
      total_loss = reconstruction_loss + kl_loss
      self.add_metric(kl_loss, name='kl_loss', aggregation='mean')
      self.add_metric(total_loss, name='total_loss', aggregation='mean')
      self.add_metric(reconstruction_loss, name='reconstruction_loss', aggregation='mean')
      return reconstruction

当我用以下代码绘制我的损失时：

vae_train = vae.fit(
        train_generator,
        steps_per_epoch=nb_train_samples,
        epochs=nb_epoch,
        validation_data=val_generator,
        validation_steps=nb_validation_samples, #141 #3963
        callbacks=[es_callback]
        )

loss = vae_train.history['loss']
val_loss = vae_train.history['val_total_loss']
plt.figure()
plt.plot(range(len(loss)), loss, 'b', label = 'Training loss')
plt.plot(range(len(val_loss)), val_loss, 'm', label='Validation loss')
plt.title('Training and validation loss')
plt.legend()
plt.show()

结果图显示的损失与控制台中显示的损失不同。由于控制台中显示的损失不是reconstruction_loss + kl_loss，而是绘制的损失。

例如这里显示的损失不正确，但绘制正确：（有趣的是 val_total_loss 显示正确）

Epoch 20/100
1266/1266 [==============================] - 82s 65ms/step - loss: 45.2503 - reconstruction_loss: 49.9395 - kl_loss: 0.5695 - val_loss: 0.0000e+00 - val_kl_loss: 0.5888 - val_total_loss: 48.9094 - val_reconstruction_loss: 48.3206

Answer 1

好吧，显然 François Chollet 最近（5 天前）做了一些更改，包括 kl_loss 和reconstruction_loss 的计算方式的更改，请参阅here。

运行之前的版本（您可以在上面的链接中找到）后，我显着减少了方程的两个成员之间的差异，甚至随着纪元的增加而减小（从纪元 7 开始，差异为 <.2>

似乎 VAE 容易被低估重建损失，这是一个持续存在的问题，为此，我鼓励您在文献中挖掘一下，例如this article （可能不是最好的）。

希望对您有所帮助！至少这是一个进步。