贝叶斯模型不使用 tensorflow 概率和 keras 进行学习

Question

我想估计我的模型的认知不确定性。所以我将所有层转换为张量流概率层。该模型没有返回错误，但它也没有学到任何东西。该模型有两个输出，两个输出的损失根本没有变化。另一方面，模型的整体损失在缩小，但似乎与其他损失完全无关，我无法解释。

import numpy as np
from tensorflow import keras
import tensorflow_probability as tfp
import tensorflow as tf
from plot.plot_utils import plot_model_metrics
from Custom_Keras_layers.ProbSqueezeExcite import squeeze_excite_block

inp = keras.layers.Input(shape=[self.timesteps, self.features])
    # left side
    # 1 Conv1D block
    l = tfp.layers.Convolution1DFlipout(filters=2*self.features, kernel_size=2, padding='same', activation=tf.nn.relu)(inp)
    l = keras.layers.BatchNormalization()(l)
    if squeeze_excite == 1:
        l = squeeze_excite_block(l)
    l = keras.layers.Dropout(dropout_rate)(l, training=True)

    # 1 Conv1D block
    l = tfp.layers.Convolution1DFlipout(filters=4 * self.features, kernel_size=4, padding='same', activation=tf.nn.relu)(l)
    l = keras.layers.BatchNormalization()(l)
    if squeeze_excite == 1:
        l = squeeze_excite_block(l)
    l = keras.layers.Dropout(dropout_rate)(l, training=True)

    # 1 lstm bock
    l = keras.layers.LSTM(32, recurrent_dropout=dropout_rate, dropout=dropout_rate)(l, training=True)

    # letf output layer
    l = tfp.layers.DenseFlipout(self.classes, activation=tf.nn.softmax, name='left')(l)

    # right side
    # 1 Conv1D block
    r = tfp.layers.Convolution1DFlipout(filters=2 * self.features, kernel_size=2, padding='same', activation=tf.nn.relu)(inp)
    r = keras.layers.BatchNormalization()(r)
    if squeeze_excite == 1:
        r = squeeze_excite_block(r)
    r = keras.layers.Dropout(dropout_rate)(r, training=True)

    # 1 Conv1D block
    r = tfp.layers.Convolution1DFlipout(filters=4 * self.features, kernel_size=4, padding='same', activation=tf.nn.relu)(r)
    r = keras.layers.BatchNormalization()(r)
    if squeeze_excite == 1:
        r = squeeze_excite_block(r)
    r = keras.layers.Dropout(dropout_rate)(r, training=True)

    # 1 lstm bock
    r = keras.layers.LSTM(32, recurrent_dropout=dropout_rate, dropout=dropout_rate)(r, training=True)

    # letf output layer
    r = tfp.layers.DenseFlipout(self.classes, activation=tf.nn.softmax, name='right')(r)

    model = keras.models.Model(inputs=inp, outputs=[l, r])

    optimizer = tf.train.AdamOptimizer(learning_rate=lr)
    losses = {
        "left": self._neg_log_likelihood_bayesian,
        "right": self._neg_log_likelihood_bayesian}
    model.compile(optimizer=optimizer, loss=losses, metrics=['accuracy'])
    self.model = model

损失函数定义如下：

    def _neg_log_likelihood_bayesian(self, y_true, y_pred):
    labels_distribution = tfp.distributions.Categorical(logits=y_pred)
    neg_log_likelihood = -tf.reduce_mean(labels_distribution.log_prob(tf.argmax(y_true, axis=-1)))
    kl = sum(self.model.losses) / self.trainNUM
    loss = neg_log_likelihood + kl
    return loss

任何帮助将不胜感激。总损失从 45000 开始，而两个输出损失在 1.3 左右。这对我来说很奇怪。

Answer 1

感谢github tensorflow问题论坛上的这篇帖子，我找到了解决方法https://github.com/tensorflow/probability/issues/282 您必须在每个 tfp 层内缩放 KL 总和：

 kernel_divergence_fn=lambda q, p, _: tfp.distributions.kl_divergence(q, p) / tf.to_float(train.num_examples))

此外，我将损失函数更改为：

neg_log_likelihood = tf.nn.softmax_cross_entropy_with_logits_v2（标签=y_true， logits=y_pred)

这对我有用，现在我的模型正在正确训练。

Answer 2

整体损失可能已经包括了先前的损失 kl(sampled weights||prior)，所以可以重复计算吗？ （我不确定 Keras 是如何处理这个问题的。）另一个想法是尝试使用 reduce_sum 而不是 reduce_mean。