如何在 TensorFlow 中锁定张量的特定值？

Question

我正在尝试将彩票假设应用于用 TensorFlow 2.0（使用 Keras 接口）编写的简单神经网络，如下所示：

net = models.Sequential()
net.add(layers.Dense(256, activation="softsign", name="Dense0", bias_initializer="ones"))
net.add(layers.Dense(128, activation="softsign", name="Dense1", bias_initializer="ones"))
net.add(layers.Dense(64, activation="softsign", name="Dense2", bias_initializer="ones"))
net.add(layers.Dense(32, activation="softsign", name="Dense3", bias_initializer="ones"))
net.add(layers.Dense(1, activation="tanh", name="Output", bias_initializer="ones"))

然后我使用 Adam 优化器和二元交叉熵损失来训练我的网络：

net.compile(optimizer=optimizers.Adam(learning_rate=0.001),
            loss=losses.BinaryCrossentropy(), metrics=["accuracy"])
net.fit(x_train, y_train, epochs=10, batch_size=32, validation_data=(x_test, y_test))

在训练过程之后，我想在我的网络中锁定特定的权重。问题是，我只能使用tensorflow.Variable(..., trainable=False) 将张量锁定为不可训练（据我所知），但这样做我将图的整个节点设置为不可训练，我只想要特定的边缘。我可以使用以下代码遍历我网络的所有张量实例：

for i in range(len(net.layers)):
    for j in range(net.layers[i].variables[0].shape[0]):
        for k in range(net.layers[i].variables[0][j].shape[0]):
            ...

但我不知道下一步该做什么。有人知道我可以做到这一点的简单方法吗？

Answer 1

也许您可以对 Dense 层进行子类化？类似的东西

class PrunableDense(keras.layers.Dense):


    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.deleted_channels = None
        self.deleted_bias = None
        self._kernel=None
        self._bias=None


    def build(self, input_shape):
        last_dim = input_shape[-1]
        self._kernel = self.add_weight(
            'kernel',
            shape=[last_dim, self.units],
            initializer=self.kernel_initializer,
            regularizer=self.kernel_regularizer,
            constraint=self.kernel_constraint,
            dtype=self.dtype,
            trainable=True)
        self.deleted_channels = tf.ones([last_dim, self.units]) # we'll use this to prune the network
        if self.use_bias:
            self._bias = self.add_weight(
                'bias',
                shape=[self.units,],
                initializer=self.bias_initializer,
                regularizer=self.bias_regularizer,
                constraint=self.bias_constraint,
                dtype=self.dtype,
                trainable=True)
            self.deleted_bias = tf.ones([self.units,])

    @property
    def kernel(self):
        """gets called whenever self.kernel is used"""
        # only the weights that haven't been deleted should be non-zero
        # deleted weights are 0.'s in self.deleted_channels
        return self.deleted_channels * self._kernel  

    @property
    def bias(self):
        #similar to kernel
        if not self.use_bias:
            return None
        else:
            return self.deleted_bias * self._bias

    def prune_kernel(self, to_be_deleted):
        """
        Delete some channels
        to_be_deleted should be a tensor or numpy array of shape kernel.shape
        containing 1's at the locations where weights should be kept, and 0's 
        at the locations where weights should be deleted.
        """
        self.deleted_channels *= to_be_deleted

    def prune_bias(self, to_be_deleted):
        assert(self.use_bias)
        self.deleted_bias *= to_be_deleted

    def prune_kernel_below_threshold(self, threshold=0.01):
        to_be_deleted = tf.cast(tf.greater(self.kernel, threshold), tf.float32)
        self.deleted_channels *= to_be_deleted

    def prune_bias_below_threshold(self, threshold=0.01):
        assert(self.use_bias)
        to_be_deleted = tf.cast(tf.greater(self.bias, threshold), tf.float32)
        self.deleted_bias *= to_be_deleted

我没有对此进行过广泛的测试，它肯定需要一些改进，但我认为这个想法应该可行。

编辑：我写了上面假设你想像彩票假设一样修剪网络，但如果你只想冻结部分权重，你可以做类似的事情，但是添加一个非零的 frozen_kernel 属性仅在 self.deleted_channels 为 0 的地方输入，并将其添加到可训练内核中。

编辑 2：在之前的编辑中，我的意思如下：

class FreezableDense(keras.layers.Dense):


    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.trainable_channels = None
        self.trainable_bias = None
        self._kernel1 = None
        self._bias1 = None
        self._kernel2 = None
        self._bias2 = None


    def build(self, input_shape):
        last_dim = input_shape[-1]
        self._kernel1 = self.add_weight(
            'kernel1',
            shape=[last_dim, self.units],
            initializer=self.kernel_initializer,
            regularizer=self.kernel_regularizer,
            constraint=self.kernel_constraint,
            dtype=self.dtype,
            trainable=True)
        self._kernel2 = tf.zeros([last_dim, self.units])
        self.trainable_channels = tf.ones([last_dim, self.units]) # we'll use this to freeze parts of the network
        if self.use_bias:
            self._bias1 = self.add_weight(
                'bias',
                shape=[self.units,],
                initializer=self.bias_initializer,
                regularizer=self.bias_regularizer,
                constraint=self.bias_constraint,
                dtype=self.dtype,
                trainable=True)
            self._bias2 = tf.zeros([self.units,])
            self.trainable_bias = tf.ones([self.units,])

    @property
    def kernel(self):
        """gets called whenever self.kernel is used"""
        # frozen 
        return self.trainable_channels * self._kernel1 + (1 - self.trainable_channels) * self._kernel2

    @property
    def bias(self):
        #similar to kernel
        if not self.use_bias:
            return None
        else:
            return self.trainable_bias * self._bias1 + (1 - self.trainable_bias) * self._bias2

    def freeze_kernel(self, to_be_frozen):
        """
        freeze some channels
        to_be_frozen should be a tensor or numpy array of shape kernel.shape
        containing 1's at the locations where weights should be kept trainable, and 0's 
        at the locations where weights should be frozen.
        """
        # we want to do two things: update the weights in self._kernel2 
        # and update self.trainable_channels
        # first we update self._kernel2 with all newly frozen weights
        newly_frozen = 1 - tf.maximum((1 - to_be_frozen) - (1 - self.trainable_channels), 0)
        # the above should have 0 only where to_be_frozen is 0 and self.trainable_channels is 1
        # if I'm not mistaken that is
        newly_frozen_weights = (1-newly_frozen)*self._kernel1
        self._kernel2 += newly_frozen_weights

        # now we update self.trainable_channels:
        self.trainable_channels *= to_be_frozen

    def prune_bias(self, to_be_deleted):
        assert(self.use_bias)
        newly_frozen = 1 - tf.maximum((1 - to_be_frozen) - (1 - self.trainable_bias), 0)
        newly_frozen_bias = (1-newly_frozen)*self._bias1
        self._bias2 += newly_frozen_bias
        self.trainable_bias *= to_be_frozen

（同样没有经过精心测试，肯定需要一些改进，但我认为这个想法应该可行）

编辑 3： 谷歌搜索得到了我最初找不到的东西：https://www.tensorflow.org/model_optimization/api_docs/python/tfmot/sparsity/keras migth 提供了更容易构建修剪模型的工具。

编辑4（进一步解释_kernel2和_bias2的作用）：

为简单起见，我将解释没有偏见，但经过必要的修改后，一切都与偏见相同。假设密集层的输入是 n 维，输出是 m 维，那么密集层所做的就是将输入乘以 m×n 矩阵，我们简称为 K（它是内核） .

通常我们希望通过一些基于梯度的优化方法来学习 K 的正确条目，但在您的情况下，您希望保持某些条目固定。这就是为什么在这个自定义的 Dense 层中，我们将 K 拆分如下：

K = T * K1 + (1 - T) * K2，

在哪里

• T 是一个由 0 和 1 组成的 m×n 矩阵，
• 星号表示元素乘法
• 1 是 m×n 矩阵，每个条目为 1
• K1 是一个可以学习的 m×n 矩阵
• K2 是一个 m×n 矩阵，在训练期间是固定的（常数）。

如果我们查看 K 的条目，那么 K[i,j] = T[i,j]*K1[i,j] + (1-T[i,j])*K2[i,j ] = K1[i,j] 如果 T[i,j]==1 否则 K2[i,j]。因为在后一种情况下，K1[i,j] 的值对乘以 K 的结果没有影响，它的梯度是 0 并且不应该改变（即使它确实由于数值错误而改变，也不应该改变对 K[i,j] 的值有影响）。

所以本质上，T[i,j]==0 的 K 的条目 K[i,j] 是固定的（值存储在 K2 中），而 T[i,j]==1 的条目可以训练。