恢复 Tensorflow 模型：在检查点文件中找不到 batch_norm 层的 gamma/scale答案

【问题标题】：restoring Tensorflow model: cannot find gamma/scale for batch_norm layers in the checkpoint file恢复 Tensorflow 模型：在检查点文件中找不到 batch_norm 层的 gamma/scale
【发布时间】：2017-05-05 20:50:13
【问题描述】：

我能够恢复模型并从检查点文件中提取权重、偏差和 batch_norm 层的参数。但是对于多个检查点文件（初始模型等），我找不到 BN 层的缩放/伽马因子。

例如，在公共 inceptionV3 检查点中，我可以定位： InceptionV3/Mixed_5d/Branch_2/Conv2d_0a_1x1/BatchNorm/moving_mean (DT_FLOAT) [64] InceptionV3/Mixed_5d/Branch_2/Conv2d_0a_1x1/BatchNorm/moving_variance (DT_FLOAT) [64] InceptionV3/Mixed_5d/Branch_2/Conv2d_0a_1x1/BatchNorm/beta (DT_FLOAT) [64]

但是，没有像InceptionV3/Mixed_5d/Branch_2/Conv2d_0a_1x1/BatchNorm/gamma 这样的东西。

如何获取伽玛值或默认重新缩放为 1？

非常感谢！

【问题讨论】：

标签： tensorflow restore checkpoint batch-normalization

【解决方案1】：

所以大部分网络使用来自 SLIM 的 batch_norm，默认情况下没有缩放/伽玛参数。

scale：如果为 True，则乘以 gamma。如果 False，gamma 是不曾用过。当下一层是线性的（例如nn.relu）时，这可以是禁用，因为缩放可以由下一层完成。

https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/layers/python/layers/layers.py#L365-L386

【讨论】：

【解决方案2】：

我对 slim 库的预训练 inceptionV2 也有同样的问题。

首先我使用了这个 arg_scope，我遇到了这个问题：

def _batch_norm_arg_scope(list_ops,
                          use_batch_norm=True,
                          batch_norm_decay=0.9997,
                          batch_norm_epsilon=0.001,
                          batch_norm_scale=False,
                          train_batch_norm=False):
    """Slim arg scope for InceptionV2 batch norm."""
    if use_batch_norm:
        batch_norm_params = {
            'is_training': train_batch_norm,
            'scale': batch_norm_scale,
            'decay': batch_norm_decay,
            'epsilon': batch_norm_epsilon
        }
        normalizer_fn = slim.batch_norm
    else:
        normalizer_fn = None
        batch_norm_params = None

    return slim.arg_scope(list_ops,
                          normalizer_fn=normalizer_fn,
                          normalizer_params=batch_norm_params)

我使用 slim 库中的 arg_scope 解决了。

with slim.arg_scope(inception_v2.inception_v2_arg_scope()):

就是这样：

def inception_arg_scope(weight_decay=0.00004,
                        use_batch_norm=True,
                        batch_norm_decay=0.9997,
                        batch_norm_epsilon=0.001,
                        activation_fn=tf.nn.relu):
  """Defines the default arg scope for inception models.

  Args:
    weight_decay: The weight decay to use for regularizing the model.
    use_batch_norm: "If `True`, batch_norm is applied after each convolution.
    batch_norm_decay: Decay for batch norm moving average.
    batch_norm_epsilon: Small float added to variance to avoid dividing by zero
      in batch norm.
    activation_fn: Activation function for conv2d.

  Returns:
    An `arg_scope` to use for the inception models.
  """
  batch_norm_params = {
      # Decay for the moving averages.
      'decay': batch_norm_decay,
      # epsilon to prevent 0s in variance.
      'epsilon': batch_norm_epsilon,
      # collection containing update_ops.
      'updates_collections': tf.GraphKeys.UPDATE_OPS,
      # use fused batch norm if possible.
      'fused': None,
  }
  if use_batch_norm:
    normalizer_fn = slim.batch_norm
    normalizer_params = batch_norm_params
  else:
    normalizer_fn = None
    normalizer_params = {}
  # Set weight_decay for weights in Conv and FC layers.
  with slim.arg_scope([slim.conv2d, slim.fully_connected],
                      weights_regularizer=slim.l2_regularizer(weight_decay)):
    with slim.arg_scope(
        [slim.conv2d],
        weights_initializer=slim.variance_scaling_initializer(),
        activation_fn=activation_fn,
        normalizer_fn=normalizer_fn,
        normalizer_params=normalizer_params) as sc:
      return sc

【讨论】：