深度学习：当学习率太高时答案

【问题标题】：Deep Learning: when learning rate is too high深度学习：当学习率太高时
【发布时间】：2020-06-15 04:31:43
【问题描述】：

当我在 Keras 中改变 SGD 的学习率时，我发现我的代码中有一些非常奇怪的地方：

def build_mlp():
    model = Sequential()
    model.add(Conv2D(24, nb_row=3, nb_col=3, border_mode='same', activation='relu', input_shape=(28, 28, 1)))
    model.add(BatchNormalization(momentum=0.8))
    model.add(Conv2D(24, nb_row=3, nb_col=3, border_mode='same', activation='relu'))
    model.add(BatchNormalization(momentum=0.8))
    model.add(Flatten())
    model.add(Dense(128, activation='relu'))
    model.add(Dense(64, activation='relu'))
    model.add(Dense(10, activation='softmax'))
    model.summary()

    return model


model = build_mlp()
model.compile(loss='categorical_crossentropy', optimizer=SGD(lr=0.0005), metrics=['accuracy'])

在使用 MNIST 数据集进行训练期间，我每 5 个 epoch 将学习率提高一倍。我预计当学习率增加时，损失会发散和振荡。但是，我发现学习率从 0.4 增加到 0.8 后，损失和准确率不再变化。部分记录在这里：

Epoch, Learning rate, Accuracy, Loss
45,0.05119999870657921,0.67200000166893,5.286721663475037
46,0.05119999870657921,0.44419999949634076,8.957198877334594
47,0.05119999870657921,0.21029999982565642,12.728459935188294
48,0.05119999870657921,0.09939999926835298,14.515956773757935
49,0.05119999870657921,0.09949999924749137,14.514344959259033
50,0.10239999741315842,0.09939999926835298,14.515956773757935
51,0.10239999741315842,0.09979999924078584,14.509509530067444
52,0.10239999741315842,0.10109999923035502,14.488556008338929
53,0.10239999741315842,0.10089999923482537,14.49177963256836
54,0.10239999741315842,0.09979999924078584,14.509509530067444
55,0.20479999482631683,0.09899999927729368,14.522404017448425
56,0.20479999482631683,0.10129999965429307,14.4853324508667
57,0.20479999482631683,0.10119999963790179,14.486944255828858
58,0.20479999482631683,0.10129999965429307,14.4853324508667
59,0.20479999482631683,0.10119999963790179,14.486944255828858
60,0.40959998965263367,0.10129999965429307,14.4853324508667
61,0.40959998965263367,0.10119999963790179,14.486944255828858
62,0.40959998965263367,0.10129999965429307,14.4853324508667
63,0.40959998965263367,0.10139999965205788,14.48372064113617
64,0.40959998965263367,0.09189999906346202,14.636842398643493
65,0.8191999793052673,0.10099999930709601,14.490167903900147
66,0.8191999793052673,0.10099999930709601,14.490167903900147
67,0.8191999793052673,0.10099999930709601,14.490167903900147
68,0.8191999793052673,0.10099999930709601,14.490167903900147
69,0.8191999793052673,0.10099999930709601,14.490167903900147
70,1.6383999586105347,0.10099999930709601,14.490167903900147
71,1.6383999586105347,0.10099999930709601,14.490167903900147
72,1.6383999586105347,0.10099999930709601,14.490167903900147
73,1.6383999586105347,0.10099999930709601,14.490167903900147

正如我们所见，在 epoch 65 之后，损失固定在 14.490167903900147 并且不再变化。对这种现象有任何想法吗？任何建议表示赞赏！

【问题讨论】：

标签： python tensorflow keras deep-learning

【解决方案1】：

发生的情况是，您的高学习率已使层的权重超出范围。这反过来会导致 softmax 函数输出正好是 0 和 1 或非常接近这些数字的值。网络变得“过于自信”。

因此，无论输入如何，您的网络都会像这样输出 10 维向量：

[0, 1, 0, 0, 0, 0, 0, 0, 0, 0]
[0, 0, 0, 0, 0, 0, 0, 0, 0, 1]
[0, 0, 0, 1, 0, 0, 0, 0, 0, 0]
...

平均每十次猜对一次，因此准确率保持在 10%。

为了计算网络的损失，Keras 计算每个样本的损失，然后取平均值。在这种情况下，损失是分类交叉熵，相当于取目标标签概率的负对数。

如果为1，则负对数为0：

-np.log(1.0) = 0.0

但是如果它是 0 呢？未定义 0 的对数，因此 Keras 为该值添加了一些平滑：

-np.log(0.0000001) = 16.11809565095832

因此，10 个样本中有 9 个的损失为 16.11809565095832，10 个样本中有 1 个为 0。因此平均而言：

16.11809565095832 * 0.9 = 14.506286085862488

【讨论】：