批量标准化使训练变得更糟

Question

我正在尝试使用 Pytorch 实现批量标准化，并使用简单的全连接神经网络来逼近给定函数。

代码如下。结果表明，没有批量归一化的神经网络比使用批量归一化技术的神经网络表现更好。这意味着批量归一化会使训练变得更糟。有人可以解释这个结果吗？谢谢！

import matplotlib.pyplot as plt
import numpy as np
import torch

class Net(torch.nn.Module):
    
    def __init__(self, num_inputs, num_outputs, hidden_size=256, is_bn=True):
        super(Net, self).__init__()
        self.num_inputs = num_inputs
        self.num_outputs = num_outputs
        self.is_bn = is_bn
        
        # no bias is needed if batch normalization
        if self.is_bn:
            self.linear1 = torch.nn.Linear(num_inputs, hidden_size, bias=False)
            self.linear2 = torch.nn.Linear(hidden_size, hidden_size, bias=False)
        else:            
            self.linear1 = torch.nn.Linear(num_inputs, hidden_size)
            self.linear2 = torch.nn.Linear(hidden_size, hidden_size)
                
        self.linear3 = torch.nn.Linear(hidden_size, num_outputs)
        
        if self.is_bn:
            self.bn1 = torch.nn.BatchNorm1d(hidden_size)
            self.bn2 = torch.nn.BatchNorm1d(hidden_size)

        self.activation = torch.nn.ReLU()
        
    def forward(self, inputs):
        x = inputs
        if self.is_bn:
            x = self.activation(self.bn1(self.linear1(x)))
            x = self.activation(self.bn2(self.linear2(x)))
        else:
            x = self.activation(self.linear1(x))
            x = self.activation(self.linear2(x))
        out = self.linear3(x)        
        return out


torch.manual_seed(0)    # reproducible

Nx = 100
x = torch.linspace(-1., 1., Nx)
x = torch.reshape(x, (Nx, 1))
y = torch.sin(3*x)

fcn_bn, fcn_no_bn = Net(num_inputs=1, num_outputs=1, is_bn=True), Net(num_inputs=1, num_outputs=1, is_bn=False)

criterion = torch.nn.MSELoss()
optimizer_bn = torch.optim.Adam(fcn_bn.parameters(), lr=0.001)
optimizer_no_bn = torch.optim.Adam(fcn_no_bn.parameters(), lr=0.001)

total_epoch = 5000

# record loss history    
loss_history_bn = np.zeros(total_epoch)
loss_history_no_bn = np.zeros(total_epoch)

fcn_bn.train()
fcn_no_bn.train()
for epoch in range(total_epoch):
        
    optimizer_bn.zero_grad()
    loss = criterion(fcn_bn(x), y)    
    loss_history_bn[epoch] = loss.item()
    loss.backward()
    optimizer_bn.step()

    optimizer_no_bn.zero_grad()
    loss = criterion(fcn_no_bn(x), y)    
    loss_history_no_bn[epoch] = loss.item()
    loss.backward()
    optimizer_no_bn.step()
    
    if epoch%1000 == 0:
        print("epoch: %d; MSE (with bn): %.2e; MSE (without bn): %.2e"%(epoch, loss_history_bn[epoch], loss_history_no_bn[epoch]))
        
fcn_bn.eval()
fcn_no_bn.eval()

plt.figure()
plt.semilogy(np.arange(total_epoch), loss_history_bn, label='neural network (with bn)')
plt.semilogy(np.arange(total_epoch), loss_history_no_bn, label='neural network (without bn)')
plt.legend()

plt.figure()
plt.plot(x, y, '-', label='exact')
plt.plot(x, fcn_bn(x).detach(), 'o', markersize=2, label='neural network (with bn)')
plt.plot(x, fcn_no_bn(x).detach(), 'o', markersize=2, label='neural network (without bn)')
plt.legend()

plt.figure()
plt.plot(x, np.abs(fcn_bn(x).detach() - y), 'o', markersize=2, label='neural network (with bn)')
plt.plot(x, np.abs(fcn_no_bn(x).detach() - y), 'o', markersize=2, label='neural network (without bn)')
plt.legend()

plt.show()

结果如下：

epoch: 0; MSE (with bn): 3.99e-01; MSE (without bn): 4.84e-01
epoch: 1000; MSE (with bn): 4.70e-05; MSE (without bn): 1.27e-06
epoch: 2000; MSE (with bn): 1.81e-04; MSE (without bn): 7.93e-07
epoch: 3000; MSE (with bn): 2.73e-04; MSE (without bn): 7.45e-07
epoch: 4000; MSE (with bn): 4.04e-04; MSE (without bn): 5.68e-07

Answer 1

要为 Khalid linked in the comments 的答案提供另一种观点，它更加关注泛化性能而不是训练损失，请考虑以下几点：

已假定批量标准化具有正则化效果。 Luo et al. 将 BN 视为总体归一化和 gamma 衰减的分解，并观察到与您类似的训练损失曲线（将 BN 与无 BN 进行比较 - 但是请注意，它们使用普通 SGD 而不是 Adam）。有几件事会影响 BN（如 Khalid 的link 中所述）：例如，批量大小一方面应该足够大，以便对总体参数进行稳健估计，但是随着批量大小的增加泛化性能也可能下降（参见 Luo 等人的论文：要点是，较小的批量大小会导致嘈杂的总体参数估计，本质上会扰乱输入）。

在您的情况下，我不会凭直觉预期会有很大的不同（考虑到您的数据是如何设置的），但也许对 BN 的理论分析有更深入了解的人仍然可以提供见解。