pytorch 1D Dropout导致学习不稳定答案

【问题标题】：pytorch 1D Dropout leads to unstable learningpytorch 1D Dropout导致学习不稳定
【发布时间】：2020-05-05 23:54:11
【问题描述】：

我正在 pytorch 中实现类似 Inception 的 CNN。在卷积层块之后，我有三个完全连接的线性层，然后是一个 sigmoid 激活来给我最终的回归输出。我正在测试这个网络中 dropout 层的效果，但它给了我一些意想不到的结果。

代码如下：

class MyInception(nn.Module):
    def __init__(self, in_channels, verbose=False):
        super(MyInception, self).__init__()
        self.v = verbose
        ic=in_channels; oc=16
        self.inceptionBlock1 = InceptionBlock(in_channels=ic, out_channels=oc, maxpool=False, verbose=verbose) 
        self.inceptionBlock2 = InceptionBlock(in_channels=oc * 6, out_channels=oc, maxpool=False, verbose=verbose) 
        self.pool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1)

        self.regressor = nn.Sequential(
            nn.Linear(oc * 6 * 35 * 35, 1024, bias=True),
            nn.ReLU(inplace=True),
            nn.Dropout(p=0.2, inplace=False),  # <--- Dropout 1
            nn.Linear(1024, 128, bias=True),
            nn.ReLU(inplace=True),
            nn.Dropout(p=0.2, inplace=False),  # <--- Dropout 2
            nn.Linear(128, 1, bias=True),
            nn.Sigmoid()
        )

    def forward(self, x):
        x = self.inceptionBlock1(x)
        x = self.inceptionBlock2(x)
        x = self.pool(x)
        x = torch.flatten(x, 1)
        x = self.regressor(x)
        return x


def train(epochs=10, dot_every=25):
    running = pd.DataFrame(columns=['Epoch','Round','TrainLoss','TestLoss','LearningRate'])
    for epoch in range(epochs):
        train_losses = []
        model.train()
        counter = 0

        for images, targets in train_loader:
            images = images.to(device)
            targets = targets.to(device)

            optimizer.zero_grad()
            outputs = model(images)
            loss = loss_fn(torch.flatten(outputs), targets)
            train_losses.append( loss.item() )
            loss.backward()
            optimizer.step()

            counter += 1
            if counter % dot_every == 0: 
                print(".",  end='.', flush=True)
                test_loss = test()
            else:
                test_loss = -1.
            lr = np.squeeze(scheduler.get_lr())
            running = running.append(pd.Series([epoch, counter, loss.item(), test_loss, lr], index=running.columns), ignore_index=True)

        test_loss = test()
        train_loss = np.mean(np.asarray(train_losses))
        running = running.append(pd.Series([epoch, counter, train_loss, test_loss, lr], index=running.columns), ignore_index=True)
        print("")
        print(f"Epoch {epoch+1}, Train Loss: {np.round(train_loss,4)}, Test Loss: {np.round(test_loss, 4)}, Learning Rate: {np.format_float_scientific(lr, precision=4)}")
    return running


def test():
    model.eval()
    test_losses = []
    for i, (images,targets) in enumerate(test_loader):
        images = images.to(device)
        targets = targets.to(device)
        outputs = model(images)
        loss = loss_fn(torch.flatten(outputs), targets)
        test_losses.append( loss.item() )

    mean_loss = np.mean(np.asarray(test_losses))
    return mean_loss

# instantiate the model
model = MyInception(in_channels=4, verbose=False).to(device)
# define the optimizer and loss function
optimizer = Adam(model.parameters(), lr=0.001, weight_decay=0.0001)
loss_fn = nn.MSELoss()

# run it
results = train(epochs=10, dot_every=20)

这是训练数据的 MSE 损失图。（红色=没有辍学，绿色=只有第二次辍学，蓝色=只有第一次辍学，紫色=两个辍学）带有 dropout 的运行在 epoch 边界（垂直虚线）处的损失大幅增加，双 dropout 甚至在 epoch 10 开始时的损失也有很大的跳跃。

重要的是测试损失。在第 5 个 epoch 之后，这两种情况都更加稳定并且没有太大差异，所以也许我不应该在意。但我想了解发生了什么。

【问题讨论】：

标签： pytorch conv-neural-network dropout

【解决方案1】：

我破案了。我意识到我在测试调用中将 model.train() 翻转为 model.eval() 而没有在之后将其设置回 train()。由于 Dropout 在 train 和 eval 模式下的行为不同，因此添加 Dropout 会发现该错误。

【讨论】：