Pytorch 损失在 vgg 19 模型中没有变化

Question

在 pytorch 中我做了一个模型 vgg19 用于分类 tiny imagenet：

model = nn.Sequential(
    nn.BatchNorm2d(3),
    nn.Conv2d(in_channels=3, out_channels=64, kernel_size=(3,3), padding=1),
    nn.ReLU(),
    nn.Conv2d(in_channels=64, out_channels=64, kernel_size=(3,3), padding=1),
    nn.ReLU(),
    nn.MaxPool2d((2,2)),
    nn.Conv2d(in_channels=64, out_channels=128, kernel_size=(3,3), padding=1),
    nn.ReLU(),
    nn.Conv2d(in_channels=128, out_channels=128, kernel_size=(3,3), padding=1),
    nn.ReLU(),
    nn.MaxPool2d((2,2)),
    nn.Conv2d(in_channels=128, out_channels=256, kernel_size=(3,3), padding=1),
    nn.ReLU(),
    nn.Conv2d(in_channels=256, out_channels=256, kernel_size=(3,3), padding=1),
    nn.ReLU(),
    nn.Conv2d(in_channels=256, out_channels=256, kernel_size=(3,3), padding=1),
    nn.ReLU(),
    nn.Conv2d(in_channels=256, out_channels=256, kernel_size=(3,3), padding=1),
    nn.ReLU(),
    nn.MaxPool2d((2,2)),
    nn.Conv2d(in_channels=256, out_channels=512, kernel_size=(3,3), padding=1),
    nn.ReLU(),
    nn.Conv2d(in_channels=512, out_channels=512, kernel_size=(3,3), padding=1),
    nn.ReLU(),
    nn.Conv2d(in_channels=512, out_channels=512, kernel_size=(3,3), padding=1),
    nn.ReLU(),
    nn.Conv2d(in_channels=512, out_channels=512, kernel_size=(3,3), padding=1),
    nn.ReLU(),
    nn.MaxPool2d((2,2)),
    nn.Conv2d(in_channels=512, out_channels=512, kernel_size=(3,3), padding=1),
    nn.ReLU(),
    nn.Conv2d(in_channels=512, out_channels=512, kernel_size=(3,3), padding=1),
    nn.ReLU(),
    nn.Conv2d(in_channels=512, out_channels=512, kernel_size=(3,3), padding=1),
    nn.ReLU(),
    nn.Conv2d(in_channels=512, out_channels=512, kernel_size=(3,3), padding=1),
    nn.ReLU(),
    nn.MaxPool2d((2,2)),
    nn.Flatten(),
    nn.Linear(25088, 4096),
    nn.Linear(4096, 1000),
    nn.Linear(1000, 200),
    nn.Softmax(),
    nn.Dropout2d(),
)

在学习过程中，损失保持在大致相同的值 (5.3 +- 0.01)。

###############
iter = 4000 / 80000
loss = 5.295811176300049
###############
iter = 4800 / 80000
loss = 5.298299789428711
###############
iter = 5600 / 80000
loss = 5.309792995452881
###############
iter = 6400 / 80000
loss = 5.3179707527160645
###############
iter = 7200 / 80000
loss = 5.3179707527160645

我已经尝试增加和减少lr，batch_size。但仍然不知道如何解决它。

培训代码 (epochs = 10, loss = cross_entropy, optim = Adam(lr = 0.01), X_batch.shape = (batch_size, 3, 224, 224)：

for epoch in range(num_epochs):
    i = 0
    for (X_batch, y_batch) in train_batch_gen:
        X_batch = Variable(torch.FloatTensor(X_batch)).cuda()
        y_batch = Variable(torch.LongTensor(y_batch)).cuda()
        logits = model.cuda().forward(X_batch)
        opt.zero_grad()
        loss = lossFunc(logits, y_batch)
        loss.backward()
        opt.step()

        train_loss.append(loss.data.cpu().numpy())
        if i % (batch_size*100) == 0:
            print("###############")
            print(f"iter = {i} / {80000}")
            print(f"loss = {np.mean(train_loss[-len(train_dataset) // batch_size :])}")
        i += batch_size

Answer 1

nn.CrossEntropyLoss 应用 log-softmax，但您也在模型中应用 softmax：

nn.Linear(1000, 200),
nn.Softmax(),
nn.Dropout2d(),

模型的输出必须是原始 logits，没有 nn.Softmax()。

此外，不应该在模型输出之前使用 dropout，因为这会有效地消除一些类，从而使损失惩罚原本正确的东西。 Dropout 只能在层之间用作正则化。