Pytorch：训练期间权重不变

Question

与这里的这个问题基本相同，但从未得到回答：Why the first convolutional layer weights don't change during training?

我只想观察卷积层的权重在训练过程中的变化。我怎样才能做到这一点？无论我做什么，即使损失减少，权重似乎也保持不变。

尽管模型略有不同，但我正在尝试在此处学习本教程：https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html#sphx-glr-beginner-blitz-cifar10-tutorial-py

型号

class CNN(nn.Module):
    def __init__(self):
        super(Digit_Classifier, self).__init__()
        self.conv1 = nn.Conv2d(1,6,3)
        self.pool1 = nn.MaxPool2d(2)
        self.conv2 = nn.Conv2d(6,16,3)
        self.pool2 = nn.MaxPool2d(2)
        self.out = nn.Linear(400, 10)

    def forward(self, inputs):
        x = self.pool1(F.relu(self.conv1(inputs)))
        x = self.pool2(F.relu(self.conv2(x)))
        x = torch.flatten(x, start_dim=1)
        x = self.out(x)
        return x

培训

def train(epochs=100):
    criterion = nn.CrossEntropyLoss()
    net = CNN()
    optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)
    losses = []
    
    for epoch in range(epochs):  # loop over the dataset multiple times

        running_loss = 0.0
        for i, data in enumerate(trainloader, 0):
            # get the inputs; data is a list of [inputs, labels]
            inputs, labels = data

            # zero the parameter gradients
            optimizer.zero_grad()

            outputs = net(inputs)
            loss = criterion(outputs, labels)
            loss.backward()
            optimizer.step()

            running_loss += loss.item()            
            w = model.conv1._parameters['weight']
            print(w)            

            losses.append(running_loss / z)
            if i % 2000 == 1999:    # print every 2000 mini-batches
                print('[%d, %5d] loss: %.3f' % (epoch + 1, i + 1, running_loss / 2000))
                running_loss = 0.0
    return net

Answer 1

如果不使用任何归一化模块，权重越接近网络的输入，梯度越小，因此变化也会越小，因此变化可能以小数形式显示，不再显示在您的print() 声明中。要查看更改，我建议将权重从一次迭代保存到下一次迭代，然后减去它们以显示差异：

...
w = model.conv1._parameters['weight'].detach()
print(w-w_previous)
w_previous = w
...