具有 L1 正则化的逻辑回归模型

Question

我正在尝试在逻辑模型上应用 L1 正则化

class LogisticRegression(nn.Module):


    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(input_size, num_classes)
    def forward(self, x):
        x = x.reshape(-1, 784)
        output = self.linear(x)
        return output

    def training_step(self, batch):
        images, labels = batch 
        output = self(images)                 
        loss = F.cross_entropy(output, labels)
    
    
        acc = accuracy(output, labels)           
        return {'Training_loss': loss, 'Training_acc': acc}
      
    def training_epoch_end(self, outputs):
        batch_losses = [x['Training_loss'] for x in outputs]
        epoch_loss = torch.stack(batch_losses).mean()   
        batch_accs = [x['Training_acc'] for x in outputs]
        epoch_acc = torch.stack(batch_accs).mean()     
        return {'Training_loss': epoch_loss.item(), 'Training_acc': epoch_acc.item()}

    def epoch_end(self, epoch, result):
        print("Epoch [{}], Training_loss: {:.4f}, Training_acc: {:.4f}".format(epoch, result['Training_loss'], result['Training_acc']))
model = LogisticRegression()

但我认为我做错了，准确性没有改变。

L1=0.2
def evaluate(model_b, trainloader):
    outputs = [model_b.training_step(batch) for batch in trainloader]
    return model_b.training_epoch_end(outputs)

def fit(epochs, lr, model_b, trainloader, opt_func=torch.optim.SGD):
    history = []
    optimizer = opt_func(model_b.parameters(), lr)
    for epoch in range(epochs):
        ##### Training Phase 
        for batch in trainloader:
            loss = model_b.training_step(batch)['Training_loss']
            
            loss_Lasso = loss  + 0.5 * L1 #  L1 reg
            
            loss_Lasso.backward()
            optimizer.step()
            optimizer.zero_grad()
        result = evaluate_b(model_b, trainloader)
        model_b.epoch_end(epoch, result)
        history.append(result)
    return history

谁能帮助我解决我所缺少的内容以及如何真正应用 L1 正则化？ 还有，L1正则化是不是叫lasso？

Answer 1

我相信 l1-norm 是一种 Lasso 正则化，是的，但是 there are others。

在您的 sn-p L1 中设置为常数，而不是您应该测量模型参数的 l1-norm。然后将其与您的网络损失相加，就像您所做的那样。在您的示例中只有一个层，因此您只需要self.linear 的参数。首先收集所有参数，然后用torch.norm 测量总规范。你也可以使用nn.L1Loss。

params = torch.cat([x.view(-1) for x in model.linear.parameters()])
L1 = lamb*torch.norm(params, p=1)

lamb 是您的 lambda 正则化参数，model 是从 LogisticRegression 类初始化的。