强制 NN 权重始终在某个范围内

Question

我有一个简单的模型：

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.fc1 = nn.Linear(3, 10)
        self.fc2 = nn.Linear(10, 30)
        self.fc3 = nn.Linear(30, 2)

    def forward(self, x):
        x = torch.relu(self.fc1(x))
        x = torch.relu(self.fc2(x))
        x = torch.tanh(self.fc3(x)) 
        return x

    net = Model()

我怎样才能使权重始终在某个值（例如 -1,1）之间？

我尝试了以下方法：

self.fc1 = torch.tanh(nn.Linear(3, 10))

我不完全确定这会始终将它们保持在该值（即使梯度更新试图将它们推得更远）。

但出现以下错误：

TypeError: tanh(): argument 'input' (position 1) must be Tensor, not Linear

Answer 1

根据讨论.pytorch，您可以创建额外的类来剪辑给定范围之间的权重。 Link 参与讨论。

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.fc1 = nn.Linear(3, 10)
        self.fc2 = nn.Linear(10, 30)
        self.fc3 = nn.Linear(30, 2)

    def forward(self, x):
        x = torch.relu(self.fc1(x))
        x = torch.relu(self.fc2(x))
        x = torch.tanh(self.fc3(x)) 
        return x

你应该添加重量剪：

class WeightClipper(object):
    
    def __call__(self, module):
        # filter the variables to get the ones you want
        if hasattr(module, 'weight'):
            w = module.weight.data
            w = w.clamp(-1,1)
            module.weight.data = w


model = Model()
clipper = WeightClipper()
model.apply(clipper)