我的批量累积实施是否正确？

Question

我想知道我用于训练批量累积模型的代码是否正确。尤其是关于损失计算的部分，因为我不太确定这是正确的方法。 这是我的代码：

def train (start_epochs, n_epochs, best_acc, train_generator, val_generator, model, optimizer, criterion, checkpoint_path, best_model_path):


#num_epochs = 25
  since = time.time()

  #best_model_wts = copy.deepcopy(model.state_dict())
  #best_acc = 0.0
  train_loss = []
  val_loss = []
  train_acc = []
  val_acc = []

  batch_accumulation = 8

  for epoch in tqdm(range(start_epochs, n_epochs+1)):

    running_train_loss = 0.0
    running_val_loss = 0.0

    running_train_corrects = 0
    running_val_corrects = 0

    optimizer.zero_grad
    #Training
    model.train()
    for i, (faces, labels) in tqdm(enumerate(train_generator)):
      
      faces = faces.to(device)
      labels = labels.to(device)

      #forward
      outputs = model(faces)

      #predictions of the model determined using the torch.max() function, which returns the index of the maximum value in a tensor.
      _, preds = torch.max(outputs[1], 1)

      #pass the model outputs and the true image labels to the loss function
      loss = criterion(outputs[1], labels)
      #loss = loss / batch_accumulation
      running_train_loss += loss.item()
      # Backprop and Adam optimisation
      loss.backward()
      # Track the accuracy and loss
      running_train_corrects += torch.sum(preds == labels.data)

      if (i+1)% batch_accumulation == 0:
        optimizer.step()
        optimizer.zero_grad # zero the gradient buffers 
       
    # calculate average losses and accuracy  
    epoch_train_loss = running_train_loss / len(train_generator.dataset)
    epoch_train_acc = ((running_train_corrects.double() / len(train_generator.dataset)) * 100)
    train_loss.append(epoch_train_loss)
    train_acc.append(epoch_train_acc)

    print('Train Loss: {:.4f} Train Acc: {:.2f}%'.format(epoch_train_loss, epoch_train_acc))

    #Validation
    with torch.set_grad_enabled(False):
      model.eval()
      for i , (faces_val, labels_val) in tqdm(enumerate(val_generator)):

        faces_val = faces_val.to(device)
        labels_val = labels_val.to(device)
        
        if (i+1)% batch_accumulation == 0:

          outputs_val = model(faces_val)

          _, preds_val = torch.max(outputs_val[1], 1)
          loss_val = criterion(outputs_val[1], labels_val)

          running_val_loss += loss_val.item() 
          #running_val_loss = running_val_loss +((1 /(i+1)) * (loss.item() - running_val_loss))
          running_val_corrects += torch.sum(preds_val == labels_val.data)

    # calculate average losses and accuracy 
    epoch_val_loss = running_val_loss / len(validation_generator.dataset)
    epoch_val_acc = (running_val_corrects.double() / len(validation_generator.dataset)) * 100
    val_loss.append(epoch_val_loss)
    val_acc.append(epoch_val_acc)

    print('Validation Loss: {:.4f} Validation Acc: {:.2f}%'.format(epoch_val_loss, epoch_val_acc))

我得到了奇怪的 epoch train 结果（例如 456.890），并且我确信验证部分中的 if 语句。

Answer 1

验证阶段不需要gradient accumulation（实名），所以这部分在那边：

if (i+1)% batch_accumulation == 0:    
    outputs_val = model(faces_val)

没有任何意义（不需要if）。该技术仅用于训练，以使小批量的梯度估计更准确，因此我们应该重点关注它。

梯度累积

每次我们运行backward() 计算的梯度都会添加到图的叶子中。通常，我们在整个批次中使用mean（将总和除以批次中的元素数量）。在这里，我们累积损失，因此我们应该将它除以累积步骤数，这给了我们（实际上你已经把它注释掉了）：

loss = criterion(outputs[1], labels)
loss = loss / batch_accumulation

否则损失可能太大（可能是这里的情况），即使学习率非常低也会使网络不稳定）。

你也可以运行这个：

running_train_loss += loss.item()

按累计计算。

最后，正如@Dishin H Goyani 所指出的，zero_grad 是一个函数，因此您应该运行：

optimizer.zero_grad()

Answer 2

你可能缺少括号

optimizer.zero_grad # zero the gradient buffers 

正确的调用方式是

optimizer.zero_grad()