动量和（小）批量训练兼容吗？答案

【问题标题】：Are momentum and (mini-) batch training compatible?动量和（小）批量训练兼容吗？
【发布时间】：2017-06-16 12:33:40
【问题描述】：

我有一个运行良好的反向传播模型，但是我想实施批量训练。

批量训练前的代码（在反向传播函数中），伪代码：

forevery(connection in this.connections.in){
  // Adjust weight
  var deltaWeight = rate * gradient + momentum * connection.previousDeltaWeight;
  connection.weight += deltaWeight;
  connection.previousDeltaWeight = deltaWeight;
}

// Adjust bias
var deltaBias = rate * this.error.responsibility + momentum * this.previousDeltaBias;
this.bias += deltaBias;

this.previousDeltabias = deltaBias;

而新代码是：

forevery(connection in this.connections.in){
  // Adjust weight
  var deltaWeight = rate * gradient * this.mask + momentum * connection.previousDeltaWeight;
  connection.totalDeltaWeight += deltaWeight;
  if(update){
    connection.weight += connection.totalDeltaWeight;
    connection.previousDeltaWeight = connection.totalDeltaWeight;
    connection.totalDeltaWeight = 0;
  }
}

// Adjust bias
var deltaBias = rate * this.error.responsibility + momentum * this.previousDeltaBias;
this.totalDeltaBias += deltaBias;
if(update){
  this.bias += this.totalDeltaBias;
  this.previousDeltaBias = this.totalDeltaBias;
  this.totalDeltaBias = 0;
}

因此，如果批量大小为 4，则使用 update=false 调用反向传播 3 次，使用 update=true 调用第四次。批量训练工作正常，但是当我打开动量(=0.9) 时，所有值都开始溢出。可能是什么问题？

【问题讨论】：

标签： neural-network backpropagation

【解决方案1】：

哇。我积蓄的势头是错误的。我将它包含在batch_size 次这是错误的，所以它现在只包含一次。

forevery(connection in this.connections.in){
  // Adjust weight
  var deltaWeight = rate * gradient * this.mask;
  connection.totalDeltaWeight += deltaWeight;
  if(update){
    connection.totalDeltaWeight += momentum * connection.previousDeltaWeight;
    connection.weight += connection.totalDeltaWeight;
    connection.previousDeltaWeight = connection.totalDeltaWeight;
    connection.totalDeltaWeight = 0;
  }
}

// note: MINI_BATCH SHALL BE OPTIMIZED SOON

// Adjust bias
var deltaBias = rate * this.error.responsibility;
this.totalDeltaBias += deltaBias;
if(update){
  this.totalDeltaBias += momentum * this.previousDeltaBias;
  this.bias += this.totalDeltaBias;
  this.previousDeltaBias = this.totalDeltaBias;
  this.totalDeltaBias = 0;
}

【讨论】：