修改感知器变成梯度下降

Question

根据this 视频，感知器和梯度下降算法之间的实质性差异非常小。他们基本上将其指定为：

感知器： Δw_i = η(y - ŷ)x_i

梯度下降： Δw_i = η(y - α)x_i

我已经实现了感知器算法的工作版本，但我不明白需要更改哪些部分才能将其转换为梯度下降。

下面是我的感知器代码的承载部分，我想这些是我需要修改的组件。但是哪里？我需要改变什么？我不明白。

这是出于教学原因，我已经想通了，但仍然对渐变感到困惑，请参阅 更新 下面

      iteration = 0;
      do 
      {
          iteration++;
          globalError = 0;
          //loop through all instances (complete one epoch)
          for (p = 0; p < number_of_files__train; p++) 
          {
              // calculate predicted class
              output = calculateOutput( theta, weights, feature_matrix__train, p, globo_dict_size );
              // difference between predicted and actual class values
              localError = outputs__train[p] - output;
              //update weights and bias
              for (int i = 0; i < globo_dict_size; i++) 
              {
                  weights[i] += ( LEARNING_RATE * localError * feature_matrix__train[p][i] );
              }
              weights[ globo_dict_size ] += ( LEARNING_RATE * localError );

              //summation of squared error (error value for all instances)
              globalError += (localError*localError);
          }

          /* Root Mean Squared Error */
          if (iteration < 10) 
              System.out.println("Iteration 0" + iteration + " : RMSE = " + Math.sqrt( globalError/number_of_files__train ) );
          else
              System.out.println("Iteration " + iteration + " : RMSE = " + Math.sqrt( globalError/number_of_files__train ) );
      } 
      while(globalError != 0 && iteration<=MAX_ITER);

这是我的感知器的症结所在：

  static int calculateOutput( int theta, double weights[], double[][] feature_matrix, int file_index, int globo_dict_size )
  {
     //double sum = x * weights[0] + y * weights[1] + z * weights[2] + weights[3];
     double sum = 0;

     for (int i = 0; i < globo_dict_size; i++) 
     {
         sum += ( weights[i] * feature_matrix[file_index][i] );
     }
     //bias
     sum += weights[ globo_dict_size ];

     return (sum >= theta) ? 1 : 0;
  }

只是我把caculateOutput 方法替换成这样的：

public static double [] gradientDescent(final double [] theta_in, final double alpha, final int num_iters, double[][] data ) 
{
    final double m = data.length;   
    double [] theta = theta_in;
    double theta0 = 0;
    double theta1 = 0;
    for (int i = 0; i < num_iters; i++) 
    {                        
        final double sum0 = gradientDescentSumScalar0(theta, alpha, data );
        final double sum1 = gradientDescentSumScalar1(theta, alpha, data);                                   
        theta0 = theta[0] - ( (alpha / m) * sum0 ); 
        theta1 = theta[1] - ( (alpha / m) * sum1 );                        
        theta = new double [] { theta0, theta1 };
    }
    return theta;
}

---

更新编辑

---

在这一点上，我想我已经很接近了。

我了解如何计算假设，并且我认为我已经正确地做到了，但是，这段代码仍然存在严重错误。我很确定这与我对gradient 的计算有关。当我运行它时，错误会剧烈波动，然后转到infinity，然后只是NaaN。

  double cost, error, hypothesis;
  double[] gradient;
  int p, iteration;

  iteration = 0;
  do 
  {
    iteration++;
    error = 0.0;
    cost = 0.0;

    //loop through all instances (complete one epoch)
    for (p = 0; p < number_of_files__train; p++) 
    {

      // 1. Calculate the hypothesis h = X * theta
      hypothesis = calculateHypothesis( theta, feature_matrix__train, p, globo_dict_size );

      // 2. Calculate the loss = h - y and maybe the squared cost (loss^2)/2m
      cost = hypothesis - outputs__train[p];

      // 3. Calculate the gradient = X' * loss / m
      gradient = calculateGradent( theta, feature_matrix__train, p, globo_dict_size, cost, number_of_files__train);

      // 4. Update the parameters theta = theta - alpha * gradient
      for (int i = 0; i < globo_dict_size; i++) 
      {
          theta[i] = theta[i] - LEARNING_RATE * gradient[i];
      }

    }

    //summation of squared error (error value for all instances)
    error += (cost*cost);       

  /* Root Mean Squared Error */
  if (iteration < 10) 
      System.out.println("Iteration 0" + iteration + " : RMSE = " + Math.sqrt(  error/number_of_files__train  ) );
  else
      System.out.println("Iteration " + iteration + " : RMSE = " + Math.sqrt( error/number_of_files__train ) );
  //System.out.println( Arrays.toString( weights ) );

  } 
  while(cost != 0 && iteration<=MAX_ITER);


}

static double calculateHypothesis( double[] theta, double[][] feature_matrix, int file_index, int globo_dict_size )
{
    double hypothesis = 0.0;

     for (int i = 0; i < globo_dict_size; i++) 
     {
         hypothesis += ( theta[i] * feature_matrix[file_index][i] );
     }
     //bias
     hypothesis += theta[ globo_dict_size ];

     return hypothesis;
}

static double[] calculateGradent( double theta[], double[][] feature_matrix, int file_index, int globo_dict_size, double cost, int number_of_files__train)
{
    double m = number_of_files__train;

    double[] gradient = new double[ globo_dict_size];//one for bias?

    for (int i = 0; i < gradient.length; i++) 
    {
        gradient[i] = (1.0/m) * cost * feature_matrix[ file_index ][ i ] ;
    }

    return gradient;
}

Answer 1

当您具有像(sum >= theta) ? 1 : 0 这样的不可微激活函数时，感知器规则只是梯度下降的近似值。正如他们在视频末尾所问的那样，您不能在那里使用梯度，因为这个阈值函数不可微（好吧，它的梯度不是为 x=0 定义的，并且梯度在其他任何地方都为零）。如果您使用sigmoid 之类的平滑函数而不是此阈值，则可以计算实际梯度。

在这种情况下，您的体重更新将是LEARNING_RATE * localError * feature_matrix__train[p][i] * output_gradient[i]。对于sigmoid的情况，我发给你的链接也显示了如何计算output_gradient。

总而言之，要从感知器更改为梯度下降，您必须

1. 使用导数（梯度）不为零的激活函数 到处。
2. 应用链式规则定义新的更新规则