学习 MNIST 数据库时训练损失非常高

Question

我正在开发我的 ANN从头开始它应该对手写数字（0-9）的MNIST database 进行分类。我的前馈全连接 ANN必须由...组成的：

1. 一个输入层，有28x28 = 784个节点（即每张图像的特征）
2. 一个隐藏层，具有任意数量的神经元（浅层网络）
3. 一个输出层，带有10 节点（每个数字一个）

   和不得不计算梯度 w.r.t.由于权重和偏差反向传播算法，最后，它应该学习利用动量梯度下降算法。

   损失函数是：cross_entropy on "softmaxed" 网络的输出，因为任务是关于分类.

   每个隐藏的神经元都由相同的激活函数激活，我选择了sigmoid；同时输出的神经元被identity 函数激活。

   数据集分为：

   1. 60.000 训练对 (image, label) - 用于训练
   2. 5000 验证对 (image, label) - 用于评估并选择最小化验证损失的网络
   3. 5000 测试对 (image, label) - 用于测试使用新指标（例如准确性）选择的模型

      数据已通过调用sklearn.utils.shuffle 方法进行了混洗。

      这些是我的网络在训练损失、验证损失和验证准确性方面的表现：

      E(0) on TrS is: 798288.7537714319  on VS is: 54096.50409967187  Accuracy: 12.1 %
      E(1) on TrS is: 798261.8584179751  on VS is: 54097.23663558976  Accuracy: 12.1 %
      ...
      E(8) on TrS is: 798252.1191081362  on VS is: 54095.5016235736  Accuracy: 12.1 %
      ...
      E(17) on TrS is: 798165.2674011206  on VS is: 54087.2823473459  Accuracy: 12.8 %
      E(18) on TrS is: 798155.0888987815  on VS is: 54086.454077456074  Accuracy: 13.22 %
      ...
      E(32) on TrS is: 798042.8283810444  on VS is: 54076.35518400717  Accuracy: 19.0 %
      E(33) on TrS is: 798033.2512910366  on VS is: 54075.482037626025  Accuracy: 19.36 %
      E(34) on TrS is: 798023.431899881  on VS is: 54074.591145985265  Accuracy: 19.64 %
      E(35) on TrS is: 798013.4023181734  on VS is: 54073.685418577166  Accuracy: 19.759999999999998 %
      E(36) on TrS is: 798003.1960815473  on VS is: 54072.76783050559  Accuracy: 20.080000000000002 %
      ...
      E(47) on TrS is: 797888.8213232228  on VS is: 54062.70342708315  Accuracy: 21.22 %
      E(48) on TrS is: 797879.005388998  on VS is: 54061.854566864626  Accuracy: 21.240000000000002 %
      E(49) on TrS is: 797869.3890292909  on VS is: 54061.02482142968  Accuracy: 21.26 %
      Validation loss is minimum at epoch: 49
      

      正如你所看到的，损失非常高，学习是非常减缓。

      这是我的代码：

      import numpy as np
      from scipy.special import expit
      from matplotlib import pyplot as plt
      from mnist.loader import MNIST
      from sklearn.utils import shuffle
      
      
      def relu(a, derivative=False):
          f_a = np.maximum(0, a)
          if derivative:
              return (a > 0) * 1
          return f_a  
      
      def softmax(y):
          e_y = np.exp(y - np.max(y))
          return e_y / e_y.sum()
      
      def cross_entropy(y, t, derivative=False, post_process=True):
          epsilon = 10 ** -308
          if post_process:
              if derivative:
                  return y - t
              sm = softmax(y)
              sm = np.clip(sm, epsilon, 1 - epsilon)  # avoids log(0)
              return -np.sum(np.sum(np.multiply(t, np.log(sm)), axis=0))
      
      def sigmoid(a, derivative=False):
          f_a = expit(a)
          if derivative:
              return np.multiply(f_a, (1 - f_a))
          return f_a
      
      def identity(a, derivative=False):
          f_a = a
          if derivative:
              return np.ones(np.shape(a))
          return f_a
      
      def accuracy_score(targets, predictions):
          correct_predictions = 0
          for item in range(np.shape(predictions)[1]):
              argmax_idx = np.argmax(predictions[:, item])
              if targets[argmax_idx, item] == 1:
                  correct_predictions += 1
          return correct_predictions / np.shape(predictions)[1]
      
      
      def one_hot(targets):
          return np.asmatrix(np.eye(10)[targets]).T
      
      
      def plot(epochs, loss_train, loss_val):
          plt.plot(epochs, loss_train)
          plt.plot(epochs, loss_val, color="orange")
          plt.legend(["Training Loss", "Validation Loss"])
          plt.xlabel("Epochs")
          plt.ylabel("Loss")
          plt.grid(True)
          plt.show()
      
      class NeuralNetwork:
      
          def __init__(self):
              self.layers = []
      
          def add_layer(self, layer):
              self.layers.append(layer)
      
          def build(self):
              for i, layer in enumerate(self.layers):
                  if i == 0:
                      layer.type = "input"
                  else:
                      layer.type = "output" if i == len(self.layers) - 1 else "hidden"
                      layer.configure(self.layers[i - 1].neurons)
      
          def fit(self, X_train, targets_train, X_val, targets_val, max_epochs=50):
              e_loss_train = []
              e_loss_val = []
      
              # Getting the minimum loss on validation set
              predictions_val = self.predict(X_val)
              min_loss_val = cross_entropy(predictions_val, targets_val)
      
              best_net = self  # net which minimize validation loss
              best_epoch = 0  # epoch where the validation loss is minimum
      
              # batch mode
              for epoch in range(max_epochs):
                  predictions_train = self.predict(X_train)
                  self.back_prop(targets_train, cross_entropy)
                  self.learning_rule(l_rate=0.00001, momentum=0.9)
                  loss_train = cross_entropy(predictions_train, targets_train)
                  e_loss_train.append(loss_train)
      
                  # Validation
                  predictions_val = self.predict(X_val)
                  loss_val = cross_entropy(predictions_val, targets_val)
                  e_loss_val.append(loss_val)
      
                  print("E(%d) on TrS is:" % epoch, loss_train, " on VS is:", loss_val, " Accuracy:",
                        accuracy_score(targets_val, predictions_val) * 100, "%")
      
                  if loss_val < min_loss_val:
                      min_loss_val = loss_val
                      best_epoch = epoch
                      best_net = self
        
              plot(np.arange(max_epochs), e_loss_train, e_loss_val)
      
              return best_net
      
          # Matrix of predictions where the i-th column corresponds to the i-th item
          def predict(self, dataset):
              z = dataset.T
              for layer in self.layers:
                  z = layer.forward_prop_step(z)
              return z
      
          def back_prop(self, target, loss):
              for i, layer in enumerate(self.layers[:0:-1]):
                  next_layer = self.layers[-i]
                  prev_layer = self.layers[-i - 2]
                  layer.back_prop_step(next_layer, prev_layer, target, loss)
      
          def learning_rule(self, l_rate, momentum):
              # Momentum GD
              for layer in [layer for layer in self.layers if layer.type != "input"]:
                  layer.update_weights(l_rate, momentum)
                  layer.update_bias(l_rate, momentum)
      
      
      class Layer:
      
          def __init__(self, neurons, type=None, activation=None):
              self.dE_dW = None  # derivatives dE/dW where W is the weights matrix
              self.dE_db = None  # derivatives dE/db where b is the bias
              self.dact_a = None  # derivative of the activation function
              self.out = None  # layer output
              self.weights = None  # input weights
              self.bias = None  # layer bias
              self.w_sum = None  # weighted_sum
              self.neurons = neurons  # number of neurons
              self.type = type  # input, hidden or output
              self.activation = activation  # activation function
              self.deltas = None  # for back-prop
      
          def configure(self, prev_layer_neurons):
              self.set_activation()
              self.weights = np.asmatrix(np.random.normal(-0.1, 0.02, (self.neurons, prev_layer_neurons)))
              self.bias = np.asmatrix(np.random.normal(-0.1, 0.02, self.neurons)).T 
      
      
          def set_activation(self):
              if self.activation is None:
                  if self.type == "hidden":
                      self.activation = sigmoid
                  elif self.type == "output":
                      self.activation = identity  # will be softmax in cross entropy calculation
      
          def forward_prop_step(self, z):
              if self.type == "input":
                  self.out = z
              else:
                  self.w_sum = np.dot(self.weights, z) + self.bias
                  self.out = self.activation(self.w_sum)
              return self.out
      
          def back_prop_step(self, next_layer, prev_layer, target, local_loss):
              if self.type == "output":
                  self.dact_a = self.activation(self.w_sum, derivative=True)
                  self.deltas = np.multiply(self.dact_a,
                                            local_loss(self.out, target, derivative=True))
              else:
                  self.dact_a = self.activation(self.w_sum, derivative=True)  # (m,batch_size)
                  self.deltas = np.multiply(self.dact_a, np.dot(next_layer.weights.T, next_layer.deltas))
      
              self.dE_dW = self.deltas * prev_layer.out.T
      
              self.dE_db = np.sum(self.deltas, axis=1)
      
          def update_weights(self, l_rate, momentum):
              # Momentum GD
              self.weights = self.weights - l_rate * self.dE_dW
              self.weights = -l_rate * self.dE_dW + momentum * self.weights
      
          def update_bias(self, l_rate, momentum):
              # Momentum GD
              self.bias = self.bias - l_rate * self.dE_db
              self.bias = -l_rate * self.dE_db + momentum * self.bias
      
      
      if __name__ == '__main__':
          mndata = MNIST(path="data", return_type="numpy")
          X_train, targets_train = mndata.load_training()  # 60.000 images, 28*28 features
          X_val, targets_val = mndata.load_testing()  # 10.000 images, 28*28 features
      
          X_train = X_train / 255  # normalization within [0;1]
          X_val = X_val / 255  # normalization within [0;1]
      
          X_train, targets_train = shuffle(X_train, targets_train.T)
          X_val, targets_val = shuffle(X_val, targets_val.T)
      
          # Getting the test set splitting the validation set in two equal parts
          # Validation set size decreases from 10.000 to 5000 (of course)
          X_val, X_test = np.split(X_val, 2)  # 5000 images, 28*28 features
          targets_val, targets_test = np.split(targets_val, 2)
          X_test, targets_test = shuffle(X_test, targets_test.T)
      
          targets_train = one_hot(targets_train)
          targets_val = one_hot(targets_val)
          targets_test = one_hot(targets_test)
      
          net = NeuralNetwork()
          d = np.shape(X_train)[1]  # number of features, 28x28
          c = np.shape(targets_train)[0]  # number of classes, 10
      
          # Shallow network with 1 hidden neuron
          # That is 784, 1, 10
          for m in (d, 1, c):
              layer = Layer(m)
              net.add_layer(layer)
      
          net.build()
      
          best_net = net.fit(X_train, targets_train, X_val, targets_val, max_epochs=50)
      

      我做了什么：

      1. 设置500而不是1隐藏神经元
      2. 添加许多隐藏层
      3. 降低/提高学习率 (l_rate) 值
      4. 减少/增加momentum（并将其设置为0）
      5. 将sigmoid 替换为relu

         但问题仍然存在。

         这些是我用于计算的公式（当然，您可以从源代码中查看它们）：

         笔记：公式中的 f 和 g 代表隐藏层激活函数和输出层激活函数。

Answer 1

看起来您没有将数据集标准化为。损失较大的原因是每个像素代表 uint8 的范围从 0 到 255，并且当您使用交叉熵时，损失值很大。

要完成这项工作，您应该在 for 循环中将 train_X 和 val_X 与 255 分开：

        for epoch in range(max_epochs):
            X_train = X_train/255 ############ add this
            predictions_train = self.predict(X_train)
            self.back_prop(targets_train, cross_entropy)
            self.learning_rule(l_rate=0.00001, momentum=0.9)
            loss_train = cross_entropy(predictions_train, targets_train)
            e_loss_train.append(loss_train)

            # Validation
            X_val = X_val/255 ############ add this
            predictions_val = self.predict(X_val)
            loss_val = cross_entropy(predictions_val, targets_val)
            e_loss_val.append(loss_val)

            print("E(%d) on TrS is:" % epoch, loss_train, " on VS is:", loss_val, " Accuracy:",
                  accuracy_score(targets_val, predictions_val) * 100, "%")

            if loss_val < min_loss_val:
                min_loss_val = loss_val
                best_epoch = epoch
                best_net = self

注意：我假设predictions_val 返回的值介于 0 和 1 之间。如果不是，则将它们除以 predictions_val = predictions_val/255。

希望这会有所帮助！！！！！！！