【cs231n】两层神经网络的反向传播

文章目录

• 矩阵求导方法
• 证明
• 两层神经网络的梯度求导

矩阵求导方法

• 维度相容原则：假设每个中间变量量的维度都不不⼀一样，看怎么摆能把雅克比矩阵的维度摆成矩阵乘法规则允许的形式。只要把矩阵维度倒腾顺了了，公式也就对了了。
• 设有$f ( Y ) : \mathbb { R } ^ { m \times p } \rightarrow \mathbb { R }$f(Y):Rm×p→R，$Y = A X + B : \mathbb { R } ^ { n \times p } \rightarrow \mathbb { R } ^ { m \times p }$Y=AX+B:Rn×p→Rm×p，则$\nabla _ { X } f ( A X + B ) = A ^ { T } \nabla _ { Y } f$∇X​f(AX+B)=AT∇Y​f，即$\frac { \partial f } { \partial X} = A ^ { T } \frac { \partial f} { \partial Y }$∂X∂f​=AT∂Y∂f​
• 设有$f ( Y ) : \mathbb { R } ^ { m \times p } \rightarrow \mathbb { R }$f(Y):Rm×p→R，$Y = X A+ B : \mathbb { R } ^ { m \times n } \rightarrow \mathbb { R } ^ { m \times p }$Y=XA+B:Rm×n→Rm×p，则$\nabla _ { X } f (XA + B ) = { \nabla _ { Y } f}A ^ { T }$∇X​f(XA+B)=∇Y​fAT，即$\frac { \partial f } { \partial X} = \frac { \partial f} { \partial Y }{A ^ { T } }$∂X∂f​=∂Y∂f​AT

证明

在前向传播过程中，X的shape(N,D)，W的shape(D,C)，Y=XW。现在，我们假设N = 2, D = 2, C = 3。那么
$X = \left( \begin{array} { l l } { x _ { 1,1 } } & { x _ { 1,2 } } \\ { x _ { 2,1 } } & { x _ { 2,2 } } \end{array} \right) \qquad W = \left( \begin{array} { l l l } { w _ { 1,1 } } & { w _ { 1,2 } } & { w _ { 1,3 } } \\ { w _ { 2,1 } } & { w _ { 2,2 } } & { w _ { 2,3 } } \end{array} \right)$X=(x1,1​x2,1​​x1,2​x2,2​​)W=(w1,1​w2,1​​w1,2​w2,2​​w1,3​w2,3​​)$Y = X W = \left( \begin{array} { l l } { x _ { 1,1 } w _ { 1,1 } + x _ { 1,2 } w _ { 2,1 } } & { x _ { 1,1 } w _ { 1,2 } + x _ { 1,2 } w _ { 2,2 } } & { x _ { 1,1 } w _ { 1,3 } + x _ { 1,2 } w _ { 2,3 } } \\ { x _ { 2,1 } w _ { 1,1 } + x _ { 2,2 } w _ { 2,1 } } & { x _ { 2,1 } w _ { 1,2 } + x _ { 2,2 } w _ { 2,2 } } & { x _ { 2,1 } w _ { 1,3 } + x _ { 2,2 } w _ { 2,3 } } \end{array} \right)$Y=XW=(x1,1​w1,1​+x1,2​w2,1​x2,1​w1,1​+x2,2​w2,1​​x1,1​w1,2​+x1,2​w2,2​x2,1​w1,2​+x2,2​w2,2​​x1,1​w1,3​+x1,2​w2,3​x2,1​w1,3​+x2,2​w2,3​​)在前向传播结束后，我们通过输出Y计算得到损失函数L，然后求得$\frac { \partial L } { \partial Y }$∂Y∂L​$\frac { \partial L } { \partial Y } = \left( \begin{array} { l l l } { \frac { \partial L } { \partial y _ { 1,1 } } } & { \frac { \partial L } { \partial y _ { 1,2 } } } & { \frac { \partial L } { \partial y _ { 1,3 } } } \\ { \frac { \partial L } { \partial y _ { 2,1 } } } & { \frac { \partial L } { \partial y _ { 2,2 } } } & { \frac { \partial L } { \partial y _ { 2,3 } } } \end{array} \right)$∂Y∂L​=(∂y1,1​∂L​∂y2,1​∂L​​∂y1,2​∂L​∂y2,2​∂L​​∂y1,3​∂L​∂y2,3​∂L​​)
然后计算$\frac { \partial L } { \partial X } = \frac { \partial L } { \partial Y } \frac { \partial Y } { \partial X }$∂X∂L​=∂Y∂L​∂X∂Y​，$\frac { \partial L } { \partial W } = \frac { \partial L } { \partial Y } \frac { \partial Y } { \partial W }$∂W∂L​=∂Y∂L​∂W∂Y​。$\frac { \partial L } { \partial X }$∂X∂L​与X的维度是一致的，所以首先来展开$\frac { \partial L } { \partial X }$∂X∂L​。
$X = \left( \begin{array} { l l } { x _ { 1,1 } } & { x _ { 1,2 } } \\ { x _ { 2,1 } } & { x _ { 2,2 } } \end{array} \right) \Longrightarrow \frac { \partial L } { \partial X } = \left( \begin{array} { l l } { \frac { \partial L } { \partial x _ { 1,1 } } } & { \frac { \partial L } { \partial x _ { 1,2 } } } \\ { \frac { \partial L } { \partial x _ { 2,1 } } } & { \frac { \partial L } { \partial x _ { 2,2 } } } \end{array} \right)$X=(x1,1​x2,1​​x1,2​x2,2​​)⟹∂X∂L​=(∂x1,1​∂L​∂x2,1​∂L​​∂x1,2​∂L​∂x2,2​∂L​​)先对其中一个元素$\frac { \partial L } { \partial x _ { 1,1 }}$∂x1,1​∂L​进行计算:$\frac { \partial L } { \partial x _ { 1,1 } } = \sum _ { i = 1 } ^ { N } \sum _ { j = 1 } ^ { C} \frac { \partial L } { \partial y _ { i , j } } \frac { \partial y _ { i , j } } { \partial x _ { 1,1 } } = \frac { \partial L } { \partial Y } \cdot \frac { \partial Y } { \partial x _ { 1,1 } }$∂x1,1​∂L​=i=1∑N​j=1∑C​∂yi,j​∂L​∂x1,1​∂yi,j​​=∂Y∂L​⋅∂x1,1​∂Y​ 其中$L$L、$x_{1,1}$x1,1​、$\frac { \partial L } { \partial x _ { 1,1 } }$∂x1,1​∂L​都为一个实数，$\frac { \partial Y } { \partial x _ { 1,1 } }$∂x1,1​∂Y​是一个与$Y$Y相同维度的矩阵。$\frac { \partial Y } { \partial x _ { 1,1 } } = \left( \begin{array} { c c c } { w _ { 1,1 } } & { w _ { 1,2 } } & { w _ { 1,3 } } \\ { 0 } & { 0 } & { 0 } \end{array} \right)$∂x1,1​∂Y​=(w1,1​0​w1,2​0​w1,3​0​)综上，就可以求得$\begin{aligned} \frac { \partial L } { \partial x _ { 1,1 } } & = \frac { \partial L } { \partial Y } \frac { \partial Y } { \partial x _ { 1,1 } } \\ & = \left( \begin{array} { c } { \frac { \partial L } { \partial y _ { 2,1 } } } & { \frac { \partial L } { \partial y _ { 2,2 } } } & { \frac { \partial L } { \partial y _ { 2,3 } } } \\ { \frac { \partial L } { \partial y _ { 2,1 } } } & { \frac { \partial L } { \partial y _ { 2,2 } } } & { \frac { \partial L } { \partial y _ { 2,3 } } } \end{array} \right) \cdot \left( \begin{array} { c c c } { w _ { 1,1 } } & { w _ { 1,2 } } & { w _ { 1,3 } } \\ { 0 } & { 0 } & { 0 } \end{array} \right) \\ & = \frac { \partial L } { \partial y _ { 1,1 } } w _ { 1,1 } + \frac { \partial L } { \partial y _ { 1,2 } } w _ { 1,2 } + \frac { \partial L } { \partial y _ { 1,3 } } w _ { 1,3 } \end{aligned}$∂x1,1​∂L​​=∂Y∂L​∂x1,1​∂Y​=(∂y2,1​∂L​∂y2,1​∂L​​∂y2,2​∂L​∂y2,2​∂L​​∂y2,3​∂L​∂y2,3​∂L​​)⋅(w1,1​0​w1,2​0​w1,3​0​)=∂y1,1​∂L​w1,1​+∂y1,2​∂L​w1,2​+∂y1,3​∂L​w1,3​​同理，求得$\frac { \partial L } { \partial x _ { 1,2 } }$∂x1,2​∂L​、$\frac { \partial L } { \partial x _ { 2,1 } }$∂x2,1​∂L​、$\frac { \partial L } { \partial x _ { 2,2 } }$∂x2,2​∂L​

$\begin{aligned}\frac { \partial L } { \partial X } & = \left( \begin{array} { c c } { \frac { \partial L } { \partial y _ { 1,1 } } w _ { 1,1 } + \frac { \partial L } { \partial y _ { 1,2 } } w _ { 1,2 } + \frac { \partial L } { \partial y _ { 1,3 } } w _ { 1,3 } } & { \frac { \partial L } { \partial y _ { 1 } 1 } w _ { 2,1 } + \frac { \partial L } { \partial y _ { 1,2 } } w _ { 2,2 } + \frac { \partial L } { \partial y _ { 1,3 } } w _ { 2,3 } } \\ { \frac { \partial L } { \partial y _ { 2,1 } } w _ { 1,1 } + \frac { \partial L } { \partial y _ { 2,2 } } w _ { 1,2 } + \frac { \partial L } { \partial y _ { 2,3 } } w _ { 1,3 } } & { \frac { \partial L } { \partial y _ { 2,1 } } w _ { 2,1 } + \frac { \partial L } { \partial y _ { 2,2 } } w _ { 2,2 } + \frac { \partial L } { \partial y _ { 2,3 } } w _ { 2,3 } } \end{array} \right)\\ & = \left( \begin{array} { l l l } { \frac { \partial L } { \partial y _ { 1,1 } } } & { \frac { \partial L } { \partial y _ { 1,2 } } } & { \frac { \partial L } { \partial y _ { 1,2 } } } \\ { \frac { \partial L } { \partial y _ { 2,1 } } } & { \frac { \partial L } { \partial y _ { 2,2 } } } & { \frac { \partial L } { \partial y _ { 2,3 } } } \end{array} \right) \left( \begin{array} { c c } { w _ { 1,1 } } & { w _ { 2,1 } } \\ { w _ { 1,2 } } & { w _ { 2,2 } } \\ { w _ { 1,3 } } & { w _ { 2,3 } } \end{array} \right) \\ & =\frac { \partial L } { \partial Y } W ^ { T } \end{aligned}$∂X∂L​​=(∂y1,1​∂L​w1,1​+∂y1,2​∂L​w1,2​+∂y1,3​∂L​w1,3​∂y2,1​∂L​w1,1​+∂y2,2​∂L​w1,2​+∂y2,3​∂L​w1,3​​∂y1​1∂L​w2,1​+∂y1,2​∂L​w2,2​+∂y1,3​∂L​w2,3​∂y2,1​∂L​w2,1​+∂y2,2​∂L​w2,2​+∂y2,3​∂L​w2,3​​)=(∂y1,1​∂L​∂y2,1​∂L​​∂y1,2​∂L​∂y2,2​∂L​​∂y1,2​∂L​∂y2,3​∂L​​)⎝⎛​w1,1​w1,2​w1,3​​w2,1​w2,2​w2,3​​⎠⎞​=∂Y∂L​WT​用同样的分批求导方法，可以得到$\frac { \partial L } { \partial W } = X ^ { T } \frac { \partial L } { \partial Y }$∂W∂L​=XT∂Y∂L​

两层神经网络的梯度求导

两层神经网络的结构如下图所示，outputlayer最终输出的是样本属于每一类的概率，最后的损失函数Loss可以由SVM或者Softmax求得。

将神经网络得计算过程转化为计算图如下，其中hiddenlayer中有一个**函数relu，依据计算图，使用链式求导法则，求四个参数的倒数。

其中$S_1=XW_1+b1$S1​=XW1​+b1，$S_{1relu} = relu(S_1)$S1relu​=relu(S1​)，$S_2 = S_{1relu}W_2+b2$S2​=S1relu​W2​+b2。求$\frac { \partial L} { \partial W_1}、\frac { \partial L} { \partial W_2}、\frac { \partial L} { \partial b_1}、\frac { \partial L} { \partial b_2}$∂W1​∂L​、∂W2​∂L​、∂b1​∂L​、∂b2​∂L​

• $\frac { \partial L} { \partial W_2}$∂W2​∂L​
  $\frac { \partial L} { \partial W_2} = S_{1relu}^T{\frac { \partial L} { \partial S_2} }$∂W2​∂L​=S1reluT​∂S2​∂L​
• $\frac { \partial L} { \partial b_2}$∂b2​∂L​
  $\frac { \partial L} { \partial b_2} = {\frac { \partial L} { \partial S_2} }{\frac { \partial S_2} { \partial b_2} } =\sum_i{\frac { \partial L} { \partial S_{2ij}} }$∂b2​∂L​=∂S2​∂L​∂b2​∂S2​​=i∑​∂S2ij​∂L​
• $\frac { \partial L} { \partial W_1}$∂W1​∂L​
  $\frac { \partial L} { \partial W_1} = X^T{\frac { \partial L} { \partial S_1} }$∂W1​∂L​=XT∂S1​∂L​
  其中，$\frac { \partial L} { \partial S_1} = {\frac { \partial L} { \partial S_{1relu}}} {\frac { \partial S_1} { \partial S_{1relu}}} = {\frac { \partial L} { \partial S_2}}W_2^T(S_1>0)$∂S1​∂L​=∂S1relu​∂L​∂S1relu​∂S1​​=∂S2​∂L​W2T​(S1​>0)
  $\frac { \partial L} { \partial W_1} = X^T{\frac { \partial L} { \partial S_2}}W_2^T(S_1>0)$∂W1​∂L​=XT∂S2​∂L​W2T​(S1​>0)
• $\frac { \partial L} { \partial b_1}$∂b1​∂L​
  $\frac { \partial L} { \partial b_1} = {\frac { \partial L} { \partial S_1} }{\frac { \partial S_1} { \partial b_1} } = \sum_i{\frac { \partial L} { \partial S_{1ij}} }$∂b1​∂L​=∂S1​∂L​∂b1​∂S1​​=i∑​∂S1ij​∂L​
  综上，求的四个参数的梯度之后，就可以进行梯度下降的训练过程。

classifiers/neural_net.py

def loss(self, X, y=None, reg=0.0):
    """
    Compute the loss and gradients for a two layer fully connected neural
    network.
    Inputs:
    - X: Input data of shape (N, D). Each X[i] is a training sample.
    - y: Vector of training labels. y[i] is the label for X[i], and each y[i] is
      an integer in the range 0 <= y[i] < C. This parameter is optional; if it
      is not passed then we only return scores, and if it is passed then we
      instead return the loss and gradients.
    - reg: Regularization strength.
    Returns:
    If y is None, return a matrix scores of shape (N, C) where scores[i, c] is
    the score for class c on input X[i].
    If y is not None, instead return a tuple of:
    - loss: Loss (data loss and regularization loss) for this batch of training
      samples.
    - grads: Dictionary mapping parameter names to gradients of those parameters
      with respect to the loss function; has the same keys as self.params.
    """
    # Unpack variables from the params dictionary
    W1, b1 = self.params['W1'], self.params['b1']
    W2, b2 = self.params['W2'], self.params['b2']
    N, D = X.shape

    # Compute the forward pass
    scores = None
    z1 = np.dot(X,W1)+b1
    a1 = np.maximum(0,z1)
    scores = np.dot(a1,W2)+b2
    
    # If the targets are not given then jump out, we're done
    if y is None:
      return scores

    # Compute the loss
    loss = None
    scores -=np.max(scores,axis=1).reshape(N,1)
    correct_scores = np.exp(scores[np.arange(N),y])
    sum_line = np.sum(np.exp(scores),axis=1)
    loss = np.sum(-np.log(correct_scores/sum_line))
    loss /= N
    loss += reg*(np.sum(W1**2)+np.sum(W2**2))


    # Backward pass: compute gradients
    grads = {}
    dS2 = np.exp(scores)/sum_line.reshape(N,1)
    dS2[np.arange(N),y] -=1
    grads['W2'] = np.dot(a1.T,dS2)/N + 2 * reg * W2
    grads['b2'] = np.sum(dS2, axis=0)/N
    dz1 = dS2.dot(W2.T)*(z1>0)
    grads['W1'] = np.dot(X.T,dz1)/N + 2 * reg * W1
    grads['b1'] = np.sum(dz1, axis=0) / N

    return loss, grads

参考
https://github.com/soloice/Matrix_Derivatives
http://cs231n.stanford.edu/handouts/linear-backprop.pdf
http://cs231n.stanford.edu/handouts/derivatives.pdf