BP神经网络推导

示意图

BP神经网络推导

符号说明

$\begin{aligned} \boldsymbol{y}^{0}: & 输入，\boldsymbol{y}\in \mathbb{R}^{s0\times1} \\ \boldsymbol{z}^{l}: &第l层输出\boldsymbol{z}^{(l)} \in \mathbb{R}^{sl \times 1} \\ \boldsymbol{y}^{l}:&第l层输出\boldsymbol{y}^{(l)} \in \mathbb{R}^{sl \times 1} \\ \boldsymbol{\sigma}:&**函数\\ sl:& 表示l层 \boldsymbol{y}^{(l)} \boldsymbol{z}^{(l)}的向量维数 \\ \boldsymbol{t}: &表示真实值 \\ L:&一共L层 \\ f^l_{i}:& 表示\frac{\partial{y^l_i}}{\partial{z^l_i}} \\ \boldsymbol{I}(i):&表示为列向量，且在第i行为1，其余位置为0; \\ \delta^l_i: &表示 \frac{\partial{E}}{\partial{y^l_i}} \\ \boldsymbol{\delta}^l: &表示\frac{\partial{E}}{\partial{\boldsymbol{y}^l}} ，即为： \begin{pmatrix} \delta^l_1 ,\delta^l_2 ,\cdots,\delta^l_{sl} \end{pmatrix} \end{aligned}$
它们之间的关系：
$\begin{aligned} \boldsymbol{z}^{l} &=\boldsymbol{w}^{l}*\boldsymbol{y}^{l-1}\\ z^{l}_i &= \sum_{j=1}^{s(l-1)}w_{ij}*y^{l-1}_j \\ \boldsymbol{y}^{l} &=\boldsymbol{\sigma}(\boldsymbol{y}^{l}) \\ y^{l}_i &= \sigma(z^l_i) \end{aligned} f(\boldsymbol{x})$

矩阵相关求导说明

符号说明
$\boldsymbol{y}:列向量,\boldsymbol{y} \in \mathbb{R}^{n \times 1}$
$\boldsymbol{x}:列向量,\boldsymbol{x} \in \mathbb{R}^{m \times 1}$
$f(\boldsymbol{x}):实值标量函数，记做 f: \mathbb{R}^m \to \mathbb{R}$
公式
$\begin{aligned} \frac{\partial{\boldsymbol{y}^ \mathrm{T}}}{\partial{\boldsymbol{x}}} &= \begin{pmatrix} \frac{\partial{y_1}}{\partial{x_1}} & \cdots & \frac{\partial{y_n}}{\partial{x_1}} \\ \vdots & & \vdots \\ \frac{\partial{y_1}}{\partial{x_m}} & \cdots & \frac{\partial{y_n}}{\partial{x_m}} \end{pmatrix} \\ \frac{\partial{\boldsymbol{y}^ \mathrm{T}}}{\partial{\boldsymbol{y}}} &= \mathbf{E}_{n \times n} \\ \frac{f(\boldsymbol{x})}{\partial{\boldsymbol{x}}} &= [ \frac{f(\boldsymbol{x})}{\partial{x_1}} , \cdots ,\frac{f(\boldsymbol{x})}{\partial{x_m}}]^{\mathrm{T}} \end{aligned}$

公式推导

误差定义
$\begin{aligned} E &=\frac{1}{m}\sum_{p=1}^{m}(E_p) \\ E_p &= \frac{1}{2}(\boldsymbol{y}^L - \boldsymbol{t}^L)^2 \\ &=\frac{1}{2}\sum_{i=1}^{sL}(y^L_i - t_i)^2 \end{aligned}$

其中m为样本数，为了推导简单，让m=1
求 $\frac{\partial{E}}{\partial{w^L_{ij}}}$
几点说明
$\begin{aligned} \frac{\partial{z_k^l}}{\partial{w^l_{ij}}} &= \begin{cases} z^{l-1}_j & z = i \\ 0 & k \ne i \end{cases} \\ \frac{\partial{\boldsymbol{z}^l } }{\partial{w_{ij}}} &= [\frac{\partial{z_1^l}}{\partial{w_{ij}^l}} ,\cdots,\frac{\partial{z_{sl}^l}}{\partial{w_{ij}^l}}]^{\mathrm{T}} \in \mathbb{R}^{sl \times 1} \\ &=\boldsymbol{I}(i).z_i^{l-1} \\ \\ \frac{\partial{\boldsymbol{y}^l}}{\partial{(\boldsymbol{z}^{l})^{\mathrm{T}}}} &= \begin{pmatrix} \frac{\partial{y_1^l}}{\partial{z_1^l}}& \cdots & \frac{\partial{y_1^l}}{\partial{z_{sl}^l}} \\ \vdots & & \vdots \\ \frac{\partial{y_{sl}^l}}{\partial{z_1^l}}& \cdots & \frac{\partial{y_{sl}^l}}{\partial{z_{sl}^l}} \end{pmatrix} \in \mathbb{R}^{sl \times sl} \\ &= \begin{pmatrix} f^l_1& & & \\ &f^l_2 \\ & &\ddots \\ & & &f^l_{(sl)} \end{pmatrix} \\ \\ \frac{\partial{\boldsymbol{z}^{l}}}{\partial{(\boldsymbol{y}^{l-1})^{\mathrm{T}}}}&= \begin{pmatrix} \frac{\partial{z_1^l}}{\partial{y_1^{l-1}}}& \cdots & \frac{\partial{z_1^l}}{\partial{y_{s(l-1)}^{l-1}}} \\ \vdots & & \vdots \\ \frac{\partial{z_{sl}^l}}{\partial{y_1^{l-1}}}& \cdots & \frac{\partial{z_{sl}^l}}{\partial{y_{s(l-1)}^{(l-1)}}} \end{pmatrix} \\ &= \begin{pmatrix} w_{11}^l& \cdots & w_{(s(l-1))1}^l \\ \vdots & & \vdots \\ w_{(sl)1}^l& \cdots & w_{(sl)(s(l-1))}^l \end{pmatrix} \in \mathbb{R}^{sl \times s(l-1)} \\ \\ \frac{\partial{\boldsymbol{y}^l}}{\partial{\boldsymbol{y}^{l-1}}} &= \frac{\partial{\boldsymbol{y}^l}}{\partial{\boldsymbol{z}^{l}}} .\frac{\partial{\boldsymbol{z}^l}}{\partial{\boldsymbol{y}^{l-1}}} \\ &=\frac{\partial{\boldsymbol{y}^l}}{\partial{(\boldsymbol{z}^{l})^{\mathrm{T}}}} . \frac{\partial{(\boldsymbol{z}^{l})^{\mathrm{T}}}}{\partial{\boldsymbol{z}^{l}}} . \frac{\partial{\boldsymbol{z}^{l}}}{\partial{(\boldsymbol{y}^{l-1})^{\mathrm{T}}}} . \frac{\partial{(\boldsymbol{y}^{l-1})^{\mathrm{T}}}}{\partial{\boldsymbol{y}^{l-1}}} \\ &= \frac{\partial{\boldsymbol{y}^l}}{\partial{(\boldsymbol{z}^{l})^{\mathrm{T}}}} .\frac{\partial{\boldsymbol{z}^{l}}}{\partial{(\boldsymbol{y}^{l-1})^{\mathrm{T}}}} \\ &= \begin{pmatrix} f^l_1w_{11}^l & \cdots & f^l_1w_{(s(l-1))1} \\ \vdots & & \vdots \\ f^l_{sl}w_{(sl)1}^l& \cdots & f^l_{sl}w_{(sl)(s(l-1))}^l \end{pmatrix} \in \mathbb{R^{sl \times s(l-1)}} \\ \\ \frac{\partial{E}}{\partial{(\boldsymbol{y}^L)^{\mathrm{T}}}} &=[y^L_1 - t_1,\cdots, y^L_{sL} - t_{sL}] \\ \\ \frac{\partial{\boldsymbol{z}^l}}{\partial{z^{l-1}_i}} &=[w^l_{1i},w^l_{2i},\cdots,w^l_{(sl)i}]^{\mathrm{T}} \\ \\ \frac{\partial{\boldsymbol{y}^l}}{\partial{y^{l-1}_i}} &= \frac{\partial{\boldsymbol{y}^l}}{\partial{\boldsymbol{z}^l}}. \frac{\partial{\boldsymbol{z}^l}}{\partial{y^{l-1}_i}} \\ &= \begin{pmatrix} f^l_1& & & \\ &f^l_2 \\ & &\ddots \\ & & &f^l_{(sl)} \end{pmatrix}. \begin{pmatrix} w^l_{1i} \\ w^l_{2i} \\ \cdots \\ w^l_{(sl)i} \end{pmatrix} \\ &= \begin{pmatrix} f^l_1w^l_{1i} \\ f^l_2 w^l_{2i} \\ \vdots \\ f^l_{(sl)} w^l_{(sl)i} \end{pmatrix} \end{aligned}$ ∂wijl∂zkl∂wij∂zl∂(zl)T∂yl∂(yl−1)T∂zl∂yl−1∂yl∂(yL)T∂E∂zil−1∂zl∂yil−1∂yl={zjl−10z=ik̸=i=[∂wijl∂z1l,⋯,∂wijl∂zsll]T∈Rsl×1=I(i).zil−1=⎝⎜⎜⎜⎛∂z1l∂y1l⋮∂z1l∂ysll⋯⋯∂zsll∂y1l⋮∂zsll∂ysll⎠⎟⎟⎟⎞∈Rsl×sl=⎝⎜⎜⎛f1lf2l⋱f(sl)l⎠⎟⎟⎞=⎝⎜⎜⎜⎜⎛∂y1l−1∂z1l⋮∂y1l−1∂zsll⋯⋯∂ys(l−1)l−1∂z1l⋮∂ys(l−1)(l−1)∂zsll⎠⎟⎟⎟⎟⎞=⎝⎜⎛w11l⋮w(sl)1l⋯⋯w(s(l−1))1l⋮w(sl)(s(l−1))l⎠⎟⎞∈Rsl×s(l−1)=∂zl∂yl.∂yl−1∂zl=∂(zl)T∂yl.∂zl∂(zl)T.∂(yl−1)T∂zl.∂yl−1∂(yl−1)T=∂(zl)T∂yl.∂(yl−1)T∂zl=⎝⎜⎛f1lw11l⋮fsllw(sl)1l⋯⋯f1lw(s(l−1))1⋮fsllw(sl)(s(l−1))l⎠⎟⎞∈Rsl×s(l−1)=[y1L−t1,⋯,ysLL−tsL]=[w1il,w2il,⋯,w(sl)il]T=∂zl∂yl.∂yil−1∂zl=⎝⎜⎜⎛f1lf2l⋱f(sl)l⎠⎟⎟⎞.⎝⎜⎜⎛w1ilw2il⋯w(sl)il⎠⎟⎟⎞=⎝⎜⎜⎜⎛f1lw1ilf2lw2il⋮f(sl)lw(sl)il⎠⎟⎟⎟⎞

求解
$\begin{aligned} \frac{\partial{E}}{\partial{w^L_{ij}}} &= \frac{\partial{E}}{\partial{\boldsymbol{y}^L}} . \frac{\partial{\boldsymbol{y}^L}}{\partial{w^L_{ij}}} \\ &=\frac{\partial{E}}{\partial{(\boldsymbol{y}^L)^{\mathrm{T}}}} . \frac{\partial{(\boldsymbol{y}^L)^{\mathrm{T}}}}{\partial{\boldsymbol{y}^L}} . \frac{\partial{\boldsymbol{y}^L}}{\partial{w^L_{ij}}} . \\ &= \begin{pmatrix} y^L_1 - t_1,\cdots, y^L_{sL} - t_{sL} \end{pmatrix} . \boldsymbol{I}(i).z_i^{L-1} \\ \\ \frac{\partial{E}}{\partial{w^{L-1}_{ij}}} &= \frac{\partial{E}}{\partial{\boldsymbol{y}^L}} . \frac{\partial{\boldsymbol{y}^L}}{\partial{\boldsymbol{y}^{L-1}}}. \frac{\partial{\boldsymbol{y}^{L-1}}}{\partial{w^{L-1}_{ij}}} \\ &= \begin{pmatrix} y^L_1 - t_1,\cdots, y^L_{sL} - t_{sL} \end{pmatrix} . \begin{pmatrix} f^L_1w_{11}^L & \cdots & f^L_1w_{(s(L-1))1} \\ \vdots & & \vdots \\ f^L_{sL}w_{(sL)1}^L& \cdots & f^L_{sL}w_{(sL)(s(L-1))}^L \end{pmatrix} . \boldsymbol{I}(i).z_i^{l-1} \\ &= \sum_{k=1}^{sl}(y_k^L-t_k) f_1^L w^L_{ki}z^{L-1}_j \end{aligned}$ ∂wijL∂E∂wijL−1∂E=∂yL∂E.∂wijL∂yL=∂(yL)T∂E.∂yL∂(yL)T.∂wijL∂yL.=(y1L−t1,⋯,ysLL−tsL).I(i).ziL−1=∂yL∂E.∂yL−1∂yL.∂wijL−1∂yL−1=(y1L−t1,⋯,ysLL−tsL).⎝⎜⎛f1Lw11L⋮fsLLw(sL)1L⋯⋯f1Lw(s(L−1))1⋮fsLLw(sL)(s(L−1))L⎠⎟⎞.I(i).zil−1=k=1∑sl(ykL−tk)f1LwkiLzjL−1
另一种定义方法
$\begin{aligned} \frac{\partial{E}}{\partial{w^L_{ij}}} &= \frac{\partial{E}}{\partial{y^L_i}} . \frac{\partial{y^L_i}}{\partial{w^L_{ij}}} \\ &= (y^L_i - h_i) z_i^{L-1} \\ \\ \frac{\partial{E}}{\partial{w^{L-1}_{ij}}} &= \frac{\partial{E}}{\partial{\boldsymbol{y}^L}} . \frac{\partial{\boldsymbol{y}^L}}{\partial{y^{L-1}_i}}. \frac{\partial{y^{L-1}_i}}{\partial{w^{L-1}_{ij}}} \\ &= \begin{pmatrix} y^L_1 - t_1,\cdots, y^L_{sL} - t_{sL} \end{pmatrix} . \begin{pmatrix} f^L_1w^L_{1i} \\ f^L_2 w^L_{2i} \\ \vdots \\ f^L_{(sL)} w^L_{(sL)i} \end{pmatrix} .z_i^{L-1} \\ &= \sum_{k=1}^{sl}(y_k^L-t_k) f_1^L w^L_{ki}z^{L-1}_j \\ \\ \delta^{l-1}_i&=\frac{\partial{E}}{\partial{y^{l-1}_i}} \\ &=\frac{\partial{E}}{\partial{\boldsymbol{y}^{l}}}. \frac{\partial{\boldsymbol{y}^{l}}}{\partial{y^{l-1}_i}} \\ &= \begin{pmatrix} \delta^l_1 ,\delta^l_2 ,\cdots,\delta^l_{sl} \end{pmatrix}. \begin{pmatrix} f^l_1w^l_{1i} \\ f^l_2 w^l_{2i} \\ \vdots \\ f^l_{(sl)} w^l_{(sl)i} \end{pmatrix} \\ &=\sum_{k=1}^{sl}\delta^l_k f^l_k w^l_{(sl)i} \\ \\ \frac{\partial{E}}{\partial{w^L_{ij}}} &= \frac{\partial{E}}{\partial{y^L_i}} . \frac{\partial{y^L_i}}{\partial{w^L_{ij}}} \\ &= \delta^L_i z_i^{L-1} \\ &= (y^L_i - h_i) z_i^{L-1} \\ \\ \frac{\partial{E}}{\partial{w^{l}_{ij}}} &= \frac{\partial{E}}{\partial{\boldsymbol{y}^l}} . \frac{\partial{y^{l}_i}}{\partial{w^{l}_{ij}}} \\ &=\delta^{l}_i z_i^{l-1} \\ &=\sum_{k=1}^{s(l+1)} \delta^{(l+1)}_k f^{l+1}_k w^{l+1}_{ki}z_j^{l-1} \end{aligned}$ ∂wijL∂E∂wijL−1∂Eδil−1∂wijL∂E∂wijl∂E=∂yiL∂E.∂wijL∂yiL=(yiL−hi)ziL−1=∂yL∂E.∂yiL−1∂yL.∂wijL−1∂yiL−1=(y1L−t1,⋯,ysLL−tsL).⎝⎜⎜⎜⎛f1Lw1iLf2Lw2iL⋮f(sL)Lw(sL)iL⎠⎟⎟⎟⎞.ziL−1=k=1∑sl(ykL−tk)f1LwkiLzjL−1=∂yil−1∂E=∂yl∂E.∂yil−1∂yl=(δ1l,δ2l,⋯,δsll).⎝⎜⎜⎜⎛f1lw1ilf2lw2il⋮f(sl)lw(sl)il⎠⎟⎟⎟⎞=k=1∑slδklfklw(sl)il=∂yiL∂E.∂wijL∂yiL=δiLziL−1=(yiL−hi)ziL−1=∂yl∂E.∂wijl∂yil=δilzil−1=k=1∑s(l+1)δk(l+1)fkl+1wkil+1zjl−1