BP推导 - 爱码网

图

BP推导

符号
$x_{i j}$ :第j行数据，第i个神经元输入

$w_{i j}^{l}$ ：第l层的第i个神经元的到第l+1层的第j个神经元的权重

$b_{l}$ ：第l层的偏置bias

$z_{i}^{l}$ ：第l层第i个神经元的的输入

$a_{i}^{l} $ ：第I层第i个神经元的输出

$σ ()$ ：**函数

$δ_{i}^{l}$ ：第I层第i个神经元的的误差

推导

向前传播

第一层

$a_{1}^{(1)} = x_{1 j}$

$a_{2}^{(1)} = x_{2 j}$

第二层

$z_{1}^{(2)} = a_{1}^{(1)} w_{1}^{(1)} + a_{2}^{(1)} w_{3}^{(1)} + b_{1} = x_{1 j} w_{1}^{(1)} + x_{2 j} w_{3}^{(1)} + b_{1}$

$z_{2}^{(2)} = a_{1}^{(1)} w_{2}^{(1)} + a_{2}^{(1)} w_{4}^{(1)} + b_{1} = x_{1 j} w_{2}^{(1)} + x_{2 j} w_{4}^{(1)} + b_{1}$

$a_{1}^{(2)} = σ (z_{1}^{(2)}) = σ (x_{1 j} w_{1}^{(1)} + x_{2 j} w_{3}^{(1)} + b_{1})$

$a_{2}^{(2)} = σ (z_{2}^{(2)}) = σ (x_{1 j} w_{2}^{(1)} + x_{2 j} w_{4}^{(1)} + b_{1})$

第三层

$z_{1}^{(3)} = a_{1}^{(2)} w_{1}^{(2)} + a_{2}^{(2)} w_{3}^{(2)} + b_{2} = σ (x_{1 j} w_{1}^{(1)} + x_{2 j} w_{3}^{(1)} + b_{1}) w_{1}^{(2)} + σ (x_{1 j} w_{2}^{(1)} + x_{2 j} w_{4}^{(1)} + b_{1}) w_{3}^{(2)} + b_{2}$

$z_{2}^{(3)} = a_{1}^{(2)} w_{2}^{(2)} + a_{2}^{(2)} w_{4}^{(2)} + b_{2} = σ (x_{1 j} w_{1}^{(1)} + x_{2 j} w_{3}^{(1)} + b_{1}) w_{2}^{(2)} + σ (x_{1 j} w_{2}^{(1)} + x_{2 j} w_{4}^{(1)} + b_{1}) w_{4}^{(2)} + b_{2}$

$a_{1}^{(3)} = σ (z_{1}^{(3)}) = σ (σ (x_{1 j} w_{1}^{(1)} + x_{2 j} w_{3}^{(1)} + b_{1}) w_{1}^{(2)} + σ (x_{1 j} w_{2}^{(1)} + x_{2 j} w_{4}^{(1)} + b_{1}) w_{3}^{(2)} + b_{2})$

$a_{2}^{(3)} = σ (z_{2}^{(3)}) = σ (σ (x_{1 j} w_{1}^{(1)} + x_{2 j} w_{3}^{(1)} + b_{1}) w_{2}^{(2)} + σ (x_{1 j} w_{2}^{(1)} + x_{2 j} w_{4}^{(1)} + b_{1}) w_{4}^{(2)} + b_{2})$

反向传播

对每一层的残差求导，求梯度，更新权重。

w=w-\alpha {{\partial E}\over \partial w}

对第三层-第二层权重更新

$E_{t o a t a l} = E_{1}^{(3)} + E_{2}^{(3)}$ = $\frac{1}{2} (y - a_{1}^{(3)})^{2} + \frac{1}{2} (y - a_{2}^{(3)})^{2} = \frac{1}{2} (y - [σ (z_{1}^{3})]])^{2} + \frac{1}{2} (y - [σ (z_{2}^{(3)})])^{2}$

对 $w_{1}^{2}$ 取值更新，也就是先对 $w_{1}^{2}$ 求导,通过链式法则得到

$\frac{\partial E_{t o t a l}}{\partial w_{1}^{(2)}} = \frac{\partial [\frac{1}{2} (y - a_{1}^{(3)})^{2} + \frac{1}{2} (y - a_{2}^{(3)})^{2}]}{\partial a_{1}^{(3)}} * \frac{\partial [σ (z_{1}^{(3)})]}{\partial z_{1}^{(3)}} * \frac{\partial (a_{1}^{(2)} w_{1}^{(2)} + a_{2}^{(2)} w_{3}^{(2)} + b_{2})}{\partial w_{1}^{(2)}}$

$= (a_{1}^{(3)} - y) * σ (z_{1}^{(3)})^{'} * a_{1}^{(2)}$

对第二层-第一层权重更新

$\frac{\partial E_{t o t a l}}{\partial w_{1}^{(1)}} = \frac{\partial E_{t o t a l}}{\partial a_{1}^{(2)}} * \frac{\partial σ (z_{1}^{(2)})}{\partial z_{1}^{(2)}} * \frac{\partial (a_{1}^{(1)} w_{1}^{(1)} + a_{2}^{(1)} w_{3}^{(1)} + b_{1})}{\partial w_{1}^{(1)}}$

$\frac{\partial E_{t o t a l}}{\partial a_{1}^{(2)}} = \frac{E_{1}^{(3)}}{\partial a_{1}^{(2)}} + \frac{E_{1}^{(3)}}{\partial a_{1}^{(2)}} = \frac{\partial [\frac{1}{2} (y - a_{1}^{(3)})^{2}]}{\partial a_{1}^{(3)}} * \frac{\partial [σ (z_{1}^{(3)})]}{\partial z_{1}^{(3)}} * \frac{\partial (a_{1}^{(2)} w_{1}^{(2)} + a_{2}^{(2)} w_{3}^{(2)} + b_{2})}{\partial a_{1}^{(2)}} + \frac{\partial [\frac{1}{2} (y - a_{2}^{(3)})^{2}]}{\partial a_{2}^{(3)}} * \frac{\partial [σ (z_{2}^{(3)})]}{\partial z_{2}^{(3)}} * \frac{\partial (a_{1}^{(2)} w_{2}^{(2)} + a_{2}^{(2)} w_{4}^{(2)} + b_{2})}{\partial a_{1}^{(2)}}$

$= (a_{1}^{(3)} - y) * σ (z_{1}^{(3)})^{'} * w_{1}^{(2)} + (a_{2}^{(3)} - y) * σ (z_{2}^{(3)})^{'} * w_{2}^{(2)}$

$= [(a_{1}^{(3)} - y) * σ (z_{1}^{(3)})^{'} * w_{1}^{(2)} + (a_{2}^{(3)} - y) * σ (z_{2}^{(3)})^{'} * w_{2}^{(2)}] * σ (z_{1}^{(2)})^{'} a_{1}^{(1)}$