Deep learning I - III Shallow Neural Network - Backpropagation intuition反向传播算法启发

Backpropagation intuition

---

简单的2层浅神经网络，第一层的activation function为tanh(z)$tanh(z)$，第二层的activation function为sigmoid(z)$sigmoid(z)$。
神经网络architecture如下图：

使用计算流图(computational graphs)表示如下图：

在下面的公式中,loga[2] means lna[2]$\log a^{[2]}\mathrm{m}\mathrm{e}\mathrm{a}\mathrm{n}\mathrm{s}\ln a^{[2]}$；da[2],dz[2]$\mathrm{d}{a}^{[2]},\mathrm{d}{z}^{[2]}$等等是标记相应的导数的符号；并且，下面的公式是单个instance的，并没有矩阵化。

L(a[2],y)=−yloga[2]−(1−y)log(1−a[2])(1.1)$$\begin{array}{}\text{(1.1)}& \mathcal{L}(a^{[2]},y)=-y\log a^{[2]}-(1-y)\log (1-a^{[2]})\end{array}$$

da[2][1×1]=dda[2]L(a[2],y)=−ya[2]+1−y1−a[2](1.2)$$\begin{array}{}\text{(1.2)}& \mathrm{d}{a}_{[1\times 1]}^{[2]}=\frac{\mathrm{d}}{\mathrm{d}{a}^{[2]}}\mathcal{L}(a^{[2]},y)=-\frac{y}{a^{[2]}}+\frac{1-y}{1-a^{[2]}}\end{array}$$

g(z[2])=sigmoid(z[2])=a[2](1.3)$$\begin{array}{}\text{(1.3)}& g(z^{[2]})=sigmoid(z^{[2]})=a^{[2]}\end{array}$$

dz[2][1×1]=ddz[2]L(a[2],y)=dda[2]L(a[2],y)⋅ddz[2]a[2]=da[2]⋅g′(z[2])=(−ya[2]+1−y1−a[2])⋅(g(z[2])(1−g(z[2])))=(−ya[2]+1−y1−a[2])⋅a[2]⋅(1−a[2])=a[2]−y(1.4)$$\begin{array}{}\text{(1.4)}& \begin{array}{rl}\mathrm{d}{z}_{[1\times 1]}^{[2]}& =\frac{\mathrm{d}}{\mathrm{d}{z}^{[2]}}\mathcal{L}(a^{[2]},y)\\ & =\frac{\mathrm{d}}{\mathrm{d}{a}^{[2]}}\mathcal{L}(a^{[2]},y)\cdot \frac{\mathrm{d}}{\mathrm{d}{z}^{[2]}}{a}^{[2]}\\ & =\mathrm{d}{a}^{[2]}\cdot g^{\prime }(z^{[2]})\\ & =(-\frac{y}{a^{[2]}}+\frac{1-y}{1-a^{[2]}})\cdot (g(z^{[2]})(1-g(z^{[2]})))\\ & =(-\frac{y}{a^{[2]}}+\frac{1-y}{1-a^{[2]}})\cdot a^{[2]}\cdot (1-a^{[2]})\\ & =a^{[2]}-y\end{array}\end{array}$$

dW[2][1×4]=ddW[2]L(a[2],y)=dda[2]L(a[2],y)⋅ddz[2]a[2]⋅ddW[2]z[2]=dz[2]⋅x=dz[2][1×1](a[1][4×1])T(1.5)$$\begin{array}{}\text{(1.5)}& \begin{array}{rl}\mathrm{d}{W}_{[1\times 4]}^{[2]}& =\frac{\mathrm{d}}{\mathrm{d}{W}^{[2]}}\mathcal{L}(a^{[2]},y)\\ & =\frac{\mathrm{d}}{\mathrm{d}{a}^{[2]}}\mathcal{L}(a^{[2]},y)\cdot \frac{\mathrm{d}}{\mathrm{d}{z}^{[2]}}{a}^{[2]}\cdot \frac{\mathrm{d}}{\mathrm{d}{W}^{[2]}}{z}^{[2]}\\ & =\mathrm{d}{z}^{[2]}\cdot x\\ & =\mathrm{d}{z}_{[1\times 1]}^{[2]}(a_{[4\times 1]}^{[1]}{)}^T\end{array}\end{array}$$

db[2][1×1]=ddb[2]L(a[2],y)=dda[2]L(a[2],y)⋅ddz[2]a[2]⋅ddb[2]z[2]=dz[2][1×1](1.6)$$\begin{array}{}\text{(1.6)}& \begin{array}{rl}\mathrm{d}{b}_{[1\times 1]}^{[2]}& =\frac{\mathrm{d}}{\mathrm{d}{b}^{[2]}}\mathcal{L}(a^{[2]},y)\\ & =\frac{\mathrm{d}}{\mathrm{d}{a}^{[2]}}\mathcal{L}(a^{[2]},y)\cdot \frac{\mathrm{d}}{\mathrm{d}{z}^{[2]}}{a}^{[2]}\cdot \frac{\mathrm{d}}{\mathrm{d}{b}^{[2]}}{z}^{[2]}\\ & =\mathrm{d}{z}_{[1\times 1]}^{[2]}\end{array}\end{array}$$

da[1][4×1]=dda[1]L(a[2],y)=dda[2]L(a[2],y)⋅ddz[2]a[2]⋅dda[1]z[2]=dz[2]⋅W[2]=(W[2][1×4])Tdz[2][1×1](1.7)$$\begin{array}{}\text{(1.7)}& \begin{array}{rl}\mathrm{d}{a}_{[4\times 1]}^{[1]}& =\frac{\mathrm{d}}{\mathrm{d}{a}^{[1]}}\mathcal{L}(a^{[2]},y)\\ & =\frac{\mathrm{d}}{\mathrm{d}{a}^{[2]}}\mathcal{L}(a^{[2]},y)\cdot \frac{\mathrm{d}}{\mathrm{d}{z}^{[2]}}{a}^{[2]}\cdot \frac{\mathrm{d}}{\mathrm{d}{a}^{[1]}}{z}^{[2]}\\ & =\mathrm{d}{z}^{[2]}\cdot W^{[2]}\\ & =(W_{[1\times 4]}^{[2]}{)}^T\mathrm{d}{z}_{[1\times 1]}^{[2]}\end{array}\end{array}$$

g(z[1])=tanh(z[1])=a[1](1.8)$$\begin{array}{}\text{(1.8)}& g(z^{[1]})=\tanh (z^{[1]})=a^{[1]}\end{array}$$

dz[1][4×1]=ddz[1]L(a[2],y)=dda[2]L(a[2],y)⋅ddz[2]a[2]⋅dda[1]z[2]⋅ddz[1]a[1]=da[1]⋅g′(z[1])=(W[2][1×4])Tdz[2][1×1]∗g′(z[1])[4×1](1.9)$$\begin{array}{}\text{(1.9)}& \begin{array}{rl}\mathrm{d}{z}_{[4\times 1]}^{[1]}& =\frac{\mathrm{d}}{\mathrm{d}{z}^{[1]}}\mathcal{L}(a^{[2]},y)\\ & =\frac{\mathrm{d}}{\mathrm{d}{a}^{[2]}}\mathcal{L}(a^{[2]},y)\cdot \frac{\mathrm{d}}{\mathrm{d}{z}^{[2]}}{a}^{[2]}\cdot \frac{\mathrm{d}}{\mathrm{d}{a}^{[1]}}{z}^{[2]}\cdot \frac{\mathrm{d}}{\mathrm{d}{z}^{[1]}}{a}^{[1]}\\ & =\mathrm{d}{a}^{[1]}\cdot g^{\prime }(z^{[1]})\\ & =(W_{[1\times 4]}^{[2]}{)}^T\mathrm{d}{z}_{[1\times 1]}^{[2]}\ast g^{\prime }(z^{[1]}{)}_{[4\times 1]}\end{array}\end{array}$$

dW[1][4×3]=ddW[1]L(a[2],y)=dda[2]L(a[2],y)⋅ddz[2]a[2]⋅dda[1]z[2]⋅ddz[1]a[1]⋅ddW[1]z[1]=dz[1]⋅x=dz[1][4×1](a[0][3×1])T(1.10)$$\begin{array}{}\text{(1.10)}& \begin{array}{rl}\mathrm{d}{W}_{[4\times 3]}^{[1]}& =\frac{\mathrm{d}}{\mathrm{d}{W}^{[1]}}\mathcal{L}(a^{[2]},y)\\ & =\frac{\mathrm{d}}{\mathrm{d}{a}^{[2]}}\mathcal{L}(a^{[2]},y)\cdot \frac{\mathrm{d}}{\mathrm{d}{z}^{[2]}}{a}^{[2]}\cdot \frac{\mathrm{d}}{\mathrm{d}{a}^{[1]}}{z}^{[2]}\cdot \frac{\mathrm{d}}{\mathrm{d}{z}^{[1]}}{a}^{[1]}\cdot \frac{\mathrm{d}}{\mathrm{d}{W}^{[1]}}{z}^{[1]}\\ & =\mathrm{d}{z}^{[1]}\cdot x\\ & =\mathrm{d}{z}_{[4\times 1]}^{[1]}(a_{[3\times 1]}^{[0]}{)}^T\end{array}\end{array}$$

db[1][4×1]=ddW[1]L(a[2],y)=dda[2]L(a[2],y)⋅ddz[2]a[2]⋅dda[1]z[2]⋅ddz[1]a[1]⋅ddb[1]z[1]=dz[1][4×1](1.11)$$\begin{array}{}\text{(1.11)}& \begin{array}{rl}\mathrm{d}{b}_{[4\times 1]}^{[1]}& =\frac{\mathrm{d}}{\mathrm{d}{W}^{[1]}}\mathcal{L}(a^{[2]},y)\\ & =\frac{\mathrm{d}}{\mathrm{d}{a}^{[2]}}\mathcal{L}(a^{[2]},y)\cdot \frac{\mathrm{d}}{\mathrm{d}{z}^{[2]}}{a}^{[2]}\cdot \frac{\mathrm{d}}{\mathrm{d}{a}^{[1]}}{z}^{[2]}\cdot \frac{\mathrm{d}}{\mathrm{d}{z}^{[1]}}{a}^{[1]}\cdot \frac{\mathrm{d}}{\mathrm{d}{b}^{[1]}}{z}^{[1]}\\ & =\mathrm{d}{z}_{[4\times 1]}^{[1]}\end{array}\end{array}$$

下面是vectorization后的反向传播算法公式：

L(A[2],Y)=1m∑i=1m−y(i)logA[2](i)−(1−y(i))log(1−A[2](i))(2.1)$$\begin{array}{}\text{(2.1)}& \mathcal{L}(A^{[2]},Y)=\frac{1}{m}\sum _{i=1}^m-y^{(i)}\log A^{[2](i)}-(1-y^{(i)})\log (1-A^{[2](i)})\end{array}$$

dA[2][1×m]=[(−Y(1)A[2](1)+1−Y(1)1−A[2](1)),⋯,(−Y(m)A[2](m)+1−Y(m)1−A[2](m))](2.2)$$\begin{array}{}\text{(2.2)}& \begin{array}{rl}\mathrm{d}{A}_{[1\times m]}^{[2]}& =[(-\frac{Y^{(1)}}{A^{[2](1)}}+\frac{1-Y^{(1)}}{1-A^{[2](1)}}),\cdots ,(-\frac{Y^{(m)}}{A^{[2](m)}}+\frac{1-Y^{(m)}}{1-A^{[2](m)}})]\end{array}\end{array}$$

dZ[2][1×m]=[(−Y(1)A[2](1)+1−Y(1)1−A[2](1)),⋯,(−Y(m)A[2](m)+1−Y(m)1−A[2](m))]∗[A[2](1)(1−A[2](1)),⋯,A[2](m)(1−A[2](m))]=[(A[2](1)−Y(1)),⋯,(A[2](m)−Y(m))]=A[2]−Y(2.3)$$\begin{array}{}\text{(2.3)}& \begin{array}{rl}\mathrm{d}{Z}_{[1\times m]}^{[2]}& =[(-\frac{Y^{(1)}}{A^{[2](1)}}+\frac{1-Y^{(1)}}{1-A^{[2](1)}}),\cdots ,(-\frac{Y^{(m)}}{A^{[2](m)}}+\frac{1-Y^{(m)}}{1-A^{[2](m)}})]\ast [A^{[2](1)}(1-A^{[2](1)}),\cdots ,A^{[2](m)}(1-A^{[2](m)})]\\ & =[(A^{[2](1)}-Y^{(1)}),\cdots ,(A^{[2](m)}-Y^{(m)})]\\ & =A^{[2]}-Y\end{array}\end{array}$$

dW[2][1×4]=1mdZ[2][1×m](A[1][4×m])T(2.4)$$\begin{array}{}\text{(2.4)}& \mathrm{d}{W}_{[1\times 4]}^{[2]}=\frac{1}{m}\mathrm{d}{Z}_{[1\times m]}^{[2]}(A_{[4\times m]}^{[1]}{)}^T\end{array}$$

db[2][1×1]=1mnp.sum(dZ[2],axis=1,keepdims=True)(2.5)$$\begin{array}{}\text{(2.5)}& \mathrm{d}{b}_{[1\times 1]}^{[2]}=\frac{1}{m}np.sum(\mathrm{d}{Z}^{[2]},axis=1,keepdims=True)\end{array}$$

dZ[1][4×m]=(W[2][1×4])TdZ[2][1×m]∗g[1]′(Z[1])[4×m]$$\mathrm{d}{Z}_{[4\times m]}^{[1]}=(W_{[1\times 4]}^{[2]}{)}^T\mathrm{d}{Z}_{[1\times m]}^{[2]}\ast g^{[1]}{}^{\prime }(Z^{[1]}{)}_{[4\times m]}$$

dW[1][4×3]=1mdZ[1][4×m](A[0][3×m])T(2.6)$$\begin{array}{}\text{(2.6)}& \mathrm{d}{W}_{[4\times 3]}^{[1]}=\frac{1}{m}\mathrm{d}{Z}_{[4\times m]}^{[1]}(A_{[3\times m]}^{[0]}{)}^T\end{array}$$

db[1][4×1]=1msp.sum(dZ[1][4×m],axis=1,keepdims=True)(2.7)$$\begin{array}{}\text{(2.7)}& \mathrm{d}{b}_{[4\times 1]}^{[1]}=\frac{1}{m}sp.sum(\mathrm{d}{Z}_{[4\times m]}^{[1]},axis=1,keepdims=True)\end{array}$$

总结