Deep learning II - I Practical aspects of deep learning - Regularizing your neural network 神经网络范数正则化

Regularizing your neural network 神经网络正则化

---

Logistic regression regularization

先用简单的逻辑回归正则化作为例子，因为神经网络的参数W$W$是2维的。

1. 无正则
   

   J(w,b)=1m∑i=1mL(y^(i)−y(i))$$J(w,b)=\frac{1}{m}\sum _{i=1}^m\mathcal{L}({\hat{y}}^{(i)}-y^{(i)})$$

2. L2$L_2$ 正则
   

   J(w,b)=1m∑i=1mL(y^(i)−y(i))+λ2m||w||22$$J(w,b)=\frac{1}{m}\sum _{i=1}^m\mathcal{L}({\hat{y}}^{(i)}-y^{(i)})+\frac{\lambda }{2m}||w|{|}_2^2$$
   
   
   ||w||22=∑j=1nxw2j=wTw$$||w|{|}_2^2=\sum _{j=1}^{n_x}{w}_j^2=w^{T}w$$

3. L1$L_1$ 正则
   
   J(w,b)=1m∑i=1mL(y^(i)−y(i))+λm||w||1$$J(w,b)=\frac{1}{m}\sum _{i=1}^m\mathcal{L}({\hat{y}}^{(i)}-y^{(i)})+\frac{\lambda }{m}||w|{|}_1$$

||w||1=∑j=1nx|w|j$$||w|{|}_1=\sum _{j=1}^{n_x}|w{|}_j$$

Neural network regularization

1. Frobenius正则(类似L2$L_2$正则)
   
   J(w[1],b[1],⋯,w[l],b[l])=1m∑i=1mL(y^(i),y(i))+12m∑l=1L||w[l]||2F$$J(w^{[1]},b^{[1]},\cdots ,w^{[l]},b^{[l]})=\frac{1}{m}\sum _{i=1}^m\mathcal{L}({\hat{y}}^{(i)},y^{(i)})+\frac{1}{2m}\sum _{l=1}^L||w^{[l]}|{|}_F^2$$
   
   
   ||w[l]||2F=∑i=1n[l]∑j=1n[l−1](w[l]ij)2$$||w^{[l]}|{|}_F^2=\sum _{i=1}^{n^{[l]}}\sum _{j=1}^{n^{[l-1]}}(w_{ij}^{[l]}{)}^2$$

相较于无正则化的反向传播，正则化的反向传播在更新W$W$时，会对其进行权重衰减（weight decay），并下降。

dw[l]=(from backpropagation)+λmw[l]$$\mathrm{d}{w}^{[l]}=(frombackpropagation)+\frac{\lambda }{m}{w}^{[l]}$$

w[l]:=w[l]−αdw[l]=w[l]−αλmw[l]−α(from backpropagation)=(1−αλm)w[l]−α(from backpropagation)$$\begin{array}{rl}{w}^{[l]}:& =w^{[l]}-\alpha \mathrm{d}{w}^{[l]}\\ & =w^{[l]}-\alpha \frac{\lambda }{m}{w}^{[l]}-\alpha (frombackpropagation)\\ & =(1-\alpha \frac{\lambda }{m})w^{[l]}-\alpha (frombackpropagation)\end{array}$$