[Kaiming]Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

文章目录

• 概
• 主要内容
• PReLU
• Kaiming 初始化
• Forward case
• Backward case

He K, Zhang X, Ren S, et al. Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification[C]. international conference on computer vision, 2015: 1026-1034.

@article{he2015delving,
title={Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification},
author={He, Kaiming and Zhang, Xiangyu and Ren, Shaoqing and Sun, Jian},
pages={1026–1034},
year={2015}}

概

本文介绍了一种PReLU的**函数和Kaiming的参数初始化方法.

主要内容

PReLU

$f(y_i) = \left \{ \begin{array}{ll} y_i, & y_i >0, \\ a_i y_i, & y_i \le 0. \end{array} \right.$f(yi​)={yi​,ai​yi​,​yi​>0,yi​≤0.​
其中$a_i$ai​是作为网络的参数进行训练的.
等价于
$f(y_i)=\max(0, y_i) + a_i \min (0,y_i).$f(yi​)=max(0,yi​)+ai​min(0,yi​).
特别的, 可以一层的节点都用同一个$a$a.

Kaiming 初始化

Forward case

$\mathbf{y}_l=W_l\mathbf{x}_l+\mathbf{b}_l,$yl​=Wl​xl​+bl​,
在卷积层中时, $\mathbf{x}_l$xl​是$k\times k \times c$k×k×c的展开, 故$\mathrm{x}_l\in \mathbb{R}^{k^2c}$xl​∈Rk2c, 而$\mathbf{y}_l \in \mathbb{R}^{d}$yl​∈Rd, $W_l \in \mathbb{R^{d \times k^2c}}$Wl​∈Rd×k2c(每一行都可以视作一个kernel), 并记$n=k^2c$n=k2c.

$\mathbf{x}_l=f(\mathbf{y}_{l-1}),$xl​=f(yl−1​),
则
$c_l = d_{l-1}.$cl​=dl−1​.

假设$w_l$wl​与$x_l$xl​(注意没粗体, 表示$\mathbf{w}_l, \mathbf{x}_l$wl​,xl​中的某个元素)相互独立, 且$w_l$wl​采样自一个均值为0的对称分布之中.

则
$Var[y_l] = n_l Var [w_lx_l] = n_lVar[w_l]E[x_l^2],$Var[yl​]=nl​Var[wl​xl​]=nl​Var[wl​]E[xl2​],
除非$E[x_l]=0$E[xl​]=0, $Var[y_l] = n_lVar[w_l]Var[x_l]$Var[yl​]=nl​Var[wl​]Var[xl​], 但对于ReLu, 或者 PReLU来说这个性质是不成立的.

如果我们令$b_{l-1}=0$bl−1​=0, 易证
$E[x_l^2] = \frac{1}{2} Var[y_{l-1}],$E[xl2​]=21​Var[yl−1​],
其中$f$f是ReLU, 若$f$f是PReLU,
$E[x_l^2] = \frac{1+a^2}{2} Var[y_{l-1}].$E[xl2​]=21+a2​Var[yl−1​].
下面用ReLU分析, PReLU是类似的.

故
$Var[y_l] = \frac{1}{2} n_l ar[w_l]Var[y_{l-1}],$Var[yl​]=21​nl​ar[wl​]Var[yl−1​],
自然我们希望
$Var[y_i]=Var[y_j] \Rightarrow \frac{1}{2}n_l Var[w_l]=1, \forall l.$Var[yi​]=Var[yj​]⇒21​nl​Var[wl​]=1,∀l.

Backward case

$\tag{13} \Delta \mathbf{x}_l = \hat{W}_l \Delta \mathbf{y}_l,$Δxl​=W^l​Δyl​,(13)
$\Delta \mathbf{x}_l$Δxl​表示损失函数观念与$\mathbf{x}_l$xl​的导数, 这里的$\mathbf{y}_l$yl​与之前提到的$\mathbf{y}_l$yl​有出入, 这里需要用到卷积的梯度回传, 三言两语讲不清, $\hat{W}_l$W^l​是$W_l$Wl​的一个重排.

因为$\mathbf{x}_l=f(\mathbf{y}_{l-1})$xl​=f(yl−1​), 所以
$\Delta y_l = f'(y_l) \Delta x_{l+1}.$Δyl​=f′(yl​)Δxl+1​.

假设$f'(y_l)$f′(yl​)与$\Delta x_{l+1}$Δxl+1​相互独立, 所以
$E[\Delta y_l]=E[f'(y_l)] E[\Delta x_{l+1}] = 0,$E[Δyl​]=E[f′(yl​)]E[Δxl+1​]=0,
若$f$f为ReLU:
$E[(\Delta y_l)^2] = Var[\Delta y_l] = \frac{1}{2}Var[\Delta x_{l+1}].$E[(Δyl​)2]=Var[Δyl​]=21​Var[Δxl+1​].
若$f$f为PReLU:
$E[(\Delta y_l)^2] = Var[\Delta y_l] = \frac{1+a^2}{2}Var[\Delta x_{l+1}].$E[(Δyl​)2]=Var[Δyl​]=21+a2​Var[Δxl+1​].

下面以$f$f为ReLU为例, PReLU类似

$Var[\Delta x_l] = \hat{n}_l Var[w_l] Var[\Delta y_l] = \frac{1}{2} \hat{n}_l Var[w_l] Var[\Delta x_{l+1}],$Var[Δxl​]=n^l​Var[wl​]Var[Δyl​]=21​n^l​Var[wl​]Var[Δxl+1​],
这里$\hat{n}_l=k^2d$n^l​=k2d为$\mathbf{y}_l$yl​的长度.

和前向的一样, 我们希望$Var[\Delta x_l]$Var[Δxl​]一样, 需要
$\frac{1}{2}\hat{n}_l Var[w_l]=1, \forall l.$21​n^l​Var[wl​]=1,∀l.

是实际中，我们前向后向可以任选一个(因为误差不会累积).