GBDT梯度提升分类树原理

本文主要介绍GBDT(Gradient Boosting Decision Tree)的实现原理。

算法

其中，$F_0$F0​表示决策回归树的初始值。

损失函数为：
$\psi(y,F(x)) = -ylog_e(p) - (1-y)log_e(1-p)$ψ(y,F(x))=−yloge​(p)−(1−y)loge​(1−p)，其中$p = \frac{1}{1 + exp(-F(x))}$p=1+exp(−F(x))1​

推导过程

1. 损失函数处理

首先进行化简：

$\psi(y,F(x)) = yln(1 + exp(-F(x))) - (1-y)ln(\frac{exp(-F(x))}{1 + exp(-F(x))}) \\ = yln(1 + exp(-F(x))) - (1 - y)(-F(x) - ln(1 + exp(-F(x)))) \\ = (1 -y)F(x) + yln(1 + exp(-F(x))) + (1 - y)ln(1 + exp(-F(x))) \\ = (1 - y)F(x) + ln(1 + exp(-F(x))) \\ = -yF(x) + F(x) + ln(1 + exp(-F(x))) \\ = -yF(x) + ln(exp(F(x))) + ln(1 + exp(-F(x))) \\ = -yF(x) + ln(exp(F(x))*(1 + exp(-F(x))) \\ = -yF(x) + ln(1 + exp(F(x))) \\ = -(yF(x) - ln(1 + exp(F(x))))$ψ(y,F(x))=yln(1+exp(−F(x)))−(1−y)ln(1+exp(−F(x))exp(−F(x))​)=yln(1+exp(−F(x)))−(1−y)(−F(x)−ln(1+exp(−F(x))))=(1−y)F(x)+yln(1+exp(−F(x)))+(1−y)ln(1+exp(−F(x)))=(1−y)F(x)+ln(1+exp(−F(x)))=−yF(x)+F(x)+ln(1+exp(−F(x)))=−yF(x)+ln(exp(F(x)))+ln(1+exp(−F(x)))=−yF(x)+ln(exp(F(x))∗(1+exp(−F(x)))=−yF(x)+ln(1+exp(F(x)))=−(yF(x)−ln(1+exp(F(x))))

接着进行求导：

$\psi'(y,F(x)) = -y + \sigma(F(x))$ψ′(y,F(x))=−y+σ(F(x))，其中$\sigma(F(x)) = \frac{1}{1 + exp(-F(x))}$σ(F(x))=1+exp(−F(x))1​
$\psi''(y,F(x)) = \sigma(F(x))(1 - \sigma(F(x)))$ψ′′(y,F(x))=σ(F(x))(1−σ(F(x)))

2. 决策回归树初始值计算

$F_0(x) = \rho$F0​(x)=ρ
$F_0(x) = {argmin}_{\rho}\sum\limits_{i = 1}^N\psi(y_i,\rho) \\ = argmin_{\rho}H(\rho) \\ = -\sum\limits_{i=1}^N(y_i\rho -log(1 + exp(\rho)))$F0​(x)=argminρ​i=1∑N​ψ(yi​,ρ)=argminρ​H(ρ)=−i=1∑N​(yi​ρ−log(1+exp(ρ)))

我们需要得到一个最小值作为决策回归树的初始值，以使信息熵最小。使用的方法仍然是求导，取导数为零时的值。这里$H(\rho)$H(ρ)即代表整体的损失函数。

$H'(\rho) = -\sum\limits_{i = 1}^N(y_i - \sigma(\rho)) \\ = -\sum\limits_{i = 1}^N(y_i - \frac{1}{1 + exp(-\rho)})$H′(ρ)=−i=1∑N​(yi​−σ(ρ))=−i=1∑N​(yi​−1+exp(−ρ)1​)

导数为零时，得到结果。

$0 = -\sum\limits_{i = 1}^N(y_i -\frac{1}{1 + exp(-\rho)})$0=−i=1∑N​(yi​−1+exp(−ρ)1​)

$\sum\limits_{i=1}^Ny_i = \sum\limits_{i=1}^N\frac{1}{1+exp(-\rho)}$i=1∑N​yi​=i=1∑N​1+exp(−ρ)1​

由于$exp(\rho)$exp(ρ)为常数，所以

$\sum\limits_{i=1}^Ny_i = \frac{N}{1+exp(-\rho)}$i=1∑N​yi​=1+exp(−ρ)N​

$1 + exp(-\rho) = \frac{\sum\limits_{i=1}^N1}{\sum\limits_{i=1}^Ny_i}$1+exp(−ρ)=i=1∑N​yi​i=1∑N​1​

$exp(-\rho) = \frac{\sum\limits_{i=1}^N(1 -y_i)}{\sum\limits_{i=1}^Ny_i}$exp(−ρ)=i=1∑N​yi​i=1∑N​(1−yi​)​

对左右两边分别进行对数运算

$-\rho = log\frac{\sum\limits_{i=1}^N(1 -y_i)}{\sum\limits_{i=1}^Ny_i}$−ρ=logi=1∑N​yi​i=1∑N​(1−yi​)​

最终得到

$\rho = log\frac{\sum\limits_{i=1}^Ny_i}{\sum\limits_{i=1}^N(1 -y_i)}$ρ=logi=1∑N​(1−yi​)i=1∑N​yi​​

3. 在第m轮的学习中，CART的第j个叶子节点的得分$\gamma_{mj}$γmj​

$L(\gamma_{mj},R_{mj}) = \sum_{x_i \in R_{mj}}\psi(y,F_{m-1}(x) + \gamma_{mj})$L(γmj​,Rmj​)=∑xi​∈Rmj​​ψ(y,Fm−1​(x)+γmj​)

根据泰勒展开公式（只取前三项，所以是约等于）：
$L(\gamma_{mj},R_{mj}) \approx \sum_{x_i \in R_{mj}}\{\psi(y,F_{m-1}(x)) + \psi'(y,F_{m-1}(x))\gamma_{mj} + \frac{1}{2}\psi''(y,F_{m-1}(x))\gamma_{mj}^2 \}$L(γmj​,Rmj​)≈∑xi​∈Rmj​​{ψ(y,Fm−1​(x))+ψ′(y,Fm−1​(x))γmj​+21​ψ′′(y,Fm−1​(x))γmj2​}

令$\sigma(F(x)) = \frac{1}{1 + exp(-F(x))}$σ(F(x))=1+exp(−F(x))1​，则

$\psi(y,F(x)) = -yF(x) + ln(1 + exp(F(x))) = -(yF(x) - ln(1 + exp(F(x))))$ψ(y,F(x))=−yF(x)+ln(1+exp(F(x)))=−(yF(x)−ln(1+exp(F(x))))

$\psi'(y,F(x)) = -y + \frac{1}{1 + exp(F(x))}*exp(F(x)) \\ = -y + \frac{1}{1 + exp(-F(x))} \\ = -y + \frac{1}{1 + exp(-F(x))} = -\widetilde{y} \\ = -y + \sigma(F(x))$ψ′(y,F(x))=−y+1+exp(F(x))1​∗exp(F(x))=−y+1+exp(−F(x))1​=−y+1+exp(−F(x))1​=−y​=−y+σ(F(x))

$\psi''(y,F(x)) = \frac{exp(-F(x))}{[1 + exp(-F(x))]^2} \\ = (y - \widetilde{y})(1 - y + \widetilde{y}) \\ = (y - ( y - \frac{1}{1+exp(-F(x))}))(1 - y + y - \frac{1}{1+exp(-F(x))}) \\ = \frac{1}{1+exp(-F(x))}(1 - \frac{1}{1+exp(-F(x))}) \\ = \sigma(F(x))(1 - \sigma(F(x)))$ψ′′(y,F(x))=[1+exp(−F(x))]2exp(−F(x))​=(y−y​)(1−y+y​)=(y−(y−1+exp(−F(x))1​))(1−y+y−1+exp(−F(x))1​)=1+exp(−F(x))1​(1−1+exp(−F(x))1​)=σ(F(x))(1−σ(F(x)))

最后求$\gamma_{mj}$γmj​的值

$\gamma_{mj} = argmin_{\gamma_{mj}}L(\gamma_{mj},R_{mj}) \\ = argmin_{\gamma_{mj}}\sum_{x_i \in R_{mj}}\{\psi(y_i,F_{m-1}(x_i)) + \psi'(y_i,F_{m-1}(x_i))\gamma_{mj} + \frac{1}{2}\psi''(y_i,F_{m-1}(x_i))\gamma_{mj}^2 \}$γmj​=argminγmj​​L(γmj​,Rmj​)=argminγmj​​∑xi​∈Rmj​​{ψ(yi​,Fm−1​(xi​))+ψ′(yi​,Fm−1​(xi​))γmj​+21​ψ′′(yi​,Fm−1​(xi​))γmj2​}

导数为零时得到结果

$0 = \sum_{x_i \in R_{mj}}\{\psi'(y_i,F_{m-1}(x_i)) + \psi''(y_i,F_{m-1}(x_i))*\gamma_{mj}\}$0=∑xi​∈Rmj​​{ψ′(yi​,Fm−1​(xi​))+ψ′′(yi​,Fm−1​(xi​))∗γmj​}
$0 = \sum_{x_i \in R_{mj}}\{-\widetilde{y_i} + (y_i - \widetilde{y_i})(1 - y_i + \widetilde{y_i}) *\gamma_{mj}\}$0=∑xi​∈Rmj​​{−yi​​+(yi​−yi​​)(1−yi​+yi​​)∗γmj​}
$\sum_{x_i \in R_{mj}}\widetilde{y_i} = \sum_{x_i \in R_{mj}}(y_i - \widetilde{y_i})(1 - y_i + \widetilde{y_i}) *\gamma_{mj}$∑xi​∈Rmj​​yi​​=∑xi​∈Rmj​​(yi​−yi​​)(1−yi​+yi​​)∗γmj​
$\sum_{x_i \in R_{mj}}\widetilde{y_i} = (\sum_{x_i \in R_{mj}}(y_i - \widetilde{y_i})(1 - y_i + \widetilde{y_i}) )*\gamma_{mj}$∑xi​∈Rmj​​yi​​=(∑xi​∈Rmj​​(yi​−yi​​)(1−yi​+yi​​))∗γmj​
$\gamma_{mj} = \frac{\sum_{x_i \in R_{mj}}\widetilde{y_i}}{\sum_{x_i \in R_{mj}}(y_i - \widetilde{y_i})(1 - y_i + \widetilde{y_i})}$γmj​=∑xi​∈Rmj​​(yi​−yi​​)(1−yi​+yi​​)∑xi​∈Rmj​​yi​​​

可以将$\gamma_{mj}$γmj​看作是梯度下降中的梯度，那么就有梯度更新规则
$F_m(x) = F_{m-1}(x) + \gamma_m*learning\_rate$Fm​(x)=Fm−1​(x)+γm​∗learning_rate