SVM（五）：支持向量回归

5 支持向量回归

5.1 问题定义

给定训练样本$D= \{(\boldsymbol x_1,y_1), (\boldsymbol x_2,y_2), ..., (\boldsymbol x_m,y_m)\},y_i \in \Bbb R$D={(x1​,y1​),(x2​,y2​),...,(xm​,ym​)},yi​∈R，希望学得形如 $f(\boldsymbol x) = \boldsymbol w^T \boldsymbol x + b$f(x)=wTx+b 的模型，使$f(\boldsymbol x)与y$f(x)与y尽可能接近，其中$\boldsymbol w$w和$\boldsymbol b$b是待确定参数。

支持向量回归（SVR，Support Vector Regression）假设能够容忍$f(\boldsymbol x)$f(x)与$y$y之间最多$\epsilon$ϵ的偏差，即仅当$f(\boldsymbol x)$f(x)与$y$y之间的差别绝对值大于$\epsilon$ϵ时才计算损失。这相当于以
$f(\boldsymbol x) = \boldsymbol w^T \boldsymbol x + b$f(x)=wTx+b为中心，构建了一个宽带$2\epsilon$2ϵ的间隔带，若训练样本落入此间隔带，则被认为预测正确。

SVR优化目标如下：
$\min\;\; \frac{1}{2}||\boldsymbol w||^2 +C\sum\limits_{i=1}^{m} {l}_{\epsilon} (y_i(\boldsymbol w^T \boldsymbol x_i + b)-1)$min21​∣∣w∣∣2+Ci=1∑m​lϵ​(yi​(wTxi​+b)−1)

其中$C>0$C>0是正则化常数，${l}_{\epsilon}$lϵ​是$\epsilon$ϵ-不敏感损失（$\epsilon$ϵ-insensitive loss）损失函数：
$l_{\epsilon}(z) = \begin{cases} 0 & {if } |z| \leq \epsilon \\ |z|-\epsilon & {otherwise } \end{cases}$lϵ​(z)={0∣z∣−ϵ​if∣z∣≤ϵotherwise​

5.2 对偶问题

引入松弛变量$\xi_i^{\lor}>0, \xi_i^{\land}>0$ξi∨​>0,ξi∧​>0（两边松弛变量可能不同），优化目标变为：
$\begin{aligned} \min & \; \frac{1}{2}||w||_2^2 + C\sum\limits_{i=1}^{m}(\xi_i^{\lor}+ \xi_i^{\land}) \\ s.t. \;\; &y_i - (\boldsymbol w^T \boldsymbol x_i + b) \leq \epsilon + \xi_i^{\land}, \\& \; \boldsymbol w^T \boldsymbol x_i + b - y_i \; \leq \epsilon + \xi_i^{\lor}, \\&\xi_i^{\lor} \geq 0, \;\; \xi_i^{\land} \geq 0 \;(i = 1,2,..., m) \end{aligned}$mins.t.​21​∣∣w∣∣22​+Ci=1∑m​(ξi∨​+ξi∧​)yi​−(wTxi​+b)≤ϵ+ξi∧​,wTxi​+b−yi​≤ϵ+ξi∨​,ξi∨​≥0,ξi∧​≥0(i=1,2,...,m)​

引入拉格朗日乘子$\mu_i^{\lor} \geq 0, \mu_i^{\land} \geq 0, \alpha_i^{\lor} \geq 0, \alpha_i^{\land} \geq 0$μi∨​≥0,μi∧​≥0,αi∨​≥0,αi∧​≥0：

$L(\boldsymbol w,b,\boldsymbol \alpha^{\lor}, \boldsymbol \alpha^{\land}, \boldsymbol \xi_i^{\lor}, \boldsymbol \xi_i^{\land}, \boldsymbol \mu^{\lor}, \boldsymbol \mu^{\land}) = \frac{1}{2}||w||_2^2 + C\sum\limits_{i=1}^{m}(\xi_i^{\lor}+ \xi_i^{\land}) - \sum\limits_{i=1}^{m}\mu_i^{\lor}\xi_i^{\lor} - \sum\limits_{i=1}^{m}\mu_i^{\land}\xi_i^{\land}+ \sum\limits_{i=1}^{m}\alpha_i^{\lor}(f(\boldsymbol x_i)-y_i -\epsilon - \xi_i^{\lor}) + \sum\limits_{i=1}^{m}\alpha_i^{\land}(y_i - f(\boldsymbol x_i) -\epsilon - \xi_i^{\land})$L(w,b,α∨,α∧,ξi∨​,ξi∧​,μ∨,μ∧)=21​∣∣w∣∣22​+Ci=1∑m​(ξi∨​+ξi∧​)−i=1∑m​μi∨​ξi∨​−i=1∑m​μi∧​ξi∧​+i=1∑m​αi∨​(f(xi​)−yi​−ϵ−ξi∨​)+i=1∑m​αi∧​(yi​−f(xi​)−ϵ−ξi∧​)其中，
$f(\boldsymbol x_i) = \boldsymbol w^T \boldsymbol x_i + b$f(xi​)=wTxi​+b

优化目标$\min_{\boldsymbol w,b,\xi_i^{\lor}, \xi_i^{\land}}\; \max_{\mu_i^{\lor}, \mu_i^{\land}, \alpha_i^{\lor}, \alpha_i^{\land}}\; L(\boldsymbol w,b,\boldsymbol \alpha^{\lor}, \boldsymbol \alpha^{\land}, \boldsymbol \xi^{\lor}, \boldsymbol \xi^{\land}, \boldsymbol \mu^{\lor}, \boldsymbol \mu^{\land})$w,b,ξi∨​,ξi∧​min​μi∨​,μi∧​,αi∨​,αi∧​max​L(w,b,α∨,α∧,ξ∨,ξ∧,μ∨,μ∧)

满足KTT条件，对偶问题为：$\max_{\mu_i^{\lor}, \mu_i^{\land}, \alpha_i^{\lor}, \alpha_i^{\land}}\; \min_{\boldsymbol w,b,\xi_i^{\lor}, \xi_i^{\land}}\; L(\boldsymbol w,b,\boldsymbol \alpha^{\lor}, \boldsymbol \alpha^{\land}, \boldsymbol \xi^{\lor}, \boldsymbol \xi^{\land}, \boldsymbol \mu^{\lor}, \boldsymbol \mu^{\land})$μi∨​,μi∧​,αi∨​,αi∧​max​w,b,ξi∨​,ξi∧​min​L(w,b,α∨,α∧,ξ∨,ξ∧,μ∨,μ∧)
首先通过对$\boldsymbol w,b,\xi_i^{\lor}, \xi_i^{\land}$w,b,ξi∨​,ξi∧​求偏导，计算极小值：
$\frac{\partial L}{\partial \boldsymbol w} = 0 \;\Rightarrow \boldsymbol w = \sum\limits_{i=1}^{m}(\alpha_i^{\land} - \alpha_i^{\lor})\boldsymbol x_i$∂w∂L​=0⇒w=i=1∑m​(αi∧​−αi∨​)xi​$\frac{\partial L}{\partial b} = 0 \;\Rightarrow \sum\limits_{i=1}^{m}(\alpha_i^{\land} - \alpha_i^{\lor}) = 0$∂b∂L​=0⇒i=1∑m​(αi∧​−αi∨​)=0$\frac{\partial L}{\partial \xi_i^{\lor}} = 0 \;\Rightarrow C = \alpha^{\lor} + \mu^{\lor}$∂ξi∨​∂L​=0⇒C=α∨+μ∨$\frac{\partial L}{\partial \xi_i^{\land}} = 0 \;\Rightarrow C= \alpha^{\land}+ \mu^{\land}$∂ξi∧​∂L​=0⇒C=α∧+μ∧

代回至
$L(\boldsymbol w,b,\boldsymbol \alpha^{\lor}, \boldsymbol \alpha^{\land}, \boldsymbol \xi^{\lor}, \boldsymbol \xi^{\land}, \boldsymbol \mu^{\lor}, \boldsymbol \mu^{\land})$L(w,b,α∨,α∧,ξ∨,ξ∧,μ∨,μ∧)

$\begin{aligned} \min_{\boldsymbol w,b,\xi_i^{\lor}, \xi_i^{\land}}\; L(\boldsymbol w,b,\boldsymbol \alpha^{\lor}, \boldsymbol \alpha^{\land}, \boldsymbol \xi^{\lor}, \boldsymbol \xi^{\land}, \boldsymbol \mu^{\lor}, \boldsymbol \mu^{\land}) & = \sum\limits_{i=1}^{m}y_i(\alpha_i^{\land}- \alpha_i^{\lor}) - \epsilon(\alpha_i^{\land} + \alpha_i^{\lor})- \frac{1}{2}\sum_{i=1}^{m}\sum_{j=1}^{m}(\alpha_i^{\land} - \alpha_i^{\lor})(\alpha_j^{\land} - \alpha_j^{\lor}) \boldsymbol x_i^T \boldsymbol x_j \\ s.t. \; &\sum\limits_{i=1}^{m}(\alpha_i^{\land} - \alpha_i^{\lor}) = 0 \\ & 0 \leq \alpha_i^{\lor},\alpha_i^{\land} \leq C \; (i =1,2,...m) \end {aligned}$w,b,ξi∨​,ξi∧​min​L(w,b,α∨,α∧,ξ∨,ξ∧,μ∨,μ∧)s.t.​=i=1∑m​yi​(αi∧​−αi∨​)−ϵ(αi∧​+αi∨​)−21​i=1∑m​j=1∑m​(αi∧​−αi∨​)(αj∧​−αj∨​)xiT​xj​i=1∑m​(αi∧​−αi∨​)=00≤αi∨​,αi∧​≤C(i=1,2,...m)​

原问题最终转换为如下形式的对偶问题：
$\begin{aligned} \max_{\boldsymbol \alpha^{\land},\boldsymbol \alpha^{\lor}}\;\; & \sum\limits_{i=1}^{m}y_i(\alpha_i^{\land}- \alpha_i^{\lor}) - \epsilon(\alpha_i^{\land} + \alpha_i^{\lor})- \frac{1}{2}\sum_{i=1}^{m}\sum_{j=1}^{m}(\alpha_i^{\land} - \alpha_i^{\lor})(\alpha_j^{\land} - \alpha_j^{\lor}) \boldsymbol x_i^T \boldsymbol x_j \\ s.t. \; &\sum\limits_{i=1}^{m}(\alpha_i^{\land} - \alpha_i^{\lor}) = 0 \\ & 0 \leq \alpha_i^{\lor},\alpha_i^{\land} \leq C \; (i =1,2,...m) \end {aligned}$α∧,α∨max​s.t.​i=1∑m​yi​(αi∧​−αi∨​)−ϵ(αi∧​+αi∨​)−21​i=1∑m​j=1∑m​(αi∧​−αi∨​)(αj∧​−αj∨​)xiT​xj​i=1∑m​(αi∧​−αi∨​)=00≤αi∨​,αi∧​≤C(i=1,2,...m)​

此时，优化函数仅有$\boldsymbol \alpha^{\land},\boldsymbol \alpha^{\lor}$α∧,α∨做为参数，可采用SMO（Sequential Minimal Optimization）求解，进而得出$\boldsymbol w,b$w,b。