在输入维度为2的二分类问题中，假设训练样本完全线性可分，即所有训练样本点可以通过一条直线完全分开。
我们将离直线最近的点定义为"危险点"，意味着我们对这个点的分类已经很危险了，一不小心就会分错。这是比较直观的感觉，数学的形式化描述详见Vapnik的著作以及相关概念这里不展开。max w , b 间距 ( w , b ) s. t. y n w T x n > 0 //表示分类正确 间距 ( w , b ) = min n = 1 , … , N 距离 ( x n , w , b ) \begin{array}{cl}{\max \limits_{\mathbf{w,b}}} & {\operatorname{间距}(\mathbf{w,b})} \\ {\text { s. t. }} & { y_{n} \mathbf{w}^{T} \mathbf{x}_{n}>0\text{//表示分类正确}} \\ {} & {\operatorname{间距}(\mathbf{w,b})=\min _{n=1, \ldots, N} \operatorname{距离}\left(\mathbf{x}_{n}, \mathbf{w,b}\right)}\end{array} w , b max s. t. 间 距 ( w , b ) y n w T x n > 0 // 表示分类正确 间 距 ( w , b ) = min n = 1 , … , N 距 离 ( x n , w , b )
二维的情况下计算点到直线的距离较为容易，现在我们将其扩展到多维空间中。则分类超平面可表示为w T x + b = 0 w^Tx+b=0 w T x + b = 0 w T x ′ + b = 0 , w T x ′ ′ + b = 0 。 w^Tx'+b=0,w^Tx''+b=0。 w T x ′ + b = 0 , w T x ′ ′ + b = 0 。 w T ( x ′ ′ − x ′ ) = 0 w^T(x''-x')=0 w T ( x ′ ′ − x ′ ) = 0 ( x ′ ′ − x ′ ) (x''-x') ( x ′ ′ − x ′ ) 法向量 如图θ \theta θ distance ( x , b , w ) = ∣ ∣ ∣ x − x ′ ∣ cos ( θ ) ∣ = ∣ ∥ x − x ′ ∥ ⋅ ( x − x ′ ) w ∥ x − x ′ ∥ ⋅ ∥ w ∥ ∣ = 1 ∥ w ∥ ∣ w T x − w T x ′ ∣ \begin{aligned} \operatorname{distance}(x, b, w) &=\left|\left||x-x^{\prime}\right| \cos (\theta)\right|=\left|\left\|x-x^{\prime}\right\| \cdot \frac{\left(x-x^{\prime}\right) w}{\left\|x-x^{\prime}\right\| \cdot\|w\|}\right|= \frac{1}{\|w\|}\left|w^{T} x-w^{T} x^{\prime}\right| \end{aligned} d i s t a n c e ( x , b , w ) = ∣ ∣ ∣ x − x ′ ∣ cos ( θ ) ∣ = ∣ ∣ ∣ ∣ ∥ x − x ′ ∥ ⋅ ∥ x − x ′ ∥ ⋅ ∥ w ∥ ( x − x ′ ) w ∣ ∣ ∣ ∣ = ∥ w ∥ 1 ∣ ∣ w T x − w T x ′ ∣ ∣ w T x ′ + b = 0 w^Tx'+b=0 w T x ′ + b = 0 distance ( x , b , w ) = 1 ∥ w ∥ ∣ w T x + b ∣ \begin{aligned} \operatorname{distance}(x, b, w) &= \frac{1}{\|w\|}\left|w^{T} x+b\right| \end{aligned} d i s t a n c e ( x , b , w ) = ∥ w ∥ 1 ∣ ∣ w T x + b ∣ ∣ y n ( w T x + b ) > 0 y_n(w^Tx+b)>0 y n ( w T x + b ) > 0 distance ( x , b , w ) = 1 ∥ w ∥ y n ( w T x n + b ) \begin{aligned} \operatorname{distance}(x, b, w) &= \frac{1}{\|w\|}y_{n}\left(w^{T} x_{n}+b\right) \end{aligned} d i s t a n c e ( x , b , w ) = ∥ w ∥ 1 y n ( w T x n + b ) y n ∈ { − 1 , 1 } y_{n}\in\{-1,1\} y n ∈ { − 1 , 1 } max w , b 间距 ( w , b ) s. t. y n w T x n > 0 间距 ( w , b ) = min n = 1 , … , N 1 ∥ w ∥ y n ( w T x n + b ) \begin{array}{cl}{\max \limits_{\mathbf{w,b}}} & {\operatorname{间距}(\mathbf{w,b})} \\ {\text { s. t. }} & { y_{n} \mathbf{w}^{T} \mathbf{x}_{n}>0} \\ {} & {\operatorname{间距}(\mathbf{w,b})=\min _{n=1, \ldots, N} \frac{1}{\|w\|}y_{n}\left(w^{T} x_{n}+b\right) }\end{array} w , b max s. t. 间 距 ( w , b ) y n w T x n > 0 间 距 ( w , b ) = min n = 1 , … , N ∥ w ∥ 1 y n ( w T x n + b )
当我们对w w w b b b y n ( w T x n + b ) = 1 \begin{aligned} y_{n}\left(w^{T} x_{n}+b\right)=1 \end{aligned} y n ( w T x n + b ) = 1 间 距 ( w , b ) = 1 ∥ w ∥ 间距(\mathbf{w,b})=\frac{1}{\|w\|} 间 距 ( w , b ) = ∥ w ∥ 1 max w , b 1 ∥ w ∥ s . t . y n w T x n > 0 min n = 1 , … , N y n ( w T x n + b ) = 1 \begin{array}{cl}{\max \limits_{\mathbf{w,b}}} & {\frac{1}{\|\mathbf{w}\|}} \\ s.t. & { y_{n} \mathbf{w}^{T} \mathbf{x}_{n}>0} \\ {} & \min _{n=1, \ldots, N} y_{n}\left(w^{T} x_{n}+b\right)=1 \end{array} w , b max s . t . ∥ w ∥ 1 y n w T x n > 0 min n = 1 , … , N y n ( w T x n + b ) = 1 y n w T x n ≥ 1 > 0 y_{n} \mathbf{w}^{T} \mathbf{x}_{n}\ge1>0 y n w T x n ≥ 1 > 0 \sqrt{} 1 2 \frac{1}{2} 2 1 min w , b 1 2 w T w s . t . y n ( w T x n + b ) ≥ 1 f o r a l l n \begin{array}{cl}{\min \limits_{\mathbf{w,b}}} & {\frac{1}{2}{\mathbf{w}^{T}\mathbf{w}}} \\ s.t .& y_{n} (\mathbf{w}^{T} x_{n}+b)\ge1 \quad for\quad all \quad n\end{array} w , b min s . t . 2 1 w T w y n ( w T x n + b ) ≥ 1 f o r a l l n
因此最后得到了一个带不等式约束的凸二次规划问题。此时已经可以使用QP标准包进行求解，只需进行参数代换即可，
min w , b 1 2 w T w s . t . y n ( w T x n + b ) ≥ 1 f o r a l l n \begin{array}{cl}{\min \limits_{\mathbf{w,b}}} & {\frac{1}{2}{\mathbf{w}^{T}\mathbf{w}}} \\ s.t .& y_{n} (\mathbf{w}^{T} x_{n}+b)\ge1 \quad for\quad all \quad n\end{array} w , b min s . t . 2 1 w T w y n ( w T x n + b ) ≥ 1 f o r a l l n
首先我们要讨论的是无约束最优化问题的一阶必要条件，考虑如下问题m i n f ( x ) s . t . x ∈ R n \begin{array}{cl}min&f(x)\\s.t.& x\in R^n\end{array} m i n s . t . f ( x ) x ∈ R n f f f x ∗ x^* x ∗ ▽ f ( x ∗ ) = 0 \triangledown f(x^*)=\mathbf{0} ▽ f ( x ∗ ) = 0 f f f x ∗ x^* x ∗ R n R^n R n
▽ f ( x ∗ ) = 0 \triangledown f(x^*)=0 ▽ f ( x ∗ ) = 0 F ( x ∗ ) > 0 F(x^*)>0 F ( x ∗ ) > 0
则x ∗ x^* x ∗ 严格凸函数 的概念即Hessian矩阵F ( x ) F(x) F ( x )
我们现在想将带约束的最优化问题转换为我们已经掌握的无约束最优化问题。考虑求解最小化问题，不妨假设一个新的函数l l l f f f + ∞ +\infin + ∞ L ( x ) = { f ( x ) x ∈ 约 束 集 合 + ∞ x ∉ 约 束 集 合 L(x)=\left\{
\begin{aligned}
&f(x) &x \in 约束集合\\
&+\infin & x \notin 约束集合
\end{aligned}
\right.
L ( x ) = { f ( x ) + ∞ x ∈ 约 束 集 合 x ∈ / 约 束 集 合
首先写出SVM的原始形式min w , b 1 2 w T w s . t . y n ( w T x n + b ) ≥ 1 f o r a l l n \begin{array}{cl}{\min\limits_{\mathbf{w,b}}} & {\frac{1}{2}{\mathbf{w}^{T}\mathbf{w}}} \\ s.t .& y_{n} (\mathbf{w}^{T} x_{n}+b)\ge1 \quad for\quad all \quad n\end{array} w , b min s . t . 2 1 w T w y n ( w T x n + b ) ≥ 1 f o r a l l n L ( x , λ , μ ) = f ( x ) + λ T h ( x ) + μ T g ( x ) L ( w , b , μ ) = 1 2 w T w + ∑ i = 1 n μ T ( 1 − y n ( w T x n + b ) ) = 1 2 w T w + ∑ i = 1 n μ T − ∑ i = 1 n μ T y n ( w T x n + b ) \begin{array} {cl}L(x,\lambda,\mu) &= f(x)+\lambda^{T}h(x)+\mu^{T}g(x) \\ L(\mathbf{w},b,\mu)
&= \frac{1}{2}\mathbf{w}^T\mathbf{w}+\sum\limits_{i=1}^n \mu^T(1-y_n(\mathbf{w}^Tx_n+b)) \\ &=\frac{1}{2}\mathbf{w}^T\mathbf{w}+\sum\limits_{i=1}^n \mu^T-\sum\limits_{i=1}^n\mu^T y_n(\mathbf{w}^Tx_n+b) \end{array} L ( x , λ , μ ) L ( w , b , μ ) = f ( x ) + λ T h ( x ) + μ T g ( x ) = 2 1 w T w + i = 1 ∑ n μ T ( 1 − y n ( w T x n + b ) ) = 2 1 w T w + i = 1 ∑ n μ T − i = 1 ∑ n μ T y n ( w T x n + b ) f ( w , b ) = 1 2 w T w f(w,b)=\frac{1}{2}w^Tw f ( w , b ) = 2 1 w T w g ( w , b ) = 1 − y n ( w T x n + b ) ≤ 0 g(w,b)=1-y_n(\mathbf{w}^Tx_n+b)\le 0 g ( w , b ) = 1 − y n ( w T x n + b ) ≤ 0 min w , b max μ f ( w , b ) + μ T g ( w , b ) s . t . μ ≥ 0 \begin{array}{cl}{\min\limits_{\mathbf{w,b}}}\max\limits_{\mu} & {f(\mathbf w,b)+\mu^Tg(\mathbf w,b)} \\ s.t .& \mu\ge0\\ \end{array} w , b min μ max s . t . f ( w , b ) + μ T g ( w , b ) μ ≥ 0 μ ≥ 0 \mu\ge0 μ ≥ 0 g ( w , b ) = 1 − y n ( w T x n + b ) ≤ 0 g(w,b)=1-y_n(\mathbf{w}^Tx_n+b)\le 0 g ( w , b ) = 1 − y n ( w T x n + b ) ≤ 0 μ T g ( w , b ) ≤ 0 \mu^Tg(w,b)\le0 μ T g ( w , b ) ≤ 0 μ T g ( w , b ) = 0 \mu^Tg(w,b)=0 μ T g ( w , b ) = 0 min w , b f ( w , b ) \min\limits_{w,b}f(w,b) w , b min f ( w , b ) g ( w , b ) = 1 − y n ( w T x n + b ) > 0 g(w,b)=1-y_n(\mathbf{w}^Tx_n+b)> 0 g ( w , b ) = 1 − y n ( w T x n + b ) > 0 μ ≥ 0 \mu\ge0 μ ≥ 0 + ∞ +\infin + ∞ min w , b max μ f ( w , b ) + μ T g ( w , b ) s . t . μ ≥ 0 \begin{array}{cl}{\min\limits_{\mathbf{w,b}}}\max\limits_{\mu} & {f(\mathbf w,b)+\mu^Tg(\mathbf w,b)} \\ s.t .& \mu\ge0\\ \end{array} w , b min μ max s . t . f ( w , b ) + μ T g ( w , b ) μ ≥ 0
转换成拉格朗日表述后，问题依然难以求解，此时引入对偶函数d u a l = max μ min w , b L ( w , b , μ ) dual=\max\limits_{\mu}\min\limits_{w,b}L(w,b,\mu) d u a l = μ max w , b min L ( w , b , μ ) μ ≥ 0 , g ( w , b ) ≤ 0 \mu\ge0,g(w,b)\le0 μ ≥ 0 , g ( w , b ) ≤ 0 min w , b max μ L ( w , b , μ ) ≥ max μ min w , b L ( w , b , μ ) \min\limits_{w,b}\max\limits_{\mu}L(w,b,\mu)\ge\max\limits_{\mu}\min\limits_{w,b}L(w,b,\mu) w , b min μ max L ( w , b , μ ) ≥ μ max w , b min L ( w , b , μ ) p ∗ p^* p ∗ w = w ∗ , b = b ∗ w=w^*,b=b^* w = w ∗ , b = b ∗ d △ d^{\triangle} d △ μ = μ △ \mu=\mu^{\triangle} μ = μ △ d △ = min w , b L ( w , b , μ △ ) ≤ L ( w , b , μ △ ) ∀ w , b ∴ d △ ≤ L ( w ∗ , b ∗ , μ △ ) = f ( w ∗ , b ∗ ) + μ △ g ( w ∗ , b ∗ ) ∵ μ △ g ( w ∗ , b ∗ ) ≤ 0 ∴ d △ ≤ f ( w ∗ , b ∗ ) = p ∗ \begin{array}{cl}d^\triangle &= \min\limits_{w,b}L(w,b,\mu^\triangle) \\
&\le L(w,b,\mu^\triangle) {\forall w,b}\\ \therefore d^\triangle &\le L(w^*,b^*,\mu^\triangle)\\&=f(w^*,b^*)+\mu^\triangle g(w^*,b^*)\\\because\mu^\triangle g(w^*,b^*)&\le0\\ \therefore d^\triangle &\le f(w^*,b^*)=p^*\end{array} d △ ∴ d △ ∵ μ △ g ( w ∗ , b ∗ ) ∴ d △ = w , b min L ( w , b , μ △ ) ≤ L ( w , b , μ △ ) ∀ w , b ≤ L ( w ∗ , b ∗ , μ △ ) = f ( w ∗ , b ∗ ) + μ △ g ( w ∗ , b ∗ ) ≤ 0 ≤ f ( w ∗ , b ∗ ) = p ∗ { μ △ g ( w ∗ , b ∗ ) = 0 min w , b L ( w , b , μ △ ) = L ( w ∗ , b ∗ , μ △ ) \left\{
\begin{aligned}
\mu^\triangle g(w^*,b^*) &=0 \\
\min\limits_{w,b}L(w,b,\mu^\triangle)&=L(w^*,b^*,\mu^\triangle)
\end{aligned}
\right.
⎩ ⎨ ⎧ μ △ g ( w ∗ , b ∗ ) w , b min L ( w , b , μ △ ) = 0 = L ( w ∗ , b ∗ , μ △ ) ∂ L ∂ w ∣ w = w ∗ = 0 \frac{\partial L}{\partial w}|_{w=w^*}=0 ∂ w ∂ L ∣ w = w ∗ = 0 ∂ L ∂ b ∣ b = b ∗ = 0 \frac{\partial L}{\partial b}|_{b=b^*}=0 ∂ b ∂ L ∣ b = b ∗ = 0 w ∗ = ∑ i = 1 n μ i △ y i x i , ∑ i = 1 n μ i y i = 0 w^*=\sum\limits_{i=1}^n\mu_i^\triangle y_ix_i,\sum\limits_{i=1}^n\mu_iy_i=0 w ∗ = i = 1 ∑ n μ i △ y i x i , i = 1 ∑ n μ i y i = 0
满足{ μ ≥ 0 g ( w , b ) = 1 − y n ( w T x n + b ) ≤ 0 μ △ g ( w ∗ , b ∗ ) = 0 w ∗ − ∑ i = 1 n μ i △ y i x i = 0 ∑ i = 1 n μ i △ y i = 0 \left\{\begin{aligned}\mu &\ge0\\g(w,b)=1-y_n(\mathbf{w}^Tx_n+b)&\le 0\\\mu^\triangle g(w^*,b^*) &=0\\w^*-\sum\limits_{i=1}^n\mu_i^\triangle y_ix_i&=0\\\sum\limits_{i=1}^n\mu_i^\triangle y_i&=0\end{aligned}\right. ⎩ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎧ μ g ( w , b ) = 1 − y n ( w T x n + b ) μ △ g ( w ∗ , b ∗ ) w ∗ − i = 1 ∑ n μ i △ y i x i i = 1 ∑ n μ i △ y i ≥ 0 ≤ 0 = 0 = 0 = 0 max μ min w , b 1 2 w T w + ∑ i = 1 n μ T − ∑ i = 1 n μ T y n ( w T x n + b ) → max μ − 1 2 ∑ i = 1 n ∑ j = 1 n μ i μ j y i y j x i T x j + ∑ i = 1 n μ i → min μ 1 2 ∑ i = 1 n ∑ j = 1 n μ i μ j y i y j x i T x j − ∑ i = 1 n μ i \begin{array}{cl}&\max\limits_{\mu}\min\limits_{w,b}\frac{1}{2}\mathbf{w}^T\mathbf{w}+\sum\limits_{i=1}^n \mu^T-\sum\limits_{i=1}^n\mu^T y_n(\mathbf{w}^Tx_n+b)\\&\to\max\limits_{\mu}-\frac{1}{2}\sum\limits_{i=1}^n\sum\limits_{j=1}^n\mu_i\mu_jy_iy_jx_i^Tx_j+\sum\limits_{i=1}^n\mu_i\\&\to \min\limits_\mu \frac{1}{2}\sum\limits_{i=1}^n\sum\limits_{j=1}^n\mu_i\mu_jy_iy_jx_i^Tx_j-\sum\limits_{i=1}^n\mu_i\end{array} μ max w , b min 2 1 w T w + i = 1 ∑ n μ T − i = 1 ∑ n μ T y n ( w T x n + b ) → μ max − 2 1 i = 1 ∑ n j = 1 ∑ n μ i μ j y i y j x i T x j + i = 1 ∑ n μ i → μ min 2 1 i = 1 ∑ n j = 1 ∑ n μ i μ j y i y j x i T x j − i = 1 ∑ n μ i min μ 1 2 ∑ i = 1 n ∑ j = 1 n μ i μ j y i y j x i T x j − ∑ i = 1 n μ i s . t . ∑ i = 1 n μ i y i = 0 μ i ≥ 0 , i = 1 , 2... , n \begin{array}{cl}\min\limits_\mu &\frac{1}{2}\sum\limits_{i=1}^n\sum\limits_{j=1}^n\mu_i\mu_jy_iy_jx_i^Tx_j-\sum\limits_{i=1}^n\mu_i\\{s.t.}&\sum\limits_{i=1}^n\mu_i y_i=0\\&\mu_i \ge0 ,i=1,2...,n\end{array}
μ min s . t . 2 1 i = 1 ∑ n j = 1 ∑ n μ i μ j y i y j x i T x j − i = 1 ∑ n μ i i = 1 ∑ n μ i y i = 0 μ i ≥ 0 , i = 1 , 2 . . . , n μ \mu μ { w ∗ = ∑ i = 1 n μ i △ y i x i μ i △ ( y i ( w ∗ T x i + b ∗ ) − 1 ) = 0 y i ( w ∗ T x i + b ∗ ) − 1 ≥ 0 μ i △ ≥ 0 \left\{\begin{aligned}w^*=\sum\limits_{i=1}^n\mu_i^\triangle y_ix_i \\ \mu_i^\triangle (y_i(\mathbf{w}^{*T}x_i+b^*)-1)=0 \\y_i(\mathbf{w}^{*T}x_i+b^*)-1\ge0\\\mu_i^\triangle\ge0\end{aligned}\right. ⎩ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎧ w ∗ = i = 1 ∑ n μ i △ y i x i μ i △ ( y i ( w ∗ T x i + b ∗ ) − 1 ) = 0 y i ( w ∗ T x i + b ∗ ) − 1 ≥ 0 μ i △ ≥ 0 ∀ μ i △ = 0 \forall\mu_i^\triangle=0 ∀ μ i △ = 0 w ∗ = 0 w^*=0 w ∗ = 0 max 1 ∣ ∣ w ∣ ∣ \max\frac{1}{||w||} max ∣ ∣ w ∣ ∣ 1 ∃ μ j > 0 \exists \mu_j>0 ∃ μ j > 0 y j ( w ∗ T x j + b ∗ ) = 1 y_j(\mathbf{w}^{*T}x_j+b^*)=1 y j ( w ∗ T x j + b ∗ ) = 1 y j 2 = 1 , y_j^{2}=1, y j 2 = 1 , w ∗ = ∑ i = 1 n μ i △ y i x i w^*=\sum\limits_{i=1}^n \mu_i^\triangle y_ix_i w ∗ = i = 1 ∑ n μ i △ y i x i b ∗ = y j − ∑ i = 1 n μ △ y i ( x i ⋅ x j ) b^{*}=y_j-\sum\limits_{i=1}^n\mu^\triangle y_i(x_i·x_j) b ∗ = y j − i = 1 ∑ n μ △ y i ( x i ⋅ x j )
根据上面的推论，我们可以发现w,b都由μ i > 0 \mu_i >0 μ i > 0 w ∗ = ∑ S V μ n △ y n x n b ∗ = y n − w T x n w i t h S V \begin{aligned}w^*&=\sum\limits_{SV}\mu^\triangle_ny_nx_n \\b^*&=y_n-w^Tx_n \quad with \quad SV\end{aligned} w ∗ b ∗ = S V ∑ μ n △ y n x n = y n − w T x n w i t h S V
我们再次使用QP标准库来解决对偶后的问题min μ 1 2 ∑ i = 1 n ∑ j = 1 n μ i μ j y i y j x i T x j − ∑ i = 1 n μ i s . t . ∑ i = 1 n μ i y i = 0 μ i ≥ 0 , i = 1 , 2... , n \begin{array}{cl}\min\limits_\mu &\frac{1}{2}\sum\limits_{i=1}^n\sum\limits_{j=1}^n\mu_i\mu_jy_iy_jx_i^Tx_j-\sum\limits_{i=1}^n\mu_i\\{s.t.}&\sum\limits_{i=1}^n\mu_i y_i=0\\&\mu_i \ge0 ,i=1,2...,n\end{array}
μ min s . t . 2 1 i = 1 ∑ n j = 1 ∑ n μ i μ j y i y j x i T x j − i = 1 ∑ n μ i i = 1 ∑ n μ i y i = 0 μ i ≥ 0 , i = 1 , 2 . . . , n q n , m = y n y m x n T x m q_{n,m}=y_ny_mx_n^Tx_m q n , m = y n y m x n T x m z = ϕ ( x ) z=\phi(x) z = ϕ ( x )
我们发现计算性能瓶颈在于q n , m = y n y m z n T z m q_{n,m}=y_ny_mz_n^Tz_m q n , m = y n y m z n T z m z = ϕ ( x ) z=\phi(x) z = ϕ ( x ) z = ϕ ( x ) z=\phi(x) z = ϕ ( x ) ϕ ( x n ) ϕ ( x m ) \phi(x_n)\phi(x_m) ϕ ( x n ) ϕ ( x m ) ϕ 2 ( x ) = ( 1 , x 1 , x 2 . . . , x d , x 1 2 , x 1 x 2 , . . . , x 1 x d , x 2 2 , . . . , x 2 x d , . . . , x d 2 ) \phi_2(x)=(1,x_1,x_2...,x_d,x_1^2,x_1x_2,...,x_1x_d,x_2^2,...,x_2x_d,...,x_d^2) ϕ 2 ( x ) = ( 1 , x 1 , x 2 . . . , x d , x 1 2 , x 1 x 2 , . . . , x 1 x d , x 2 2 , . . . , x 2 x d , . . . , x d 2 ) x 1 x 2 , x 2 x 1 x_1x_2,x_2x_1 x 1 x 2 , x 2 x 1 ϕ ( x ) T ϕ ( x ′ ) = 1 + ∑ i = 1 d x i x i ′ + ∑ i = 1 d ∑ j = 1 d x i x j x i ′ x j ′ = 1 + ∑ i = 1 d x i x i ′ + ∑ i = 1 d x i x i ′ ∑ j = 1 d x j x j ′ = 1 + x T x ′ + ( x T x ′ ) ( x T x ′ ) \begin{aligned}\phi(\mathbf x)^T\phi(\mathbf x')&=1+\sum\limits_{i=1}^dx_ix'_i+\sum\limits_{i=1}^d\sum\limits_{j=1}^dx_ix_jx'_ix'_j\\&=1+\sum\limits_{i=1}^dx_ix'_i+\sum\limits_{i=1}^dx_ix'_i\sum\limits_{j=1}^dx_jx'_j\\&=1+\mathbf {x}^T\mathbf {x}'+(\mathbf {x}^T\mathbf {x}')(\mathbf {x}^T\mathbf {x}')\end{aligned} ϕ ( x ) T ϕ ( x ′ ) = 1 + i = 1 ∑ d x i x i ′ + i = 1 ∑ d j = 1 ∑ d x i x j x i ′ x j ′ = 1 + i = 1 ∑ d x i x i ′ + i = 1 ∑ d x i x i ′ j = 1 ∑ d x j x j ′ = 1 + x T x ′ + ( x T x ′ ) ( x T x ′ ) z n z m = ϕ ( x n ) ϕ ( x m ) = 1 + x T x ′ + ( x T x ′ ) ( x T x ′ ) z_nz_m=\phi(x_n)\phi(x_m)=1+x^T{x}'+({x}^T{x}')({x}^T {x}') z n z m = ϕ ( x n ) ϕ ( x m ) = 1 + x T x ′ + ( x T x ′ ) ( x T x ′ ) ϕ ( x ) ϕ ( x ′ ) \phi(x)\phi(x') ϕ ( x ) ϕ ( x ′ )
对系数进行相应的放缩可以得到如下的式子。Φ 2 ( x ) = ( 1 , x 1 , … , x d , x 1 2 , … , x d 2 ) ⇔ K Φ 2 ( x , x ′ ) = 1 + x T x ′ + ( x T x ′ ) 2 Φ 2 ( x ) = ( 1 , 2 x 1 , … , 2 x d , x 1 2 , … , x d 2 ) ⇔ K 2 ( x , x ′ ) = 1 + 2 x T x ′ + ( x T x ′ ) 2 Φ 2 ( x ) = ( 1 , 2 γ x 1 , … , 2 γ x d , γ x 1 2 , … , γ x d 2 ) ⇔ K 2 ( x , x ′ ) = 1 + 2 γ x T x ′ + γ 2 ( x T x ′ ) 2
\begin{aligned} \Phi_{2}(\mathbf{x})=\left(1, x_{1}, \ldots, x_{d}, x_{1}^{2}, \ldots, x_{d}^{2}\right) & \Leftrightarrow K_{\Phi_{2}}\left(\mathbf{x}, \mathbf{x}^{\prime}\right)=1+\mathbf{x}^{T} \mathbf{x}^{\prime}+\left(\mathbf{x}^{T} \mathbf{x}^{\prime}\right)^{2} \\ \Phi_{2}(\mathbf{x})=\left(1, \sqrt{2} x_{1}, \ldots, \sqrt{2} x_{d}, x_{1}^{2}, \ldots, x_{d}^{2}\right) & \Leftrightarrow K_{2}\left(\mathbf{x}, \mathbf{x}^{\prime}\right)=1+2 \mathbf{x}^{T} \mathbf{x}^{\prime}+\left(\mathbf{x}^{T} \mathbf{x}^{\prime}\right)^{2} \\ \Phi_{2}(\mathbf{x})=\left(1, \sqrt{2 \gamma} x_{1}, \ldots, \sqrt{2 \gamma} x_{d}, \gamma x_{1}^{2}, \ldots, \gamma x_{d}^{2}\right) &\Leftrightarrow K_{2}\left(\mathbf{x}, \mathbf{x}^{\prime}\right)=1+2 \gamma \mathbf{x}^{T} \mathbf{x}^{\prime}+\gamma^{2}\left(\mathbf{x}^{T} \mathbf{x}^{\prime}\right)^{2} \end{aligned}
Φ 2 ( x ) = ( 1 , x 1 , … , x d , x 1 2 , … , x d 2 ) Φ 2 ( x ) = ( 1 , 2 x 1 , … , 2 x d , x 1 2 , … , x d 2 ) Φ 2 ( x ) = ( 1 , 2 γ x 1 , … , 2 γ x d , γ x 1 2 , … , γ x d 2 ) ⇔ K Φ 2 ( x , x ′ ) = 1 + x T x ′ + ( x T x ′ ) 2 ⇔ K 2 ( x , x ′ ) = 1 + 2 x T x ′ + ( x T x ′ ) 2 ⇔ K 2 ( x , x ′ ) = 1 + 2 γ x T x ′ + γ 2 ( x T x ′ ) 2 K 2 ( x , x ′ ) = ( ζ + γ x T x ′ ) 2 with ζ ≥ 0 , γ > 0 K 3 ( x , x ′ ) = ( ζ + γ x T x ′ ) 3 with ζ ≥ 0 , γ > 0 ⋮ K Q ( x , x ′ ) = ( ζ + γ x T x ′ ) Q with ζ ≥ 0 , γ > 0 \begin{aligned}K_2(\mathbf{x,x'})&=(\zeta+\gamma\mathbf{x^Tx'})^2\text{with}\zeta\ge0,\gamma>0\\K_3(\mathbf{x,x'})&=(\zeta+\gamma\mathbf{x^Tx'})^3\text{with}\zeta\ge0,\gamma>0\\&\vdots\\K_Q(\mathbf{x,x'})&=(\zeta+\gamma\mathbf{x^Tx'})^Q\text{with}\zeta\ge0,\gamma>0\end{aligned} K 2 ( x , x ′ ) K 3 ( x , x ′ ) K Q ( x , x ′ ) = ( ζ + γ x T x ′ ) 2 with ζ ≥ 0 , γ > 0 = ( ζ + γ x T x ′ ) 3 with ζ ≥ 0 , γ > 0 ⋮ = ( ζ + γ x T x ′ ) Q with ζ ≥ 0 , γ > 0
已经讨论过关于有限Q维的多项式核函数，现在我们讨论关于无限维的情况，简化问题，我们讨论特征只有一个时的高斯函数K g ( x , x ′ ) = e − ( x − x ′ ) 2 K_g({x,x'})=e^{-(x-x')^2} K g ( x , x ′ ) = e − ( x − x ′ ) 2 K ( x , x ′ ) = exp ( − ( x − x ′ ) 2 ) = exp ( − ( x ) 2 ) exp ( − ( x ′ ) 2 ) exp ( 2 x x ′ ) = T a y l o r exp ( − ( x ) 2 ) exp ( − ( x ′ ) 2 ) ( ∑ i = 0 ∞ ( 2 x x ′ ) i i ! ) = ∑ i = 0 ∞ ( exp ( − ( x ′ ) 2 ) exp ( − ( x ) 2 ) 2 i i ! 2 i i ! ( x ) i ( x ′ ) i ) = Φ ( x ) T Φ ( x ′ ) \begin{aligned}K(\mathbf{x,x'})=&\exp(-(\mathbf{x}-\mathbf{x'})^2)\\=&\exp(-(\mathbf{x})^2)\exp(-(\mathbf{x'})^2)\exp(2\mathbf{x}\mathbf{x}')\\\overset{Taylor}{=}&\exp(-(\mathbf{x})^2)\exp(-(\mathbf{x'})^2)\left( \sum\limits_{i=0}^\infin \frac{(2xx')^i}{i!}\right)\\=&\sum\limits_{i=0}^\infin \left(\exp\left(-\left({x'}\right)^2\right)\exp\left(-\left({x}\right)^2\right)\sqrt \frac{2^i}{i!}\sqrt \frac{2^i}{i!}(x)^i(x')^i\right)\\=&\Phi(\mathbf x)^T\Phi(\mathbf x')\end{aligned} K ( x , x ′ ) = = = T a y l o r = = exp ( − ( x − x ′ ) 2 ) exp ( − ( x ) 2 ) exp ( − ( x ′ ) 2 ) exp ( 2 x x ′ ) exp ( − ( x ) 2 ) exp ( − ( x ′ ) 2 ) ( i = 0 ∑ ∞ i ! ( 2 x x ′ ) i ) i = 0 ∑ ∞ ( exp ( − ( x ′ ) 2 ) exp ( − ( x ) 2 ) i ! 2 i i ! 2 i ( x ) i ( x ′ ) i ) Φ ( x ) T Φ ( x ′ ) Φ ( x ) = e − x 2 ⋅ ( 1 , 2 1 ! x , 2 2 2 ! x 2 , ⋯ )
\Phi(x)=e^{-x^{2}} \cdot\left(1, \sqrt{\frac{2}{1 !}} x, \sqrt{\frac{2^{2}}{2 !}} x^{2}, \cdots\right)
Φ ( x ) = e − x 2 ⋅ ( 1 , 1 ! 2 x , 2 ! 2 2 x 2 , ⋯ ) Φ ( x ) \Phi(x) Φ ( x ) γ > 0 \gamma>0 γ > 0 K g ( x , x ′ ) = e − γ ∣ ∣ x − x ′ ∣ ∣ 2 K_g({x,x'})=e^{-\gamma||x-x'||^2} K g ( x , x ′ ) = e − γ ∣ ∣ x − x ′ ∣ ∣ 2 H = ∑ S V μ n y n exp ( − γ ∣ ∣ x − x n ∣ ∣ 2 + b ) H=\sum\limits_{SV}\mu_ny_n \exp(-\gamma||\mathbf {x-x_n}||^2+b) H = S V ∑ μ n y n exp ( − γ ∣ ∣ x − x n ∣ ∣ 2 + b ) γ \gamma γ γ \gamma γ γ \gamma γ γ \gamma γ γ = 1 2 σ 2 \gamma=\frac{1}{2\sigma^2} γ = 2 σ 2 1
我们考虑一下核函数本质是什么。我们可以某种程度上认为，核函数考虑的就是两个数据经过特征转换后的相似度，在这里是内积。但不是任何计算相似度的就可以认为是核函数，一个有效的核函数必须得满足Mercer定理。K = [ Φ ( x 1 ) T Φ ( x 1 ) Φ ( x 1 ) T Φ ( x 2 ) … Φ ( x 1 ) T Φ ( x N ) Φ ( x 2 ) T Φ ( x 1 ) Φ ( x 2 ) T Φ ( x 2 ) … Φ ( x 2 ) T Φ ( x N ) … ⋯ ⋯ ⋯ Φ ( x N ) T Φ ( x 1 ) Φ ( x N ) T Φ ( x 2 ) ⋯ Φ ( x N ) T Φ ( x N ) ]
K=\left[\begin{array}{cccc}{\Phi\left(\mathbf{x}_{1}\right)^{T} \Phi\left(\mathbf{x}_{1}\right)} & {\Phi\left(\mathbf{x}_{1}\right)^{T} \Phi\left(\mathbf{x}_{2}\right)} & {\dots} & {\mathbf{\Phi}\left(\mathbf{x}_{1}\right)^{T} \Phi\left(\mathbf{x}_{N}\right)} \\ {\Phi\left(\mathbf{x}_{2}\right)^{T} \Phi\left(\mathbf{x}_{1}\right)} & {\Phi\left(\mathbf{x}_{2}\right)^{T} \Phi\left(\mathbf{x}_{2}\right)} & {\ldots} & {\Phi\left(\mathbf{x}_{2}\right)^{T} \Phi\left(\mathbf{x}_{N}\right)} \\ {{\ldots}} & {\cdots} & {\cdots} & {\cdots} & {} \\ {\Phi\left(\mathbf{x}_{N}\right)^{T} \Phi\left(\mathbf{x}_{1}\right)} & {\Phi\left(\mathbf{x}_{N}\right)^{T} \Phi\left(\mathbf{x}_{2}\right)} & {\cdots} & {\Phi\left(\mathbf{x}_{N}\right)^{T} \Phi\left(\mathbf{x}_{N}\right)}\end{array}\right]
K = ⎣ ⎢ ⎢ ⎡ Φ ( x 1 ) T Φ ( x 1 ) Φ ( x 2 ) T Φ ( x 1 ) … Φ ( x N ) T Φ ( x 1 ) Φ ( x 1 ) T Φ ( x 2 ) Φ ( x 2 ) T Φ ( x 2 ) ⋯ Φ ( x N ) T Φ ( x 2 ) … … ⋯ ⋯ Φ ( x 1 ) T Φ ( x N ) Φ ( x 2 ) T Φ ( x N ) ⋯ Φ ( x N ) T Φ ( x N ) ⎦ ⎥ ⎥ ⎤
K是对称的
K是半正定的
可以想象SVM仍然会造成过拟合的情况发生，有两个原因，一是我们使用的核函数过于强大，SVM过于复杂，转换的维度太高。另外一个原因是我们坚持所有样本必须线性可分（即使是映射到高维空间仍然如此要求），不允许错误存在，造成的后果就是对数据点异常敏感，使得模型过于复杂。
因此我们想要降低这种过拟合的可能性，通过把严格线性可分，不允许错误分类这一条件进行松弛，变成允许错误分类，但尽可能的少。因此把硬间隔支持向量机进行适当地改造如下min b , w 1 2 w T w + C ⋅ ∑ n = 1 N [ [ y n ≠ sign ( w T x n + b ) ] ] s.t. y n ( w T x n + b ) ≥ 1 − ∞ ⋅ [ [ y n ≠ sign ( w T x n + b ) ] ]
\begin{array}{cl}{\min\limits_{b, w}} & {\frac{1}{2} w^{T} w+C \cdot \sum\limits_{n=1}^{N}[[y_{n} \neq \operatorname{sign}\left(w^{T} \mathbf{x}_{n}+b\right)]]} \\ {\text { s.t. }} & {y_{n}\left(\mathbf{w}^{T} \mathbf{x}_{n}+b\right) \geq 1-\infty \cdot[[ y_{n} \neq \operatorname{sign}\left(\mathbf{w}^{T} \mathbf{x}_{n}+b\right)]]}\end{array}
b , w min s.t. 2 1 w T w + C ⋅ n = 1 ∑ N [ [ y n = s i g n ( w T x n + b ) ] ] y n ( w T x n + b ) ≥ 1 − ∞ ⋅ [ [ y n = s i g n ( w T x n + b ) ] ] y n ( w T x n + b ) ≥ 1 y_{n}\left(\mathbf{w}^{T} \mathbf{x}_{n}+b\right) \geq 1 y n ( w T x n + b ) ≥ 1 y n ( w T x n + b ) ≥ − ∞ y_{n}\left(\mathbf{w}^{T} \mathbf{x}_{n}+b\right) \geq -\infin y n ( w T x n + b ) ≥ − ∞
但是上述的表示虽然直观，但是难以使用数学工具去最优化(QP标准包)，而且对于离边界近的错分点与离边界远的错分点，两者在此表示中是等价的，没有区别的。因此我们进行修正，引入ξ \xi ξ ξ = 0 \xi=0 ξ = 0 min b , w , ξ 1 2 w T w + C ⋅ ∑ n = 1 N ξ n s.t. y n ( w T x n + b ) ≥ 1 − ξ n ξ n ≥ 0 for all n
\begin{aligned}\min\limits_{b, w, \xi} &{\frac{1}{2} w^{T} w+C \cdot \sum_{n=1}^{N} \xi_{n}} \\ \text { s.t. } &y_{n}\left(\mathbf{w}^{T} \mathbf{x}_{n}+b\right) \geq 1-\xi_{n} \\ &\xi_{n} \geq 0 \text { for all } n\end{aligned}
b , w , ξ min s.t. 2 1 w T w + C ⋅ n = 1 ∑ N ξ n y n ( w T x n + b ) ≥ 1 − ξ n ξ n ≥ 0 for all n L ( w , b , ξ , α , β ) = 1 2 w T w + C ⋅ ∑ n = 1 N ξ n + ∑ n = 1 N α n ( 1 − ξ n − y n ( w T x n + b ) ) + ∑ n = 1 N β n ( − ξ n ) L(w,b,\xi,\alpha,\beta)={\frac{1}{2} w^{T} w+C \cdot \sum_{n=1}^{N} \xi_{n}}+\sum\limits_{n=1}^N\alpha_n(1-\xi_{n}-y_n(\mathbf{w}^{T} \mathbf{x}_{n}+b))+\sum\limits_{n=1}^N\beta_n(-\xi_n) L ( w , b , ξ , α , β ) = 2 1 w T w + C ⋅ n = 1 ∑ N ξ n + n = 1 ∑ N α n ( 1 − ξ n − y n ( w T x n + b ) ) + n = 1 ∑ N β n ( − ξ n ) min b , w , ξ max α , β 1 2 w T w + C ⋅ ∑ n = 1 N ξ n + ∑ n = 1 N α n ( 1 − ξ n − y n ( w T x n + b ) ) + ∑ n = 1 N β n ( − ξ n ) s.t. α n ≥ 0 , β n ≥ 0 \begin{aligned}\min\limits_{b, w, \xi} \max\limits_{\alpha,\beta}&{\frac{1}{2} w^{T} w+C \cdot \sum_{n=1}^{N} \xi_{n}}+\sum\limits_{n=1}^N\alpha_n(1-\xi_{n}-y_n(\mathbf{w}^{T} \mathbf{x}_{n}+b))+\sum\limits_{n=1}^N\beta_n(-\xi_n) \\ \text { s.t. } &\alpha_n \ge 0,\beta_n\ge 0\end{aligned} b , w , ξ min α , β max s.t. 2 1 w T w + C ⋅ n = 1 ∑ N ξ n + n = 1 ∑ N α n ( 1 − ξ n − y n ( w T x n + b ) ) + n = 1 ∑ N β n ( − ξ n ) α n ≥ 0 , β n ≥ 0 max α , β min b , w , ξ 1 2 w T w + C ⋅ ∑ n = 1 N ξ n + ∑ n = 1 N α n ( 1 − ξ n − y n ( w T x n + b ) ) + ∑ n = 1 N β n ( − ξ n ) s.t. α n ≥ 0 , β n ≥ 0 \begin{aligned} \max\limits_{\alpha,\beta}\min\limits_{b, w, \xi}&{\frac{1}{2} w^{T} w+C \cdot \sum_{n=1}^{N} \xi_{n}}+\sum\limits_{n=1}^N\alpha_n(1-\xi_{n}-y_n(\mathbf{w}^{T} \mathbf{x}_{n}+b))+\sum\limits_{n=1}^N\beta_n(-\xi_n) \\ \text { s.t. } &\alpha_n \ge 0,\beta_n\ge 0\end{aligned} α , β max b , w , ξ min s.t. 2 1 w T w + C ⋅ n = 1 ∑ N ξ n + n = 1 ∑ N α n ( 1 − ξ n − y n ( w T x n + b ) ) + n = 1 ∑ N β n ( − ξ n ) α n ≥ 0 , β n ≥ 0 ∂ L ∂ ξ n = C − α n − β n = 0 \begin{aligned}\frac{\partial L}{\partial \xi_n}=C-\alpha_n-\beta_n=0\end{aligned} ∂ ξ n ∂ L = C − α n − β n = 0 β n ≥ 0 \beta_n \ge0 β n ≥ 0 0 ≤ α ≤ C 0\le\alpha\le C 0 ≤ α ≤ C β n = C − α n \beta_n=C-\alpha_n β n = C − α n max α , β min b , w 1 2 w T w + ∑ n = 1 N α n ( 1 − y n ( w T x n + b ) ) s.t. C ≥ α n ≥ 0 , β n = C − α n \begin{aligned} \max\limits_{\alpha,\beta}\min\limits_{b, w}&{\frac{1}{2} w^{T} w}+\sum\limits_{n=1}^N\alpha_n(1-y_n(\mathbf{w}^{T} \mathbf{x}_{n}+b)) \\ \text { s.t. } &C\ge\alpha_n \ge 0,\beta_n= C-\alpha_n\end{aligned} α , β max b , w min s.t. 2 1 w T w + n = 1 ∑ N α n ( 1 − y n ( w T x n + b ) ) C ≥ α n ≥ 0 , β n = C − α n ∑ n = 1 N α n y n = 0 w = ∑ n = 1 N α n y n x n
\begin{array}{l}{\sum\limits_{n=1}^{N} \alpha_{n} y_{n}=0} \\ {w=\sum\limits_{n=1}^{N} \alpha_{n} y_{n} x_{n}}\end{array}
n = 1 ∑ N α n y n = 0 w = n = 1 ∑ N α n y n x n min α 1 2 ∑ i = 1 N ∑ j = 1 N α i α j y i y j x i T x j − ∑ i = 1 N α i s . t . ∑ i = 1 N α i y i = 0 C ≥ α i ≥ 0 , i = 1 , 2... , n β i = C − α i , i = 1 , 2... , n \begin{array}{cl}\min\limits_\alpha &\frac{1}{2}\sum\limits_{i=1}^N\sum\limits_{j=1}^N\alpha _i\alpha _jy_iy_jx_i^Tx_j-\sum\limits_{i=1}^N\alpha _i\\{s.t.}&\sum\limits_{i=1}^N\alpha _i y_i=0\\&C\ge\alpha _i \ge0 ,i=1,2...,n \\ &\beta_i=C-\alpha_i,i=1,2...,n\end{array}
α min s . t . 2 1 i = 1 ∑ N j = 1 ∑ N α i α j y i y j x i T x j − i = 1 ∑ N α i i = 1 ∑ N α i y i = 0 C ≥ α i ≥ 0 , i = 1 , 2 . . . , n β i = C − α i , i = 1 , 2 . . . , n α \alpha α w = ∑ n = 1 N α n y n x n w=\sum\limits_{n=1}^{N} \alpha_{n} y_{n} x_{n} w = n = 1 ∑ N α n y n x n { α n ( 1 − ξ n − y n ( w T x n + b ) ) = 0 ( C − α n ) ( − ξ n ) = 0 \left\{\begin{aligned}&\alpha_n(1-\xi_{n}-y_n(\mathbf{w}^{T} \mathbf{x}_{n}+b))=0\\ &(C-\alpha_n)(-\xi_n)=0\end{aligned} \right. { α n ( 1 − ξ n − y n ( w T x n + b ) ) = 0 ( C − α n ) ( − ξ n ) = 0 0 < α n < C 0< \alpha_n< C 0 < α n < C ξ n = 0 \xi_n=0 ξ n = 0 b = y n − w T x n b=y_n-\mathbf{w}^{T} x_{n} b = y n − w T x n α n = 0 \alpha_n=0 α n = 0 ξ n = 0 \xi_n=0 ξ n = 0 α n = C \alpha_n=C α n = C ξ n \xi_n ξ n ξ n = 1 − y n ( w T x n + b ) \xi_n=1-y_n(\mathbf{w}^{T} \mathbf{x}_{n}+b) ξ n = 1 − y n ( w T x n + b ) ξ n = 0 \xi_n=0 ξ n = 0 0 < ξ n < 1 0<\xi_n<1 0 < ξ n < 1 ξ n = 1 \xi_n=1 ξ n = 1 ξ n > 1 \xi_n>1 ξ n > 1 α , ξ \alpha,\xi α , ξ
我们之前提到过可以使用QP标准库来计算出强对偶问题中最优的情况，但是存在一个Q稠密矩阵，这个稠密矩阵大小与样本量有关，举个例子,当样本量N>30000时，将会耗费3GB的内存。所以使用QP标准包来解决这个问题似乎效率会比较低下，因此我们产生两个想法
动态的计算q n m q_{nm} q n m
采用更高效的算法
我们接下来要讨论的就是platt在1998年提出的序列最小最优化SMO算法
考虑最优化问题形如min α 1 2 ∑ i = 1 N ∑ j = 1 N α i α j y i y j K ( x i , x j ) − ∑ i = 1 N α i s . t . ∑ i = 1 N α i y i = 0 C ≥ α i ≥ 0 , i = 1 , 2... , n \begin{array}{cl}\min\limits_\alpha &\frac{1}{2}\sum\limits_{i=1}^N\sum\limits_{j=1}^N\alpha _i\alpha _jy_iy_jK(x_i,x_j)-\sum\limits_{i=1}^N\alpha _i\\{s.t.}&\sum\limits_{i=1}^N\alpha _i y_i=0\\&C\ge\alpha _i \ge0 ,i=1,2...,n \end{array} α min s . t . 2 1 i = 1 ∑ N j = 1 ∑ N α i α j y i y j K ( x i , x j ) − i = 1 ∑ N α i i = 1 ∑ N α i y i = 0 C ≥ α i ≥ 0 , i = 1 , 2 . . . , n 0 < α n < C 0< \alpha_n< C 0 < α n < C 1 − ( y i ∑ i = 1 N α i y i K ( x i , x n ) + b ) = 0 1-(y_i\sum\limits_{i=1}^N\alpha_iy_iK(x_i,x_n)+b)=0 1 − ( y i i = 1 ∑ N α i y i K ( x i , x n ) + b ) = 0 α n = 0 \alpha_n=0 α n = 0 1 − ( y i ∑ i = 1 N α i y i K ( x i , x n ) + b ) ≤ 0 1-(y_i\sum\limits_{i=1}^N\alpha_iy_iK(x_i,x_n)+b)\le0 1 − ( y i i = 1 ∑ N α i y i K ( x i , x n ) + b ) ≤ 0 α n = C \alpha_n=C α n = C ξ n = 1 − ( y i ∑ i = 1 N α i y i K ( x i , x n ) + b ) ≥ 0 \xi_n=1-(y_i\sum\limits_{i=1}^N\alpha_iy_iK(x_i,x_n)+b)\ge0 ξ n = 1 − ( y i i = 1 ∑ N α i y i K ( x i , x n ) + b ) ≥ 0 ξ n = 0 \xi_n=0 ξ n = 0 0 < ξ n < 1 0<\xi_n<1 0 < ξ n < 1 ξ n = 1 \xi_n=1 ξ n = 1 ξ n > 1 \xi_n>1 ξ n > 1
**为什么一次要对两个变量进行优化？**注意上述问题中有等式约束∑ i = 1 N α i y i = 0 \sum\limits_{i=1}^N\alpha _i y_i=0 i = 1 ∑ N α i y i = 0 α 2 \alpha_2 α 2 α 1 \alpha_1 α 1 α 1 = − y 1 ∑ i = 2 N α i y i \alpha_1=-y_1\sum\limits_{i=2}^N\alpha_iy_i α 1 = − y 1 i = 2 ∑ N α i y i α 1 , α 2 \alpha_1,\alpha_2 α 1 , α 2 α i ( i = 3 , ⋯ , N ) \alpha_i(i=3,\cdots,N) α i ( i = 3 , ⋯ , N ) min α 1 , α 2 W ( α 1 , α 2 ) = 1 2 ∑ i = 1 N ( ∑ j = 1 2 y i y j α i α j K ( x i , x j ) + ∑ j = 3 N y i y j α i α j K ( x i , x j ) ) − ( α 1 + α 2 ) − ∑ i = 3 N α i = 1 2 ∑ i = 1 2 ( ∑ j = 1 2 y i y j α i α j K ( x i , x j ) + ∑ j = 3 N y i y j α i α j K ( x i , x j ) ) + 1 2 ∑ i = 3 N ( ∑ j = 1 2 y i y j α i α j K ( x i , x j ) + ∑ j = 3 N y i y j α i α j K ( x i , x j ) ) − ( α 1 + α 2 ) − ∑ i = 3 N α i = 1 2 ∑ i = 1 2 ∑ j = 1 2 y i y j α i α j K ( x i , x j ) + ∑ i = 1 2 ∑ j = 3 N y i y j α i α j K ( x i , x j ) + 1 2 ∑ i = 3 N ∑ j = 3 N y i y j α i α j K ( x i , x j ) − ( α 1 + α 2 ) − ∑ i = 3 N α i = 1 2 K ( x 1 , x 1 ) α 1 2 + 1 2 K ( x 2 , x 2 ) α 2 2 + y 1 y 2 K ( x 1 , x 2 ) α 1 α 2 − ( α 1 + α 2 ) + y 1 α 1 ∑ i = 3 N y i α i K ( x i , x 1 ) + y 2 α 2 ∑ i = 3 N y i α i K ( x i , x 2 ) s . t . α 1 y 1 + α 2 y 2 = − ∑ i = 3 N y i α i = ζ 0 ≤ α j ≤ C , j = 1 , 2 \begin{aligned}\min\limits_{\alpha_1,\alpha_2}W(\alpha_1,\alpha_2)&=
\frac{1}{2} \sum_{i=1}^{N}\left(\sum_{j=1}^{2} y_{i} y_{j} \alpha_{i} \alpha_{j} K(x_{i}, x_{j})+\sum_{j=3}^{N} y_{i} y_{j} \alpha_{i} \alpha_{j} K(x_{i} ,x_{j})\right)-(\alpha_{1}+\alpha_{2})-\sum_{i=3}^{N} \alpha_{i}\\&=\frac{1}{2} \sum_{i=1}^{2}\left(\sum_{j=1}^{2} y_{i} y_{j} \alpha_{i} \alpha_{j} K(x_{i}, x_{j})+\sum_{j=3}^{N} y_{i} y_{j} \alpha_{i} \alpha_{j} K(x_{i} ,x_{j})\right)+\frac{1}{2} \sum_{i=3}^{N}\left(\sum_{j=1}^{2} y_{i} y_{j} \alpha_{i} \alpha_{j} K(x_{i}, x_{j})+\sum_{j=3}^{N} y_{i} y_{j} \alpha_{i} \alpha_{j} K(x_{i} ,x_{j})\right)-(\alpha_{1}+\alpha_{2})-\sum_{i=3}^{N} \alpha_{i}\\&=\frac{1}{2} \sum_{i=1}^{2}\sum_{j=1}^{2} y_{i} y_{j} \alpha_{i} \alpha_{j} K(x_{i}, x_{j})+\sum_{i=1}^{2}\sum_{j=3}^{N} y_{i} y_{j} \alpha_{i} \alpha_{j} K(x_{i} ,x_{j})+\frac{1}{2}\sum_{i=3}^{N}\sum_{j=3}^{N} y_{i} y_{j} \alpha_{i} \alpha_{j} K(x_{i} ,x_{j})-(\alpha_{1}+\alpha_{2})-\sum_{i=3}^{N} \alpha_{i}\\&=\frac{1}{2}K(x_1,x_1)\alpha_1^2+\frac{1}{2}K(x_2,x_2)\alpha_2^2+y_1y_2K(x_1,x_2)\alpha_1\alpha_2-(\alpha_1+\alpha_2)+y_1\alpha_1\sum_{i=3}^Ny_i\alpha_iK(x_i,x_1)+y_2\alpha_2\sum_{i=3}^Ny_i\alpha_iK(x_i,x_2)\\{s.t.}\qquad&\alpha_1y_1+\alpha_2y_2=-\sum_{i=3}^Ny_i\alpha_i=\zeta\\&0\le\alpha_j\le C,j=1,2
\end{aligned} α 1 , α 2 min W ( α 1 , α 2 ) s . t . = 2 1 i = 1 ∑ N ( j = 1 ∑ 2 y i y j α i α j K ( x i , x j ) + j = 3 ∑ N y i y j α i α j K ( x i , x j ) ) − ( α 1 + α 2 ) − i = 3 ∑ N α i = 2 1 i = 1 ∑ 2 ( j = 1 ∑ 2 y i y j α i α j K ( x i , x j ) + j = 3 ∑ N y i y j α i α j K ( x i , x j ) ) + 2 1 i = 3 ∑ N ( j = 1 ∑ 2 y i y j α i α j K ( x i , x j ) + j = 3 ∑ N y i y j α i α j K ( x i , x j ) ) − ( α 1 + α 2 ) − i = 3 ∑ N α i = 2 1 i = 1 ∑ 2 j = 1 ∑ 2 y i y j α i α j K ( x i , x j ) + i = 1 ∑ 2 j = 3 ∑ N y i y j α i α j K ( x i , x j ) + 2 1 i = 3 ∑ N j = 3 ∑ N y i y j α i α j K ( x i , x j ) − ( α 1 + α 2 ) − i = 3 ∑ N α i = 2 1 K ( x 1 , x 1 ) α 1 2 + 2 1 K ( x 2 , x 2 ) α 2 2 + y 1 y 2 K ( x 1 , x 2 ) α 1 α 2 − ( α 1 + α 2 ) + y 1 α 1 i = 3 ∑ N y i α i K ( x i , x 1 ) + y 2 α 2 i = 3 ∑ N y i α i K ( x i , x 2 ) α 1 y 1 + α 2 y 2 = − i = 3 ∑ N y i α i = ζ 0 ≤ α j ≤ C , j = 1 , 2 y 1 y 2 < 0 y_1y_2<0 y 1 y 2 < 0 α 1 o l d , α 2 o l d \alpha_1^{old},\alpha_2^{old} α 1 o l d , α 2 o l d α 1 n e w , α 2 n e w \alpha_1^{new},\alpha_2^{new} α 1 n e w , α 2 n e w α 1 o l d − α 2 o l d = α 1 n e w − α 2 n e w = ζ 0 ≤ α 1 ≤ C 0 ≤ α 2 ≤ C \alpha_1^{old}-\alpha_2^{old}=\alpha_1^{new}-\alpha_2^{new}=\zeta\\0\le\alpha_1\le C\\0\le\alpha_2\le C α 1 o l d − α 2 o l d = α 1 n e w − α 2 n e w = ζ 0 ≤ α 1 ≤ C 0 ≤ α 2 ≤ C 0 ≤ α 2 ≤ C α 2 o l d − α 1 o l d ≤ α 2 ≤ C + α 2 o l d − α 1 o l d 0\le\alpha_2\le C\\\alpha_2^{old}-\alpha_1^{old}\le\alpha_2\le C+\alpha_2^{old}-\alpha_1^{old} 0 ≤ α 2 ≤ C α 2 o l d − α 1 o l d ≤ α 2 ≤ C + α 2 o l d − α 1 o l d L = max ( α 2 o l d − α 1 o l d , 0 ) ≤ α 2 ≤ min ( C , C + α 2 o l d − α 1 o l d ) = H L=\max(\alpha_2^{old}-\alpha_1^{old},0)\le\alpha_2\le \min(C,C+\alpha_2^{old}-\alpha_1^{old})=H L = max ( α 2 o l d − α 1 o l d , 0 ) ≤ α 2 ≤ min ( C , C + α 2 o l d − α 1 o l d ) = H y 1 y 2 > 0 y_1y_2>0 y 1 y 2 > 0 L = max ( α 2 o l d + α 1 o l d − C , 0 ) ≤ α 2 ≤ min ( C , α 2 o l d + α 1 o l d ) = H L=\max(\alpha_2^{old}+\alpha_1^{old}-C,0)\le\alpha_2\le \min(C,\alpha_2^{old}+\alpha_1^{old})=H L = max ( α 2 o l d + α 1 o l d − C , 0 ) ≤ α 2 ≤ min ( C , α 2 o l d + α 1 o l d ) = H L ≤ α 2 n e w ≤ H L\le \alpha_2^{new} \le H L ≤ α 2 n e w ≤ H α 1 y 1 + α 2 y 2 = − ∑ i = 3 N y i α i = ζ \alpha_1y_1+\alpha_2y_2=-\sum\limits_{i=3}^Ny_i\alpha_i=\zeta α 1 y 1 + α 2 y 2 = − i = 3 ∑ N y i α i = ζ 0 ≤ α ≤ C 0\le\alpha\le C 0 ≤ α ≤ C min α 1 , α 2 W ( α 1 , α 2 ) = 1 2 K ( x 1 , x 1 ) α 1 2 + 1 2 K ( x 2 , x 2 ) α 2 2 + y 1 y 2 K ( x 1 , x 2 ) α 1 α 2 − ( α 1 + α 2 ) + y 1 α 1 ∑ i = 3 N y i α i K ( x i , x 1 ) + y 2 α 2 ∑ i = 3 N y i α i K ( x i , x 2 ) \begin{aligned}\min\limits_{\alpha_1,\alpha_2}W(\alpha_1,\alpha_2)&=\frac{1}{2}K(x_1,x_1)\alpha_1^2+\frac{1}{2}K(x_2,x_2)\alpha_2^2+y_1y_2K(x_1,x_2)\alpha_1\alpha_2-(\alpha_1+\alpha_2)+y_1\alpha_1\sum_{i=3}^Ny_i\alpha_iK(x_i,x_1)+y_2\alpha_2\sum_{i=3}^Ny_i\alpha_iK(x_i,x_2)\end{aligned} α 1 , α 2 min W ( α 1 , α 2 ) = 2 1 K ( x 1 , x 1 ) α 1 2 + 2 1 K ( x 2 , x 2 ) α 2 2 + y 1 y 2 K ( x 1 , x 2 ) α 1 α 2 − ( α 1 + α 2 ) + y 1 α 1 i = 3 ∑ N y i α i K ( x i , x 1 ) + y 2 α 2 i = 3 ∑ N y i α i K ( x i , x 2 ) α 1 y 1 = ζ − α 2 y 2 , y i 2 = 1 \alpha_1y_1=\zeta-\alpha_2y_2,y_i^2=1 α 1 y 1 = ζ − α 2 y 2 , y i 2 = 1 α 1 = ( ζ − y 2 α 2 ) y 1 \alpha_1=(\zeta-y_2\alpha_2)y_1 α 1 = ( ζ − y 2 α 2 ) y 1 α 2 \alpha_2 α 2 min α 2 W ( α 2 ) = 1 2 K ( x 1 , x 1 ) α 1 2 + 1 2 K ( x 2 , x 2 ) α 2 2 + y 1 y 2 K ( x 1 , x 2 ) α 1 α 2 − ( α 1 + α 2 ) + y 1 α 1 ∑ i = 3 N y i α i K ( x i , x 1 ) + y 2 α 2 ∑ i = 3 N y i α i K ( x i , x 2 ) = 1 2 K ( x 1 , x 1 ) ( ζ − y 2 α 2 ) 2 + 1 2 K ( x 2 , x 2 ) α 2 2 + y 2 K ( x 1 , x 2 ) ( ζ − y 2 α 2 ) α 2 − ( ζ − y 2 α 2 ) y 1 − α 2 + ( ζ − y 2 α 2 ) ∑ i = 3 N y i α i K ( x i , x 1 ) + y 2 α 2 ∑ i = 3 N y i α i K ( x i , x 2 ) \begin{aligned}\min\limits_{\alpha_2}W(\alpha_2)&=\frac{1}{2}K(x_1,x_1)\alpha_1^2+\frac{1}{2}K(x_2,x_2)\alpha_2^2+y_1y_2K(x_1,x_2)\alpha_1\alpha_2-(\alpha_1+\alpha_2)+y_1\alpha_1\sum_{i=3}^Ny_i\alpha_iK(x_i,x_1)+y_2\alpha_2\sum_{i=3}^Ny_i\alpha_iK(x_i,x_2)\\&=\frac{1}{2}K(x_1,x_1)(\zeta-y_2\alpha_2)^2+\frac{1}{2}K(x_2,x_2)\alpha_2^2+y_2K(x_1,x_2)(\zeta-y_2\alpha_2)\alpha_2-(\zeta-y_2\alpha_2)y_1-\alpha_2+(\zeta-y_2\alpha_2)\sum_{i=3}^Ny_i\alpha_iK(x_i,x_1)+y_2\alpha_2\sum_{i=3}^Ny_i\alpha_iK(x_i,x_2)\end{aligned} α 2 min W ( α 2 ) = 2 1 K ( x 1 , x 1 ) α 1 2 + 2 1 K ( x 2 , x 2 ) α 2 2 + y 1 y 2 K ( x 1 , x 2 ) α 1 α 2 − ( α 1 + α 2 ) + y 1 α 1 i = 3 ∑ N y i α i K ( x i , x 1 ) + y 2 α 2 i = 3 ∑ N y i α i K ( x i , x 2 ) = 2 1 K ( x 1 , x 1 ) ( ζ − y 2 α 2 ) 2 + 2 1 K ( x 2 , x 2 ) α 2 2 + y 2 K ( x 1 , x 2 ) ( ζ − y 2 α 2 ) α 2 − ( ζ − y 2 α 2 ) y 1 − α 2 + ( ζ − y 2 α 2 ) i = 3 ∑ N y i α i K ( x i , x 1 ) + y 2 α 2 i = 3 ∑ N y i α i K ( x i , x 2 ) α 2 \alpha_2 α 2 ∂ 2 W ∂ α 2 2 = K ( x 1 , x 1 ) − 2 K ( x 1 , x 2 ) + K ( x 2 , x 2 ) = ∣ ∣ Φ ( x 1 ) − Φ ( x 2 ) ∣ ∣ 2 ≥ 0 \begin{aligned}\frac{\partial^2 W}{\partial \alpha_2^2}&=K(x_1,x_1)-2K(x_1,x_2)+K(x_2,x_2)\\&=||\Phi(x_1)-\Phi(x_2)||^2\\& \ge 0\end{aligned} ∂ α 2 2 ∂ 2 W = K ( x 1 , x 1 ) − 2 K ( x 1 , x 2 ) + K ( x 2 , x 2 ) = ∣ ∣ Φ ( x 1 ) − Φ ( x 2 ) ∣ ∣ 2 ≥ 0 Φ ( x i ) \Phi(x_i) Φ ( x i ) α 2 \alpha_2 α 2 ∂ W ∂ α 2 = 0 \frac{\partial W}{\partial \alpha_2}=0 ∂ α 2 ∂ W = 0 ( K ( x 1 , x 1 ) + K ( x 2 , x 2 ) − 2 K ( x 1 , x 2 ) ) α 2 = y 2 ( y 2 − y 1 + ζ K ( x 1 , x 1 ) − ζ K ( x 1 , x 2 ) + ∑ i = 3 N y i α i K ( x i , x 1 ) − ∑ i = 3 N y i α i K ( x i , x 2 ) ) = y 2 ( y 2 − y 1 + ζ K ( x 1 , x 1 ) − ζ K ( x 2 , x 2 ) + ( ∑ i = 1 N y i α i K ( x i , x 1 ) − ∑ j = 1 2 y j α j K ( x j , x 1 ) + b − b ) − ( ∑ i = 1 N y i α i K ( x i , x 2 ) − ∑ j = 1 2 y j α j K ( x j , x 2 ) + b − b ) ) \begin{aligned}(K(x_1,x_1)+K(x_2,x_2)-2K(x_1,x_2))\alpha_2&=y_2(y_2-y_1+\zeta K(x_1,x_1)-\zeta K(x_1,x_2)+\sum_{i=3}^Ny_i\alpha_iK(x_i,x_1)-\sum_{i=3}^Ny_i\alpha_iK(x_i,x_2))\\&=y_2\left(y_2-y_1+\zeta K(x_1,x_1)-\zeta K(x_2,x_2)+\left(\sum_{i=1}^Ny_i\alpha_iK(x_i,x_1)-\sum_{j=1}^2y_j\alpha_jK(x_j,x_1)+b-b\right)-\left(\sum_{i=1}^Ny_i\alpha_iK(x_i,x_2)-\sum_{j=1}^2y_j\alpha_jK(x_j,x_2)+b-b\right)\right)\end{aligned} ( K ( x 1 , x 1 ) + K ( x 2 , x 2 ) − 2 K ( x 1 , x 2 ) ) α 2 = y 2 ( y 2 − y 1 + ζ K ( x 1 , x 1 ) − ζ K ( x 1 , x 2 ) + i = 3 ∑ N y i α i K ( x i , x 1 ) − i = 3 ∑ N y i α i K ( x i , x 2 ) ) = y 2 ( y 2 − y 1 + ζ K ( x 1 , x 1 ) − ζ K ( x 2 , x 2 ) + ( i = 1 ∑ N y i α i K ( x i , x 1 ) − j = 1 ∑ 2 y j α j K ( x j , x 1 ) + b − b ) − ( i = 1 ∑ N y i α i K ( x i , x 2 ) − j = 1 ∑ 2 y j α j K ( x j , x 2 ) + b − b ) ) ζ = α 1 o l d y 1 + α 2 o l d y 2 \zeta=\alpha_1^{old}y_1+\alpha_2^{old}y_2 ζ = α 1 o l d y 1 + α 2 o l d y 2 ( K ( x 1 , x 1 ) + K ( x 2 , x 2 ) − 2 K ( x 1 , x 2 ) ) α 2 n e w , u n c = y 2 ( ( K ( x 1 , x 1 ) + K ( x 2 , x 2 ) − 2 K ( x 1 , x 2 ) ) α 2 o l d y 2 + y 2 − y 1 + ( ∑ i = 1 N y i α i K ( x i , x 1 ) + b ) − ( ∑ i = 1 N y i α i K ( x i , x 2 ) + b ) ) = ( K ( x 1 , x 1 ) + K ( x 2 , x 2 ) − 2 K ( x 1 , x 2 ) ) α 2 o l d + y 2 ( ( ∑ i = 1 N y i α i K ( x i , x 1 ) + b − y 1 ) − ( ∑ i = 1 N y i α i K ( x i , x 2 ) + b − y 2 ) ) \begin{aligned}(K(x_1,x_1)+K(x_2,x_2)-2K(x_1,x_2))\alpha_2^{new,unc}&=y_2\left((K(x_1,x_1)+K(x_2,x_2)-2K(x_1,x_2))\alpha_2^{old}y_2+y_2-y_1+\left(\sum_{i=1}^Ny_i\alpha_iK(x_i,x_1)+b\right)-\left(\sum_{i=1}^Ny_i\alpha_iK(x_i,x_2)+b\right)\right)\\&=(K(x_1,x_1)+K(x_2,x_2)-2K(x_1,x_2))\alpha_2^{old}+y_2\left(\left(\sum_{i=1}^Ny_i\alpha_iK(x_i,x_1)+b-y_1\right)-\left(\sum_{i=1}^Ny_i\alpha_iK(x_i,x_2)+b-y_2\right)\right)\end{aligned} ( K ( x 1 , x 1 ) + K ( x 2 , x 2 ) − 2 K ( x 1 , x 2 ) ) α 2 n e w , u n c = y 2 ( ( K ( x 1 , x 1 ) + K ( x 2 , x 2 ) − 2 K ( x 1 , x 2 ) ) α 2 o l d y 2 + y 2 − y 1 + ( i = 1 ∑ N y i α i K ( x i , x 1 ) + b ) − ( i = 1 ∑ N y i α i K ( x i , x 2 ) + b ) ) = ( K ( x 1 , x 1 ) + K ( x 2 , x 2 ) − 2 K ( x 1 , x 2 ) ) α 2 o l d + y 2 ( ( i = 1 ∑ N y i α i K ( x i , x 1 ) + b − y 1 ) − ( i = 1 ∑ N y i α i K ( x i , x 2 ) + b − y 2 ) ) α 2 n e w , u n c = α 2 o l d + y 2 ( ( ∑ i = 1 N y i α i K ( x i , x 1 ) + b − y 1 ) − ( ∑ i = 1 N y i α i K ( x i , x 2 ) + b − y 2 ) ) K ( x 1 , x 1 ) + K ( x 2 , x 2 ) − 2 K ( x 1 , x 2 ) \alpha_2^{new,unc}=\alpha_2^{old}+\frac{y_2\left(\left(\sum_{i=1}^Ny_i\alpha_iK(x_i,x_1)+b-y_1\right)-\left(\sum_{i=1}^Ny_i\alpha_iK(x_i,x_2)+b-y_2\right)\right)}{K(x_1,x_1)+K(x_2,x_2)-2K(x_1,x_2)} α 2 n e w , u n c = α 2 o l d + K ( x 1 , x 1 ) + K ( x 2 , x 2 ) − 2 K ( x 1 , x 2 ) y 2 ( ( ∑ i = 1 N y i α i K ( x i , x 1 ) + b − y 1 ) − ( ∑ i = 1 N y i α i K ( x i , x 2 ) + b − y 2 ) ) 0 ≤ α 2 ≤ C 0\le \alpha_2 \le C 0 ≤ α 2 ≤ C α 2 n e w { H , α 2 n e w , u n c > H α 2 n e w , u n c , L ≤ α 2 n e w , u n c ≤ H L , α 2 n e w , u n c < L \alpha_2^{new}\left\{\begin{aligned}&H,&\alpha_2^{new,unc}>H\\&\alpha_2^{new,unc},&L\le \alpha_2^{new,unc}\le H\\&L,&\alpha_2^{new,unc}<L\end{aligned}\right. α 2 n e w ⎩ ⎪ ⎨ ⎪ ⎧ H , α 2 n e w , u n c , L , α 2 n e w , u n c > H L ≤ α 2 n e w , u n c ≤ H α 2 n e w , u n c < L α 2 \alpha_2 α 2 α 1 \alpha_1 α 1
第一个变量称为外层循环，我们选择违反分布条件最严重 的点，也就是说对应的计算结果与分布的要求差距最大的点，具体地，我们首先检查在边界上的支持向量0 < α i < C 0<\alpha_i<C 0 < α i < C α 2 \alpha_2 α 2 α 2 \alpha_2 α 2 ∣ ( ∑ i = 1 N y i α i K ( x i , x 1 ) + b − y 1 ) − ( ∑ i = 1 N y i α i K ( x i , x 2 ) + b − y 2 ) ∣ \left|\left(\sum\limits_{i=1}^Ny_i\alpha_iK(x_i,x_1)+b-y_1\right)-\left(\sum\limits_{i=1}^Ny_i\alpha_iK(x_i,x_2)+b-y_2\right)\right| ∣ ∣ ∣ ∣ ( i = 1 ∑ N y i α i K ( x i , x 1 ) + b − y 1 ) − ( i = 1 ∑ N y i α i K ( x i , x 2 ) + b − y 2 ) ∣ ∣ ∣ ∣ α 2 \alpha_2 α 2 α 1 \alpha_1 α 1
对在间隔边界上的支持向量点有0 < α < C 0<\alpha<C 0 < α < C 0 < α 1 < C 0<\alpha_1<C 0 < α 1 < C b 1 n e w = y 1 − ∑ i = 3 N α i y i K ( x i , x 1 ) − α 1 n e w y 1 K ( x 1 , x 1 ) − α 2 n e w y 2 K ( x 2 , x 1 ) = − ( ∑ i = 1 N y i α i K ( x i , x 1 ) + b − y 1 ) − y 1 K ( x 1 , x 1 ) ( α 1 n e w − α 1 o l d ) − y 2 K ( x 1 , x 2 ) ( α 2 n e w − α 2 o l d ) + b o l d \begin{aligned}b_1^{new}&=y_1-\sum\limits_{i=3}^N\alpha_iy_iK(x_i,x_1)-\alpha_1^{new}y_1K(x_1,x_1)-\alpha_2^{new}y_2K(x_2,x_1)\\&=-\left(\sum\limits_{i=1}^Ny_i\alpha_iK(x_i,x_1)+b-y_1\right)-y_1K(x_1,x_1)(\alpha_1^{new}-\alpha_1^{old})-y_2K(x_1,x_2)(\alpha_2^{new}-\alpha_2^{old})+b^{old}\end{aligned} b 1 n e w = y 1 − i = 3 ∑ N α i y i K ( x i , x 1 ) − α 1 n e w y 1 K ( x 1 , x 1 ) − α 2 n e w y 2 K ( x 2 , x 1 ) = − ( i = 1 ∑ N y i α i K ( x i , x 1 ) + b − y 1 ) − y 1 K ( x 1 , x 1 ) ( α 1 n e w − α 1 o l d ) − y 2 K ( x 1 , x 2 ) ( α 2 n e w − α 2 o l d ) + b o l d b 2 n e w b_2^{new} b 2 n e w 0 < α 1 < C 0<\alpha_1<C 0 < α 1 < C 0 < α 2 < C 0<\alpha_2<C 0 < α 2 < C b 2 n e w = b 1 n e w b_2^{new}=b_1^{new} b 2 n e w = b 1 n e w α 1 , α 2 \alpha_1,\alpha_2 α 1 , α 2 b 2 n e w b_2^{new} b 2 n e w b 1 n e w b_1^{new} b 1 n e w
采用近似的思想，求解二次规划问题，分解成每次优化两个变量的子问题，这些子问题拥有解析解。因此计算会高效。