思想来源于非线性最小二乘问题 . 正如前面所提到的, 信赖域方法与线搜索算法不同之处其一在于, 后者先产生搜索方向p k p_k p k α k \alpha_k α k x k + 1 = x k + α k p k x_{k+1}=x_k+\alpha_kp_k x k + 1 = x k + α k p k 同时选取搜索方向与步长 (或者说, 先划定步长的范围, 再选取搜索方向) . 迭代的方式为x k + 1 = x k + p k x_{k+1}=x_k+p_k x k + 1 = x k + p k p k p_k p k 第二章 . 其求解重心在于子问题min p ∈ R n m k ( p ) = f k + g k T p + 1 2 p T B k p , s . t . ∥ p ∥ ≤ Δ k . \min_{p\in\mathbb{R}^n}m_k(p)=f_k+g_k^Tp+\frac{1}{2}p^TB_kp,\quad\mathrm{s.t.}\Vert p\Vert\le\Delta_k. p ∈ R n min m k ( p ) = f k + g k T p + 2 1 p T B k p , s . t . ∥ p ∥ ≤ Δ k . m k m_k m k f f f
图中, 椭圆虚线为建立的模型m k m_k m k m k m_k m k f f f m k m_k m k 线搜索算法像是活力四射说话斩钉截铁的热血青年, 而信赖域算法则像是运筹帷幄决胜千里的沉稳老者 .关键, 在于求解一系列的子问题 (subproblem) . 若∥ B k − 1 g k ∥ ≤ Δ k \Vert B_k^{-1}g_k\Vert\le\Delta_k ∥ B k − 1 g k ∥ ≤ Δ k m k m_k m k p k p_k p k B k = ∇ 2 f k B_k=\nabla^2 f_k B k = ∇ 2 f k p k p_k p k
我们在每步迭代后都要对信赖域进行调整: 如果m k m_k m k f f f x k + 1 = x k x_{k+1}=x_k x k + 1 = x k . 因此我们需要用一个参数来度量m k m_k m k x k x_k x k f f f p k p_k p k ρ k = f ( x k ) − f ( x k + p k ) m k ( 0 ) − m k ( p k ) ; \rho_k=\frac{f(x_k)-f(x_k+p_k)}{m_k(0)-m_k(p_k)}; ρ k = m k ( 0 ) − m k ( p k ) f ( x k ) − f ( x k + p k ) ; ρ k \rho_k ρ k m k m_k m k x k x_k x k f f f ρ k \rho_k ρ k ρ k \rho_k ρ k ρ k \rho_k ρ k ρ k \rho_k ρ k x k + 1 = x k x_{k+1}=x_k x k + 1 = x k
算法1 (信赖域算法)Δ ^ > 0 , Δ 0 ∈ ( 0 , Δ ^ ) , η ∈ [ 0 , 1 4 ] \hat{\Delta}>0,\Delta_0\in(0,\hat{\Delta}),\eta\in\left[0,\frac{1}{4}\right] Δ ^ > 0 , Δ 0 ∈ ( 0 , Δ ^ ) , η ∈ [ 0 , 4 1 ] k = 0 , 1 , 2 , … , k=0,1,2,\ldots, k = 0 , 1 , 2 , … , 求 解 子 问 题 得 p k \quad\quad求解子问题得p_k 求 解 子 问 题 得 p k 计 算 ρ k \quad\quad计算\rho_k 计 算 ρ k \quad\quad ρ k < 1 4 \rho_k<\frac{1}{4} ρ k < 4 1 Δ k + 1 = 1 4 Δ k \quad\quad\quad\quad\Delta_{k+1}=\frac{1}{4}\Delta_k Δ k + 1 = 4 1 Δ k \quad\quad \quad\quad\quad\quad ρ k > 3 4 \rho_k>\frac{3}{4} ρ k > 4 3 ∥ p k ∥ = Δ k \Vert p_k\Vert=\Delta_k ∥ p k ∥ = Δ k Δ k + 1 = min ( 2 Δ k , Δ ^ ) \quad\quad\quad\quad\quad\quad\Delta_{k+1}=\min(2\Delta_k,\hat{\Delta}) Δ k + 1 = min ( 2 Δ k , Δ ^ ) \quad\quad\quad\quad Δ k + 1 = Δ k \quad\quad\quad\quad\quad\quad\Delta_{k+1}=\Delta_k Δ k + 1 = Δ k \quad\quad ρ k > η \rho_k>\eta ρ k > η x k + 1 = x k + p k \quad\quad\quad\quad x_{k+1}=x_k+p_k x k + 1 = x k + p k \quad\quad x k + 1 = x k \quad\quad\quad\quad x_{k+1}=x_k x k + 1 = x k
其中Δ ^ \hat{\Delta} Δ ^ ∥ p k ∥ \Vert p_k\Vert ∥ p k ∥ Δ k \Delta_k Δ k
信赖域的半径不会变的过大 (有上界Δ ^ \hat{\Delta} Δ ^
信赖域方法较于线搜索有一定的优势: 后者可能求到极大点和鞍点, 而前者求到的一定是极小点.
我们顺便将求解子问题的关键定理给出, 后面将给出证明.
定理1 向量p ∗ p^* p ∗ min p ∈ R n m ( p ) = f + g T p + 1 2 p T B p , s . t . ∥ p ∥ ≤ Δ \min_{p\in\mathbb{R}^n}m(p)=f+g^Tp+\frac{1}{2}p^TBp,\quad\mathrm{s.t.}\Vert p\Vert\le\Delta p ∈ R n min m ( p ) = f + g T p + 2 1 p T B p , s . t . ∥ p ∥ ≤ Δ p ∗ p^* p ∗ λ ≥ 0 \lambda\ge0 λ ≥ 0 ( B + λ I ) p ∗ = − g , (B+\lambda I)p^*=-g, ( B + λ I ) p ∗ = − g , λ ( Δ − ∥ p ∗ ∥ ) = 0 , \lambda(\Delta-\Vert p^*\Vert)=0, λ ( Δ − ∥ p ∗ ∥ ) = 0 , ( B + λ I ) 半 正 定 . (B+\lambda I)半正定. ( B + λ I ) 半 正 定 . B k B_k B k
下面为本章架构: 第一节我们提出Cauchy点的概念, 并给出两种基于它的近似求解子问题的策略; 第二节证明全局收敛性; 第三节给出求子问题的迭代方法; 第四节讨论当B k B_k B k ∇ 2 f k \nabla^2f_k ∇ 2 f k
Cauchy点在证明信赖域算法全局收敛性上的作用, 就如同Wolfe条件在线搜索算法上的作用. 信赖域算法无需精确搜索, 只需要每步迭代满足: 1) 迭代步在信赖域内; 2) 迭代步带来充分下降, 即可得到全局收敛性. 后者便是由Cauchy点保证. Cauchy点的计算基于对函数f f f , 事实上就是对一阶模型的极小化:p k s = arg min p ∈ R n f k + g k T p , s . t . ∥ p ∥ ≤ Δ k ; p_k^s=\arg\min_{p\in\mathbb{R}^n}f_k+g_k^Tp,\quad\mathrm{s.t.}\Vert p\Vert\le\Delta_k; p k s = arg p ∈ R n min f k + g k T p , s . t . ∥ p ∥ ≤ Δ k ; p k s p_k^s p k s m k m_k m k τ k \tau_k τ k τ k = arg min τ ≥ 0 m k ( τ p k s ) , s . t . ∥ τ p k s ∥ ≤ Δ k ; \tau_k=\arg\min_{\tau\ge0}m_k(\tau p_k^s),\quad\mathrm{s.t.}\Vert\tau p_k^s\Vert\le\Delta_k; τ k = arg τ ≥ 0 min m k ( τ p k s ) , s . t . ∥ τ p k s ∥ ≤ Δ k ; p k C = τ k p k s p_k^C=\tau_kp_k^s p k C = τ k p k s p k s = − Δ k ∥ g k ∥ g k . p_k^s=-\frac{\Delta_k}{\Vert g_k\Vert}g_k. p k s = − ∥ g k ∥ Δ k g k . τ k \tau_k τ k g k T B k g k ≤ 0 g_k^TB_kg_k\le0 g k T B k g k ≤ 0 g k T B k g k > 0 g_k^TB_kg_k>0 g k T B k g k > 0
第一种情形, 只要g k ≠ 0 g_k\ne0 g k ̸ = 0 m k ( τ p s s ) m_k(\tau p_s^s) m k ( τ p s s ) τ \tau τ τ k = 1 \tau_k=1 τ k = 1
第二种情形, m k ( τ p k s ) m_k(\tau p_k^s) m k ( τ p k s ) τ \tau τ ∥ g k ∥ 3 / ( Δ k g k T B k g k ) \Vert g_k\Vert^3/(\Delta_kg_k^TB_kg_k) ∥ g k ∥ 3 / ( Δ k g k T B k g k )
从而有p k C = − τ k Δ k ∥ g k ∥ g k , p_k^C=-\tau_k\frac{\Delta_k}{\Vert g_k\Vert}g_k, p k C = − τ k ∥ g k ∥ Δ k g k , τ k = { 1 若 g k T B k g k ≤ 0 ; min ( ∥ g k ∥ 3 / ( Δ k g k T B k g k ) , 1 ) o t h e r w i s e . \tau_k=\left\{\begin{array}{ll}1 & 若g_k^TB_kg_k\le0;\\\min(\Vert g_k\Vert^3/(\Delta_kg_k^TB_kg_k),1) & otherwise.\end{array}\right. τ k = { 1 min ( ∥ g k ∥ 3 / ( Δ k g k T B k g k ) , 1 ) 若 g k T B k g k ≤ 0 ; o t h e r w i s e . Cauchy点的选取可以看做是最速下降法的一种特殊情形 : 同样是沿着负梯度方向搜索, 只是选取了特殊的步长 . Cauchy点的计算耗费很小, 至多涉及矩阵-向量乘积 . 此外, 更重要的是, Cauchy点还是判定全局收敛性的重要手段 : 只要p k p_k p k p k C p_k^C p k C 全局收敛的 . 不过, 由第三章 中关于最速下降法的讨论, 即使使用精确线搜索, 其表现也可能不尽人意. 因此我们需要基于Cauchy点设计更好的算法.Cauchy点几乎没有用到f f f , 因此设计的新算法应当将此纳入考虑. 同时, 新算法还应当具有一定的相容性 : 当B k B_k B k p k B = − B k − 1 g k p_k^B=-B_k^{-1}g_k p k B = − B k − 1 g k ∥ p k B ∥ ≤ Δ k \Vert p_k^B\Vert\le\Delta_k ∥ p k B ∥ ≤ Δ k k k k p ∗ ( Δ ) p^*(\Delta) p ∗ ( Δ ) Δ \Delta Δ
折线算法可用于B B B
当∥ p B ∥ ≤ Δ \Vert p^B\Vert\le\Delta ∥ p B ∥ ≤ Δ p ∗ ( Δ ) = p B p^*(\Delta)=p^B p ∗ ( Δ ) = p B
当∥ p B ∥ ≫ Δ \Vert p^B\Vert\gg\Delta ∥ p B ∥ ≫ Δ 可认为 m m m p ∗ ( Δ ) ≈ − Δ g ∥ g ∥ . p^*(\Delta)\approx-\Delta\frac{g}{\Vert g\Vert}. p ∗ ( Δ ) ≈ − Δ ∥ g ∥ g . Δ \Delta Δ
折线算法是基于以上两点的近似算法, 它用两段折线近似p ∗ ( Δ ) p^*(\Delta) p ∗ ( Δ ) . 第一条轨迹来源于上面的第2条, 即使用最速下降方向p U = − g T g g T B g g p^U=-\frac{g^Tg}{g^TBg}g p U = − g T B g g T g g p U p^U p U p B p^B p B
我们用p ~ ( τ ) , τ ∈ [ 0 , 2 ] \tilde{p}(\tau), \tau\in[0,2] p ~ ( τ ) , τ ∈ [ 0 , 2 ] p ~ ( τ ) = { τ p U , 0 ≤ τ ≤ 1 , p U + ( τ − 1 ) ( p B − p U ) , 1 ≤ τ ≤ 2. \tilde{p}(\tau)=\left\{\begin{array}{ll}\tau p^U, & 0\le\tau\le1,\\p^U+(\tau-1)(p^B-p^U), & 1\le\tau\le2.\end{array}\right. p ~ ( τ ) = { τ p U , p U + ( τ − 1 ) ( p B − p U ) , 0 ≤ τ ≤ 1 , 1 ≤ τ ≤ 2 . p p p p B p^B p B p B p^B p B
定理2 设B B B
∥ p ~ ( τ ) ∥ \Vert \tilde{p}(\tau)\Vert ∥ p ~ ( τ ) ∥ τ \tau τ
m ( p ~ ( τ ) ) m(\tilde{p}(\tau)) m ( p ~ ( τ ) ) τ \tau τ
证明: 易知1, 2对τ ∈ [ 0 , 1 ] \tau\in[0,1] τ ∈ [ 0 , 1 ] τ ∈ [ 1 , 2 ] \tau\in[1,2] τ ∈ [ 1 , 2 ] h ( α ) = 1 2 ∥ p ~ ( 1 + α ) ∥ 2 = 1 2 ∥ p U + α ( p B − p U ) ∥ 2 = 1 2 ∥ p U ∥ 2 + α ( p U ) T ( p B − p U ) + 1 2 α 2 ∥ p B − p U ∥ 2 . \begin{aligned}h(\alpha)&=\frac{1}{2}\Vert\tilde{p}(1+\alpha)\Vert^2\\&=\frac{1}{2}\Vert p^U+\alpha(p^B-p^U)\Vert^2\\&=\frac{1}{2}\Vert p^U\Vert^2+\alpha(p^U)^T(p^B-p^U)+\frac{1}{2}\alpha^2\Vert p^B-p^U\Vert^2.\end{aligned} h ( α ) = 2 1 ∥ p ~ ( 1 + α ) ∥ 2 = 2 1 ∥ p U + α ( p B − p U ) ∥ 2 = 2 1 ∥ p U ∥ 2 + α ( p U ) T ( p B − p U ) + 2 1 α 2 ∥ p B − p U ∥ 2 . h ′ ( α ) ≥ 0 , α ∈ ( 0 , 1 ) h'(\alpha)\ge0,\alpha\in(0,1) h ′ ( α ) ≥ 0 , α ∈ ( 0 , 1 ) h ′ ( α ) = − ( p U ) T ( p U − p B ) + α ∥ p U − p B ∥ 2 ≥ − ( p U ) T ( p U − p B ) = g T g g T B g g T ( − g T g g T B g g + B − 1 g ) = g T g g T B − 1 g g T B g [ 1 − ( g T g ) 2 ( g T B g ) ( g T B − 1 g ) ] . \begin{aligned}h'(\alpha)&=-(p^U)^T(p^U-p^B)+\alpha\Vert p^U-p^B\Vert^2\\&\ge-(p^U)^T(p^U-p^B)\\&=\frac{g^Tg}{g^TBg}g^T\left(-\frac{g^Tg}{g^TBg}g+B^{-1}g\right)\\&=g^Tg\frac{g^TB^{-1}g}{g^TBg}\left[1-\frac{(g^Tg)^2}{(g^TBg)(g^TB^{-1}g)}\right].\end{aligned} h ′ ( α ) = − ( p U ) T ( p U − p B ) + α ∥ p U − p B ∥ 2 ≥ − ( p U ) T ( p U − p B ) = g T B g g T g g T ( − g T B g g T g g + B − 1 g ) = g T g g T B g g T B − 1 g [ 1 − ( g T B g ) ( g T B − 1 g ) ( g T g ) 2 ] . ( g T g ) 2 = ( g T B 1 2 B − 1 2 g ) 2 ≤ ∥ B 1 2 g ∥ 2 ∥ B − 1 2 g ∥ 2 = ( g T B g ) ( g T B − 1 g ) , \begin{aligned}(g^Tg)^2&=\left(g^TB^{\frac{1}{2}}B^{-\frac{1}{2}}g\right)^2\\&\le\left\Vert B^{\frac{1}{2}}g\right\Vert^2\left\Vert B^{-\frac{1}{2}}g\right\Vert^2\\&=(g^TBg)(g^TB^{-1}g),\end{aligned} ( g T g ) 2 = ( g T B 2 1 B − 2 1 g ) 2 ≤ ∥ ∥ ∥ B 2 1 g ∥ ∥ ∥ 2 ∥ ∥ ∥ B − 2 1 g ∥ ∥ ∥ 2 = ( g T B g ) ( g T B − 1 g ) , h ^ ( α ) = m ( p ~ ( 1 + α ) ) \hat{h}(\alpha)=m(\tilde{p}(1+\alpha)) h ^ ( α ) = m ( p ~ ( 1 + α ) ) h ^ ′ ( α ) ≤ 0 , α ∈ ( 0 , 1 ) \hat{h}'(\alpha)\le0,\alpha\in(0,1) h ^ ′ ( α ) ≤ 0 , α ∈ ( 0 , 1 ) h ^ ′ ( α ) = ( p B − p U ) T ( g + B p U ) + α ( p B − p U ) T B ( p B − p U ) ≤ ( p B − p U ) T ( g + B p U + B ( p B − p U ) ) = ( p B − p U ) T ( g + B p B ) = 0. \begin{aligned}\hat{h}'(\alpha)&=(p^B-p^U)^T(g+Bp^U)+\alpha(p^B-p^U)^TB(p^B-p^U)\\&\le(p^B-p^U)^T(g+Bp^U+B(p^B-p^U))\\&=(p^B-p^U)^T(g+Bp^B)=0.\end{aligned} h ^ ′ ( α ) = ( p B − p U ) T ( g + B p U ) + α ( p B − p U ) T B ( p B − p U ) ≤ ( p B − p U ) T ( g + B p U + B ( p B − p U ) ) = ( p B − p U ) T ( g + B p B ) = 0 .
此引理说明, 折线轨迹与信赖域边界至多只会交于一点. 当有交点产生时, 若∥ p U ∥ ≤ Δ \Vert p^U\Vert\le\Delta ∥ p U ∥ ≤ Δ ∥ p U + ( τ − 1 ) ( p B − p U ) ∥ 2 = Δ 2 . \Vert p^U+(\tau-1)(p^B-p^U)\Vert^2=\Delta^2. ∥ p U + ( τ − 1 ) ( p B − p U ) ∥ 2 = Δ 2 .
下面说几点注记:
若将d ( Δ ) d(\Delta) d ( Δ ) d ( Δ ) d(\Delta) d ( Δ )
当Hessian矩阵∇ 2 f ( x k ) \nabla^2f(x_k) ∇ 2 f ( x k ) B = ∇ 2 f ( x k ) B=\nabla^2f(x_k) B = ∇ 2 f ( x k ) B B B 第三章 3.2.2小节的讨论) . 在接近满足二阶充分条件的解时, p B p^B p B . 这点在后面会给出证明. 不过, 在牛顿折线算法中取修正的Hessian可能会以任一种形式影响∇ 2 f ( x k ) \nabla^2f(x_k) ∇ 2 f ( x k ) B k = ∇ 2 f ( x k ) B_k=\nabla^2f(x_k) B k = ∇ 2 f ( x k ) ( ∇ 2 f ( x k ) + λ I ) − 1 g k (\nabla^2f(x_k)+\lambda I)^{-1}g_k ( ∇ 2 f ( x k ) + λ I ) − 1 g k λ \lambda λ ∇ 2 f ( x k ) + λ I \nabla^2f(x_k)+\lambda I ∇ 2 f ( x k ) + λ I Δ k \Delta_k Δ k ∇ 2 f ( x k ) \nabla^2f(x_k) ∇ 2 f ( x k )
折线算法也可以推广到B B B p B p^B p B m m m
当B B B p B p^B p B p U p^U p U min p m ( p ) = f + g T p + 1 2 p T B p , s . t . ∥ p ∥ ≤ Δ , p ∈ s p a n { g , B − 1 g } . \min_pm(p)=f+g^Tp+\frac{1}{2}p^TBp,\quad\mathrm{s.t.} \Vert p\Vert\le\Delta,\quad p\in\mathrm{span}\{g,B^{-1}g\}. p min m ( p ) = f + g T p + 2 1 p T B p , s . t . ∥ p ∥ ≤ Δ , p ∈ s p a n { g , B − 1 g } . p p p p C p^C p C B B B B B B s p a n { g , ( B + α I ) − 1 g } , α ∈ ( − λ 1 , − 2 λ 1 ] , \mathrm{span}\{g,(B+\alpha I)^{-1}g\},\quad\alpha\in(-\lambda_1,-2\lambda_1], s p a n { g , ( B + α I ) − 1 g } , α ∈ ( − λ 1 , − 2 λ 1 ] , λ 1 \lambda_1 λ 1 B B B α \alpha α B + α I B+\alpha I B + α I α \alpha α ∥ ( B + α I ) − 1 g ∥ ≤ Δ \Vert (B+\alpha I)^{-1}g\Vert\le\Delta ∥ ( B + α I ) − 1 g ∥ ≤ Δ p = − ( B + α I ) − 1 g + v , p=-(B+\alpha I)^{-1}g+v, p = − ( B + α I ) − 1 g + v , v v v v T ( B + α I ) − 1 g ≤ 0 v^T(B+\alpha I)^{-1}g\le0 v T ( B + α I ) − 1 g ≤ 0 ∥ p ∥ ≥ ∥ ( B + α I ) − 1 g ∥ \Vert p\Vert\ge\Vert(B+\alpha I)^{-1}g\Vert ∥ p ∥ ≥ ∥ ( B + α I ) − 1 g ∥ B B B p = p C p=p^C p = p C 计算量主要在B B B B + α I B+\alpha I B + α I .
我们在之前曾多次强调, 只要迭代步带来的下降不少于Cauchy点, 则算法的全局收敛性将在一定程度上得到保证. 与线搜索Wolfe条件中的充分下降条件类似, 我们将证明折线方法、二维子空间最小化算法产生的p k p_k p k m k ( 0 ) − m k ( p k ) ≥ c 1 ∥ g k ∥ min ( Δ k , ∥ g k ∥ ∥ B k ∥ ) , c 1 ∈ ( 0 , 1 ] . m_k(0)-m_k(p_k)\ge c_1\Vert g_k\Vert\min\left(\Delta_k,\frac{\Vert g_k\Vert}{\Vert B_k\Vert}\right),\quad c_1\in(0,1]. m k ( 0 ) − m k ( p k ) ≥ c 1 ∥ g k ∥ min ( Δ k , ∥ B k ∥ ∥ g k ∥ ) , c 1 ∈ ( 0 , 1 ] . Δ k \Delta_k Δ k p k C p^C_k p k C c 1 = 1 2 c_1=\frac{1}{2} c 1 = 2 1
定理3 Cauchy点p k C p_k^C p k C m k ( 0 ) − m k ( p k C ) ≥ 1 2 ∥ g k ∥ min ( Δ k , ∥ g k ∥ ∥ B k ∥ ) . m_k(0)-m_k(p_k^C)\ge\frac{1}{2}\Vert g_k\Vert\min\left(\Delta_k,\frac{\Vert g_k\Vert}{\Vert B_k\Vert}\right). m k ( 0 ) − m k ( p k C ) ≥ 2 1 ∥ g k ∥ min ( Δ k , ∥ B k ∥ ∥ g k ∥ ) . k k k g T B g ≤ 0 g^TBg\le0 g T B g ≤ 0 p C = − Δ g ∥ g ∥ p^C=-\Delta\frac{g}{\Vert g\Vert} p C = − Δ ∥ g ∥ g m ( p C ) − m ( 0 ) = m ( − Δ g / ∥ g ∥ ) − f = − Δ ∥ g ∥ ∥ g ∥ 2 + 1 2 Δ 2 ∥ g ∥ 2 g T B g ≤ − Δ ∥ g ∥ ≤ − ∥ g ∥ min ( Δ , ∥ g ∥ ∥ B ∥ ) . \begin{aligned}m(p^C)-m(0)&=m(-\Delta g/\Vert g\Vert)-f\\&=-\frac{\Delta}{\Vert g\Vert}\Vert g\Vert^2+\frac{1}{2}\frac{\Delta^2}{\Vert g\Vert^2}g^TBg\\&\le-\Delta\Vert g\Vert\\&\le-\Vert g\Vert\min\left(\Delta,\frac{\Vert g\Vert}{\Vert B\Vert}\right).\end{aligned} m ( p C ) − m ( 0 ) = m ( − Δ g / ∥ g ∥ ) − f = − ∥ g ∥ Δ ∥ g ∥ 2 + 2 1 ∥ g ∥ 2 Δ 2 g T B g ≤ − Δ ∥ g ∥ ≤ − ∥ g ∥ min ( Δ , ∥ B ∥ ∥ g ∥ ) . g T B g > 0 g^TBg>0 g T B g > 0 τ \tau τ ∥ g ∥ 3 Δ g T B g ≤ 1 \frac{\Vert g\Vert^3}{\Delta g^TBg}\le1 Δ g T B g ∥ g ∥ 3 ≤ 1 τ = ∥ g ∥ 3 Δ g T B g \tau=\frac{\Vert g\Vert^3}{\Delta g^TBg} τ = Δ g T B g ∥ g ∥ 3 m ( p C ) − m ( 0 ) = − ∥ g ∥ 4 g T B g + 1 2 g T B g ∥ g ∥ 4 ( g T B g ) 2 = − 1 2 ∥ g ∥ 4 g T B g ≤ − 1 2 ∥ g ∥ 4 ∥ B ∥ ∥ g ∥ 2 = − 1 2 ∥ g ∥ 2 ∥ B ∥ ≤ − 1 2 ∥ g ∥ min ( Δ , ∥ g ∥ ∥ B ∥ ) . \begin{aligned}m(p^C)-m(0)&=-\frac{\Vert g\Vert^4}{g^TBg}+\frac{1}{2}g^TBg\frac{\Vert g\Vert^4}{(g^TBg)^2}\\&=-\frac{1}{2}\frac{\Vert g\Vert^4}{g^TBg}\\&\le-\frac{1}{2}\frac{\Vert g\Vert^4}{\Vert B\Vert\Vert g\Vert^2}\\&=-\frac{1}{2}\frac{\Vert g\Vert^2}{\Vert B\Vert}\\&\le-\frac{1}{2}\Vert g\Vert\min\left(\Delta,\frac{\Vert g\Vert}{\Vert B\Vert}\right).\end{aligned} m ( p C ) − m ( 0 ) = − g T B g ∥ g ∥ 4 + 2 1 g T B g ( g T B g ) 2 ∥ g ∥ 4 = − 2 1 g T B g ∥ g ∥ 4 ≤ − 2 1 ∥ B ∥ ∥ g ∥ 2 ∥ g ∥ 4 = − 2 1 ∥ B ∥ ∥ g ∥ 2 ≤ − 2 1 ∥ g ∥ min ( Δ , ∥ B ∥ ∥ g ∥ ) . g T B g < ∥ g ∥ 3 Δ g^TBg<\frac{\Vert g\Vert^3}{\Delta} g T B g < Δ ∥ g ∥ 3 τ = 1 \tau = 1 τ = 1 m ( p C ) − m ( 0 ) = − Δ ∥ g ∥ ∥ g ∥ 2 + 1 2 Δ 2 ∥ g ∥ 2 g T B g ≤ − Δ ∥ g ∥ + 1 2 Δ 2 ∥ g ∥ 2 ∥ g ∥ 3 Δ = − 1 2 Δ ∥ g ∥ ≤ − 1 2 ∥ g ∥ min ( Δ , ∥ g ∥ ∥ B ∥ ) . \begin{aligned}m(p^C)-m(0)&=-\frac{\Delta}{\Vert g\Vert}\Vert g\Vert^2+\frac{1}{2}\frac{\Delta^2}{\Vert g\Vert^2}g^TBg\\&\le-\Delta\Vert g\Vert+\frac{1}{2}\frac{\Delta^2}{\Vert g\Vert^2}\frac{\Vert g\Vert^3}{\Delta}\\&=-\frac{1}{2}\Delta\Vert g\Vert\\&\le-\frac{1}{2}\Vert g\Vert\min\left(\Delta,\frac{\Vert g\Vert}{\Vert B\Vert}\right).\end{aligned} m ( p C ) − m ( 0 ) = − ∥ g ∥ Δ ∥ g ∥ 2 + 2 1 ∥ g ∥ 2 Δ 2 g T B g ≤ − Δ ∥ g ∥ + 2 1 ∥ g ∥ 2 Δ 2 Δ ∥ g ∥ 3 = − 2 1 Δ ∥ g ∥ ≤ − 2 1 ∥ g ∥ min ( Δ , ∥ B ∥ ∥ g ∥ ) .
为满足充分下降, p k p_k p k
定理4 设p k : ∥ p k ∥ ≤ Δ k p_k: \Vert p_k\Vert\le\Delta_k p k : ∥ p k ∥ ≤ Δ k m k ( 0 ) − m k ( p k ) ≥ c 2 ( m k ( 0 ) − m k ( p k C ) ) m_k(0)-m_k(p_k)\ge c_2(m_k(0)-m_k(p^C_k)) m k ( 0 ) − m k ( p k ) ≥ c 2 ( m k ( 0 ) − m k ( p k C ) ) p k p_k p k c 1 = c 2 / 2 c_1=c_2/2 c 1 = c 2 / 2 p k p_k p k p k ∗ p_k^* p k ∗ c 1 = 1 2 c_1=\frac{1}{2} c 1 = 2 1 ∥ p k ∥ ≤ Δ k \Vert p_k\Vert\le\Delta_k ∥ p k ∥ ≤ Δ k
注意折线算法和二维子空间最小化算法都满足c 1 = 1 / 2 c_1=1/2 c 1 = 1 / 2 p k p_k p k m k ( p k ) ≤ m k ( p k C ) m_k(p_k)\le m_k(p_k^C) m k ( p k ) ≤ m k ( p k C )
信赖域算法的全局收敛性依赖于算法1中η \eta η 若η = 0 \eta=0 η = 0 x k + 1 = x k + p k x_{k+1}=x_k+p_k x k + 1 = x k + p k η > 0 \eta>0 η > 0 .
Hessian近似B k B_k B k
f f f S = { x ∣ f ( x ) ≤ f ( x 0 ) } S=\{x|f(x)\le f(x_0)\} S = { x ∣ f ( x ) ≤ f ( x 0 ) } S S S S ( R 0 ) = { x ∣ ∥ x − y ∥ < R 0 , y ∈ S } , S(R_0)=\{x|\Vert x-y\Vert<R_0,y\in S\}, S ( R 0 ) = { x ∣ ∥ x − y ∥ < R 0 , y ∈ S } , R 0 R_0 R 0
为增大定理的适用范围, 允许p k p_k p k γ ≥ 1 \gamma\ge1 γ ≥ 1 ∥ p k ∥ ≤ γ Δ k . \Vert p_k\Vert\le\gamma\Delta_k. ∥ p k ∥ ≤ γ Δ k .
下面我们分别针对η = 0 \eta=0 η = 0 η > 0 \eta>0 η > 0
定理5 设算法1中η = 0 \eta=0 η = 0 β \beta β ∥ B k ∥ ≤ β , ∀ k \Vert B_k\Vert\le\beta,\forall k ∥ B k ∥ ≤ β , ∀ k f f f S S S S ( R 0 ) S(R_0) S ( R 0 ) p k p_k p k m k ( 0 ) − m k ( p k ) ≥ c 1 ∥ g k ∥ min ( Δ k , ∥ g k ∥ ∥ B k ∥ ) , c 1 ∈ ( 0 , 1 ] m_k(0)-m_k(p_k)\ge c_1\Vert g_k\Vert\min\left(\Delta_k,\frac{\Vert g_k\Vert}{\Vert B_k\Vert}\right), \quad c_1\in(0,1] m k ( 0 ) − m k ( p k ) ≥ c 1 ∥ g k ∥ min ( Δ k , ∥ B k ∥ ∥ g k ∥ ) , c 1 ∈ ( 0 , 1 ] ∥ p k ∥ ≤ γ Δ k , γ ≥ 1. \Vert p_k\Vert\le\gamma\Delta_k,\quad \gamma\ge1. ∥ p k ∥ ≤ γ Δ k , γ ≥ 1 . lim inf k → ∞ ∥ g k ∥ = 0. \liminf_{k\to\infty}\Vert g_k\Vert=0. k → ∞ l i m i n f ∥ g k ∥ = 0 . ρ k \rho_k ρ k ∣ ρ k − 1 ∣ = ∣ ( f ( x k ) − f ( x k + p k ) ) − ( m k ( 0 ) − m k ( p k ) ) m k ( 0 ) − m k ( p k ) ∣ = ∣ m k ( p k ) − f ( x k + p k ) m k ( 0 ) − m k ( p k ) ∣ . \begin{aligned}|\rho_k-1|&=\left|\frac{(f(x_k)-f(x_k+p_k))-(m_k(0)-m_k(p_k))}{m_k(0)-m_k(p_k)}\right|\\&=\left|\frac{m_k(p_k)-f(x_k+p_k)}{m_k(0)-m_k(p_k)}\right|.\end{aligned} ∣ ρ k − 1 ∣ = ∣ ∣ ∣ ∣ m k ( 0 ) − m k ( p k ) ( f ( x k ) − f ( x k + p k ) ) − ( m k ( 0 ) − m k ( p k ) ) ∣ ∣ ∣ ∣ = ∣ ∣ ∣ ∣ m k ( 0 ) − m k ( p k ) m k ( p k ) − f ( x k + p k ) ∣ ∣ ∣ ∣ . f ( x k + p k ) = f ( x k ) + g ( x k ) T p k + ∫ 0 1 [ g ( x k + t p k ) − g ( x k ) ] T p k d t , f(x_k+p_k)=f(x_k)+g(x_k)^Tp_k+\int_0^1[g(x_k+tp_k)-g(x_k)]^Tp_k\,\mathrm{d}t, f ( x k + p k ) = f ( x k ) + g ( x k ) T p k + ∫ 0 1 [ g ( x k + t p k ) − g ( x k ) ] T p k d t , ∣ m k ( p k ) − f ( x k + p k ) ∣ = ∣ 1 2 p k T B k p k − ∫ 0 1 [ g ( x k + t p k ) − g ( x k ) ] T p k d t ∣ ≤ ( β / 2 ) ∥ p k ∥ 2 + β 1 ∥ p k ∥ 2 , \begin{aligned}|m_k(p_k)-f(x_k+p_k)|&=\left|\frac{1}{2}p_k^TB_kp_k-\int_0^1[g(x_k+tp_k)-g(x_k)]^Tp_k\,\mathrm{d}t\right|\\&\le(\beta/2)\Vert p_k\Vert^2+\beta_1\Vert p_k\Vert^2,\end{aligned} ∣ m k ( p k ) − f ( x k + p k ) ∣ = ∣ ∣ ∣ ∣ 2 1 p k T B k p k − ∫ 0 1 [ g ( x k + t p k ) − g ( x k ) ] T p k d t ∣ ∣ ∣ ∣ ≤ ( β / 2 ) ∥ p k ∥ 2 + β 1 ∥ p k ∥ 2 , β 1 \beta_1 β 1 g g g ∥ p k ∥ ≤ R 0 \Vert p_k\Vert\le R_0 ∥ p k ∥ ≤ R 0 x k x_k x k x k + t p k x_k+tp_k x k + t p k S ( R 0 ) S(R_0) S ( R 0 ) ϵ > 0 , K \epsilon>0, K ϵ > 0 , K ∥ g k ∥ ≥ ϵ , k ≥ K . \Vert g_k\Vert\ge\epsilon,\quad k\ge K. ∥ g k ∥ ≥ ϵ , k ≥ K . k ≥ K k\ge K k ≥ K m k ( 0 ) − m k ( p k ) ≥ c 1 ∥ g k ∥ min ( Δ k , ∥ g k ∥ ∥ B k ∥ ) ≥ c 1 ϵ min ( Δ k , ϵ β ) . m_k(0)-m_k(p_k)\ge c_1\Vert g_k\Vert\min\left(\Delta_k,\frac{\Vert g_k\Vert}{\Vert B_k\Vert}\right)\ge c_1\epsilon\min\left(\Delta_k,\frac{\epsilon}{\beta}\right). m k ( 0 ) − m k ( p k ) ≥ c 1 ∥ g k ∥ min ( Δ k , ∥ B k ∥ ∥ g k ∥ ) ≥ c 1 ϵ min ( Δ k , β ϵ ) . ∣ ρ k − 1 ∣ ≤ γ 2 Δ k 2 ( β / 2 + β 1 ) c 1 ϵ min ( Δ k , ϵ / β ) . |\rho_k-1|\le\frac{\gamma^2\Delta_k^2(\beta/2+\beta_1)}{c_1\epsilon\min(\Delta_k,\epsilon/\beta)}. ∣ ρ k − 1 ∣ ≤ c 1 ϵ min ( Δ k , ϵ / β ) γ 2 Δ k 2 ( β / 2 + β 1 ) . Δ k \Delta_k Δ k Δ k ≤ Δ ˉ \Delta_k\le\bar{\Delta} Δ k ≤ Δ ˉ Δ ˉ \bar{\Delta} Δ ˉ Δ ˉ = min ( 1 2 c 1 ϵ γ 2 ( β / 2 + β 1 ) , R 0 γ ) . \bar{\Delta}=\min\left(\frac{1}{2}\frac{c_1\epsilon}{\gamma^2(\beta/2+\beta_1)},\frac{R_0}{\gamma}\right). Δ ˉ = min ( 2 1 γ 2 ( β / 2 + β 1 ) c 1 ϵ , γ R 0 ) . R 0 / γ R_0/\gamma R 0 / γ ∥ p k ∥ ≤ γ Δ k ≤ γ Δ ˉ ≤ R 0 \Vert p_k\Vert\le\gamma\Delta_k\le\gamma\bar{\Delta}\le R_0 ∥ p k ∥ ≤ γ Δ k ≤ γ Δ ˉ ≤ R 0 c 1 ≤ 1 , γ ≥ 1 c_1\le1,\gamma\ge1 c 1 ≤ 1 , γ ≥ 1 Δ ˉ ≤ 1 2 c 1 ϵ γ 2 ( β / 2 + β 1 ) ≤ ϵ β \bar{\Delta}\le\frac{1}{2}\frac{c_1\epsilon}{\gamma^2(\beta/2+\beta_1)}\le\frac{\epsilon}{\beta} Δ ˉ ≤ 2 1 γ 2 ( β / 2 + β 1 ) c 1 ϵ ≤ β ϵ Δ k ≤ Δ ˉ \Delta_k\le\bar{\Delta} Δ k ≤ Δ ˉ min ( Δ k , ϵ / β ) = Δ k \min(\Delta_k,\epsilon/\beta)=\Delta_k min ( Δ k , ϵ / β ) = Δ k ∣ ρ k − 1 ∣ ≤ γ 2 Δ k 2 ( β / 2 + β 1 ) c 1 ϵ Δ k = γ 2 Δ k ( β / 2 + β 1 ) c 1 ϵ ≤ γ 2 Δ ˉ ( β / 2 + β 1 ) c 1 ϵ ≤ 1 2 . |\rho_k-1|\le\frac{\gamma^2\Delta_k^2(\beta/2+\beta_1)}{c_1\epsilon\Delta_k}=\frac{\gamma^2\Delta_k(\beta/2+\beta_1)}{c_1\epsilon}\le\frac{\gamma^2\bar{\Delta}(\beta/2+\beta_1)}{c_1\epsilon}\le\frac{1}{2}. ∣ ρ k − 1 ∣ ≤ c 1 ϵ Δ k γ 2 Δ k 2 ( β / 2 + β 1 ) = c 1 ϵ γ 2 Δ k ( β / 2 + β 1 ) ≤ c 1 ϵ γ 2 Δ ˉ ( β / 2 + β 1 ) ≤ 2 1 . Δ ˉ \bar{\Delta} Δ ˉ ρ k ≥ 1 4 \rho_k\ge\frac{1}{4} ρ k ≥ 4 1 Δ k \Delta_k Δ k [ 0 , Δ ˉ ] [0,\bar{\Delta}] [ 0 , Δ ˉ ] Δ k + 1 ≥ Δ k \Delta_{k+1}\ge\Delta_k Δ k + 1 ≥ Δ k 1 4 \frac{1}{4} 4 1 Δ k ≥ Δ ˉ \Delta_k\ge\bar{\Delta} Δ k ≥ Δ ˉ Δ k ≥ min ( Δ K , Δ ˉ / 4 ) , ∀ k ≥ K . \Delta_k\ge\min(\Delta_K,\bar{\Delta}/4),\quad \forall k\ge K. Δ k ≥ min ( Δ K , Δ ˉ / 4 ) , ∀ k ≥ K . K \mathcal{K} K ρ k ≥ 1 4 , k ∈ K \rho_k\ge\frac{1}{4},k\in\mathcal{K} ρ k ≥ 4 1 , k ∈ K k ∈ K , k ≥ K k\in\mathcal{K},k\ge K k ∈ K , k ≥ K f ( x k ) − f ( x k + 1 ) = f ( x k ) + f ( x k + p k ) ≥ 1 4 [ m k ( 0 ) − m k ( p k ) ] ≥ 1 4 c 1 ϵ min ( Δ k , ϵ / β ) . \begin{aligned}f(x_k)-f(x_{k+1})&=f(x_k)+f(x_k+p_k)\\&\ge\frac{1}{4}[m_k(0)-m_k(p_k)]\\&\ge\frac{1}{4}c_1\epsilon\min(\Delta_k,\epsilon/\beta).\end{aligned} f ( x k ) − f ( x k + 1 ) = f ( x k ) + f ( x k + p k ) ≥ 4 1 [ m k ( 0 ) − m k ( p k ) ] ≥ 4 1 c 1 ϵ min ( Δ k , ϵ / β ) . f f f lim k ∈ K , k → ∞ Δ k = 0. \lim_{k\in\mathcal{K},k\to\infty}\Delta_k=0. k ∈ K , k → ∞ lim Δ k = 0 . Δ k ≥ min ( Δ K , Δ ˉ / 4 ) \Delta_k\ge\min(\Delta_K,\bar{\Delta}/4) Δ k ≥ min ( Δ K , Δ ˉ / 4 ) K \mathcal{K} K k k k ρ k < 1 4 \rho_k<\frac{1}{4} ρ k < 4 1 Δ k \Delta_k Δ k 1 4 \frac{1}{4} 4 1 lim k → ∞ Δ k = 0 \lim_{k\to\infty}\Delta_k=0 lim k → ∞ Δ k = 0
定理6 设算法1中η ∈ ( 0 , 1 4 ) \eta\in(0,\frac{1}{4}) η ∈ ( 0 , 4 1 ) β \beta β ∥ B k ∥ ≤ β , ∀ k \Vert B_k\Vert\le\beta,\forall k ∥ B k ∥ ≤ β , ∀ k f f f S S S S ( R 0 ) S(R_0) S ( R 0 ) p k p_k p k m k ( 0 ) − m k ( p k ) ≥ c 1 ∥ g k ∥ min ( Δ k , ∥ g k ∥ ∥ B k ∥ ) , c 1 ∈ ( 0 , 1 ] m_k(0)-m_k(p_k)\ge c_1\Vert g_k\Vert\min\left(\Delta_k,\frac{\Vert g_k\Vert}{\Vert B_k\Vert}\right), \quad c_1\in(0,1] m k ( 0 ) − m k ( p k ) ≥ c 1 ∥ g k ∥ min ( Δ k , ∥ B k ∥ ∥ g k ∥ ) , c 1 ∈ ( 0 , 1 ] ∥ p k ∥ ≤ γ Δ k , γ ≥ 1. \Vert p_k\Vert\le\gamma\Delta_k,\quad \gamma\ge1. ∥ p k ∥ ≤ γ Δ k , γ ≥ 1 . lim k → ∞ ∥ g k ∥ = 0. \lim_{k\to\infty}\Vert g_k\Vert=0. k → ∞ lim ∥ g k ∥ = 0 . m : g m ≠ 0 m:g_m\ne0 m : g m ̸ = 0 ∥ g ( x ) − g m ∥ ≤ β 1 ∥ x − x m ∥ , ∀ x ∈ S ( R 0 ) . \Vert g(x)-g_m\Vert\le\beta_1\Vert x-x_m\Vert,\quad \forall x\in S(R_0). ∥ g ( x ) − g m ∥ ≤ β 1 ∥ x − x m ∥ , ∀ x ∈ S ( R 0 ) . ϵ = 1 2 ∥ g m ∥ , R = min ( ϵ β 1 , R 0 ) . \epsilon=\frac{1}{2}\Vert g_m\Vert,\quad R=\min\left(\frac{\epsilon}{\beta_1},R_0\right). ϵ = 2 1 ∥ g m ∥ , R = min ( β 1 ϵ , R 0 ) . B ( x m , R ) = { x ∣ ∥ x − x m ∥ ≤ R } ⊂ S ( R 0 ) , \mathcal{B}(x_m,R)=\{x|\Vert x-x_m\Vert\le R\}\subset S(R_0), B ( x m , R ) = { x ∣ ∥ x − x m ∥ ≤ R } ⊂ S ( R 0 ) , g g g B ( x m , R ) \mathcal{B}(x_m,R) B ( x m , R ) x ∈ B ( x m , R ) ⇒ ∥ g ( x ) ∥ ≥ ∥ g m ∥ − ∥ g ( x ) − g m ∥ ≥ 1 2 ∥ g m ∥ = ϵ . x\in\mathcal{B}(x_m,R)\Rightarrow\Vert g(x)\Vert\ge\Vert g_m\Vert-\Vert g(x)-g_m\Vert\ge\frac{1}{2}\Vert g_m\Vert=\epsilon. x ∈ B ( x m , R ) ⇒ ∥ g ( x ) ∥ ≥ ∥ g m ∥ − ∥ g ( x ) − g m ∥ ≥ 2 1 ∥ g m ∥ = ϵ . { x k } k ≥ m \{x_k\}_{k\ge m } { x k } k ≥ m B ( x m , R ) \mathcal{B}(x_m,R) B ( x m , R ) ∥ g k ∥ ≥ ϵ > 0 , k ≥ m \Vert g_k\Vert\ge\epsilon>0,k\ge m ∥ g k ∥ ≥ ϵ > 0 , k ≥ m { x k } k ≥ m \{x_k\}_{k\ge m} { x k } k ≥ m B ( x m , R ) \mathcal{B}(x_m,R) B ( x m , R ) l ≥ m l\ge m l ≥ m x l + 1 x_{l+1} x l + 1 x m x_m x m B ( x m , R ) \mathcal{B}(x_m,R) B ( x m , R ) ∥ g k ∥ ≥ ϵ , k = m , m + 1 , … , l \Vert g_k\Vert\ge\epsilon,k=m,m+1,\ldots,l ∥ g k ∥ ≥ ϵ , k = m , m + 1 , … , l f ( x m ) − f ( x l + 1 ) = ∑ k = m l f ( x k ) − f ( x k + 1 ) ≥ ∑ k = m , x k ≠ x k + 1 l η [ m k ( 0 ) − m k ( p k ) ] ≥ ∑ k = m , x k ≠ x k + 1 l η c 1 ϵ min ( Δ k , ϵ β ) . \begin{aligned}f(x_m)-f(x_{l+1})&=\sum_{k=m}^lf(x_k)-f(x_{k+1})\\&\ge\sum_{k=m,x_k\ne x_{k+1}}^l\eta[m_k(0)-m_k(p_k)]\\&\ge\sum_{k=m,x_k\ne x_{k+1}}^l\eta c_1\epsilon\min\left(\Delta_k,\frac{\epsilon}{\beta}\right).\end{aligned} f ( x m ) − f ( x l + 1 ) = k = m ∑ l f ( x k ) − f ( x k + 1 ) ≥ k = m , x k ̸ = x k + 1 ∑ l η [ m k ( 0 ) − m k ( p k ) ] ≥ k = m , x k ̸ = x k + 1 ∑ l η c 1 ϵ min ( Δ k , β ϵ ) . Δ k ≤ ϵ / β , k = m , m + 1 , … , l \Delta_k\le\epsilon/\beta,k=m,m+1,\ldots,l Δ k ≤ ϵ / β , k = m , m + 1 , … , l f ( x m ) − f ( x l + 1 ) ≥ η c 1 ϵ ∑ k = m , x k ≠ x k + 1 l Δ k ≥ η c 1 ϵ R = η c 1 ϵ min ( ϵ β 1 , R 0 ) , f(x_m)-f(x_{l+1})\ge\eta c_1\epsilon\sum_{k=m,x_k\ne x_{k+1}}^l\Delta_k\ge\eta c_1\epsilon R=\eta c_1\epsilon\min\left(\frac{\epsilon}{\beta_1},R_0\right), f ( x m ) − f ( x l + 1 ) ≥ η c 1 ϵ k = m , x k ̸ = x k + 1 ∑ l Δ k ≥ η c 1 ϵ R = η c 1 ϵ min ( β 1 ϵ , R 0 ) , x l + 1 x_{l+1} x l + 1 B ( x m , R ) \mathcal{B}(x_m,R) B ( x m , R ) Δ k > ϵ / β , \Delta_k>\epsilon/\beta, Δ k > ϵ / β , k = m , m + 1 , … , l k=m,m+1,\ldots,l k = m , m + 1 , … , l f ( x m ) − f ( x l + 1 ) ≥ η c 1 ϵ ϵ β . f(x_m)-f(x_{l+1})\ge\eta c_1\epsilon\frac{\epsilon}{\beta}. f ( x m ) − f ( x l + 1 ) ≥ η c 1 ϵ β ϵ . { f ( x k ) } k = 0 ∞ \{f(x_k)\}_{k=0}^{\infty} { f ( x k ) } k = 0 ∞ f ( x k ) ↓ f ∗ , f ∗ > − ∞ . f(x_k)\downarrow f^*, f^*>-\infty. f ( x k ) ↓ f ∗ , f ∗ > − ∞ . f ( x m ) − f ∗ ≥ f ( x m ) − f ( x l + 1 ) ≥ η c 1 ϵ min ( ϵ β , ϵ β 1 , R 0 ) = 1 2 η c 1 ∥ g m ∥ min ( ∥ g m ∥ 2 β , ∥ g m ∥ 2 β 1 , R 0 ) > 0. \begin{aligned}f(x_m)-f^*&\ge f(x_m)-f(x_{l+1})\\&\ge\eta c_1\epsilon\min\left(\frac{\epsilon}{\beta},\frac{\epsilon}{\beta_1},R_0\right)\\&=\frac{1}{2}\eta c_1\Vert g_m\Vert\min\left(\frac{\Vert g_m\Vert}{2\beta},\frac{\Vert g_m\Vert}{2\beta_1},R_0\right)>0.\end{aligned} f ( x m ) − f ∗ ≥ f ( x m ) − f ( x l + 1 ) ≥ η c 1 ϵ min ( β ϵ , β 1 ϵ , R 0 ) = 2 1 η c 1 ∥ g m ∥ min ( 2 β ∥ g m ∥ , 2 β 1 ∥ g m ∥ , R 0 ) > 0 . f ( x m ) − f ∗ ↓ 0 f(x_m)-f^*\downarrow 0 f ( x m ) − f ∗ ↓ 0 lim m → ∞ g m = 0 \lim_{m\to\infty}g_m=0 lim m → ∞ g m = 0
注意:
第二个定理中η \eta η 1 4 \frac{1}{4} 4 1 1 4 \frac{1}{4} 4 1 ρ k < 1 4 \rho_k<\frac{1}{4} ρ k < 4 1 η > 1 4 \eta>\frac{1}{4} η > 4 1 缩小了信赖域却没有更新x k x_k x k B k B_k B k p k p_k p k x k x_k x k x k x_k x k
这两个定理充分体现了信赖域算法的优越性 , 即, 即使B k B_k B k . 这主要是因为下降估计对于折线算法与二维子空间最小化算法总是成立的:m k ( 0 ) − m k ( p k ) ≥ c 1 ∥ g k ∥ min ( Δ k , ∥ g k ∥ ∥ B k ∥ ) . m_k(0)-m_k(p_k)\ge c_1\Vert g_k\Vert\min\left(\Delta_k,\frac{\Vert g_k\Vert}{\Vert B_k\Vert}\right). m k ( 0 ) − m k ( p k ) ≥ c 1 ∥ g k ∥ min ( Δ k , ∥ B k ∥ ∥ g k ∥ ) . 关键在于Cauchy点的下降估计 .)
本节中主要利用定理1来刻画子问题的解. 前面第1节中讨论的方法, 毫无疑问是近似求解子问题的方法. 这种方法具有易于实施且全局收敛的性质. 但是, 当问题规模较小时, 我们不妨更加精确地求解子问题 . 本节所要描述的方法就是, 基于矩阵分解 (与折线算法与二维子空间最小化算法中的单次分解相比) 与牛顿法求根的方法. 最后我们还将给出定理1的证明.p p p ( B + λ I ) p = − g , (B+\lambda I)p=-g, ( B + λ I ) p = − g , λ ( ∥ p ∥ − Δ ) = 0 , \lambda(\Vert p\Vert-\Delta)=0, λ ( ∥ p ∥ − Δ ) = 0 , B + λ I 半 正 定 . B+\lambda I半正定. B + λ I 半 正 定 . λ = 0 \lambda=0 λ = 0 p ( λ ) = − ( B + λ I ) − 1 g , p(\lambda)=-(B+\lambda I)^{-1}g, p ( λ ) = − ( B + λ I ) − 1 g , λ \lambda λ B + λ I B+\lambda I B + λ I λ > 0 \lambda>0 λ > 0 ∥ p ( λ ) ∥ = Δ . \Vert p(\lambda)\Vert=\Delta. ∥ p ( λ ) ∥ = Δ . B B B B = Q Λ Q T B=Q\Lambda Q^T B = Q Λ Q T Q Q Q Λ = d i a g ( λ 1 , λ 2 , … , λ n ) , \Lambda=\mathrm{diag}(\lambda_1,\lambda_2,\ldots,\lambda_n), Λ = d i a g ( λ 1 , λ 2 , … , λ n ) , λ 1 ≤ λ 2 ≤ … ≤ λ n \lambda_1\le\lambda_2\le\ldots\le\lambda_n λ 1 ≤ λ 2 ≤ … ≤ λ n B B B B + λ I = Q ( Λ + λ I ) Q T B+\lambda I=Q(\Lambda+\lambda I)Q^T B + λ I = Q ( Λ + λ I ) Q T λ ≠ λ j \lambda\ne\lambda_j λ ̸ = λ j p ( λ ) = − Q ( Λ + λ I ) − 1 Q T g = − ∑ j = 1 n q j T g λ j + λ q j , p(\lambda)=-Q(\Lambda+\lambda I)^{-1}Q^Tg=-\sum_{j=1}^n\frac{q_j^Tg}{\lambda_j+\lambda}q_j, p ( λ ) = − Q ( Λ + λ I ) − 1 Q T g = − j = 1 ∑ n λ j + λ q j T g q j , q j q_j q j Q Q Q j j j Q Q Q ∥ p ( λ ) ∥ 2 = ∑ j = 1 n ( q j T g ) 2 ( λ j + λ ) 2 . \Vert p(\lambda)\Vert^2=\sum_{j=1}^n\frac{\left(q_j^Tg\right)^2}{(\lambda_j+\lambda)^2}. ∥ p ( λ ) ∥ 2 = j = 1 ∑ n ( λ j + λ ) 2 ( q j T g ) 2 . λ > − λ 1 \lambda>-\lambda_1 λ > − λ 1 λ j + λ > 0 , ∀ j \lambda_j+\lambda>0,\forall j λ j + λ > 0 , ∀ j ∥ p ( λ ) ∥ \Vert p(\lambda)\Vert ∥ p ( λ ) ∥ ( − λ 1 , ∞ ) (-\lambda_1,\infty) ( − λ 1 , ∞ ) lim λ → ∞ ∥ p ( λ ) ∥ = 0. \lim_{\lambda\to\infty}\Vert p(\lambda)\Vert=0. λ → ∞ lim ∥ p ( λ ) ∥ = 0 . q j T g ≠ 0 q_j^Tg\ne0 q j T g ̸ = 0 lim λ → λ j ∥ p ( λ ) ∥ = ∞ . \lim_{\lambda\to\lambda_j}\Vert p(\lambda)\Vert=\infty. λ → λ j lim ∥ p ( λ ) ∥ = ∞ .
图中显示, q 1 T g , q 2 T g , q 3 T g q_1^Tg,q_2^Tg,q_3^Tg q 1 T g , q 2 T g , q 3 T g q 1 T g ≠ 0 q_1^Tg\ne0 q 1 T g ̸ = 0 λ ∗ ∈ ( − λ 1 , ∞ ) \lambda^*\in(-\lambda_1,\infty) λ ∗ ∈ ( − λ 1 , ∞ ) ∥ p ( λ ∗ ) ∥ = Δ \Vert p(\lambda^*)\Vert =\Delta ∥ p ( λ ∗ ) ∥ = Δ λ ∗ \lambda^* λ ∗ − λ 1 -\lambda_1 − λ 1 λ \lambda λ
假设q 1 T g ≠ 0 q_1^Tg\ne0 q 1 T g ̸ = 0 ∥ p ( λ ) ∥ − Δ \Vert p(\lambda)\Vert -\Delta ∥ p ( λ ) ∥ − Δ ( − λ 1 , ∞ ) (-\lambda_1,\infty) ( − λ 1 , ∞ ) B B B ∥ B − 1 g ∥ ≤ Δ \Vert B^{-1}g\Vert\le\Delta ∥ B − 1 g ∥ ≤ Δ λ = 0 \lambda=0 λ = 0 λ ∗ = 0 \lambda^*=0 λ ∗ = 0 ϕ ( λ ) = ∥ p ( λ ) ∥ − Δ = 0. \phi(\lambda)=\Vert p(\lambda)\Vert-\Delta=0. ϕ ( λ ) = ∥ p ( λ ) ∥ − Δ = 0 . 直接拿ϕ \phi ϕ . 注意到, 当λ \lambda λ − λ 1 -\lambda_1 − λ 1 ϕ \phi ϕ ϕ 1 ( λ ) ≈ C 1 λ + λ 1 + C 2 , \phi_1(\lambda)\approx\frac{C_1}{\lambda+\lambda_1}+C_2, ϕ 1 ( λ ) ≈ λ + λ 1 C 1 + C 2 , C 1 , C 2 C_1,C_2 C 1 , C 2 高度非线性 , 从而牛顿法并不可靠 (收敛很慢) . 不过如果考虑等价的方程ϕ 2 ( λ ) = 1 Δ − 1 ∥ p ( λ ) ∥ , \phi_2(\lambda)=\frac{1}{\Delta}-\frac{1}{\Vert p(\lambda)\Vert}, ϕ 2 ( λ ) = Δ 1 − ∥ p ( λ ) ∥ 1 , − λ 1 -\lambda_1 − λ 1 ϕ 2 ( λ ) ≈ 1 Δ − λ + λ 1 C 3 , \phi_2(\lambda)\approx\frac{1}{\Delta}-\frac{\lambda+\lambda_1}{C_3}, ϕ 2 ( λ ) ≈ Δ 1 − C 3 λ + λ 1 , C 3 > 0 C_3>0 C 3 > 0 ϕ 2 \phi_2 ϕ 2 − λ 1 -\lambda_1 − λ 1 接近线性 , 从而只要算法将保持λ > − λ 1 \lambda>-\lambda_1 λ > − λ 1 λ ( l + 1 ) = λ ( l ) − ϕ 2 ( λ ( l ) ) ϕ 2 ′ ( λ ( l ) ) . \lambda^{(l+1)}=\lambda^{(l)}-\frac{\phi_2(\lambda^{(l)})}{\phi_2'(\lambda^{(l)})}. λ ( l + 1 ) = λ ( l ) − ϕ 2 ′ ( λ ( l ) ) ϕ 2 ( λ ( l ) ) . B + λ I B+\lambda I B + λ I B + λ I = R T R , B+\lambda I=R^TR, B + λ I = R T R , R T R p = − g R^TRp=-g R T R p = − g q q q R T q = p R^Tq=p R T q = p ∥ q ∥ 2 = ∥ R − T p ∥ 2 = p T R − 1 R − T p = p T ( B + λ I ) − 1 p = p T Q ( Λ + λ I ) − 1 Q T p = ∑ j = 1 n ( q j T p ) 2 λ + λ j = ∑ j = 1 n ( q j T ∑ k = 1 n q k T g λ + λ k q k ) 2 λ + λ j = ∑ j = 1 n ( q j T g ) 2 ( λ + λ j ) 3 . \begin{aligned}\Vert q\Vert^2&=\Vert R^{-T}p\Vert^2\\&=p^TR^{-1}R^{-T}p\\&=p^T(B+\lambda I)^{-1}p\\&=p^TQ(\Lambda+\lambda I)^{-1}Q^Tp\\&=\sum_{j=1}^n\frac{(q_j^Tp)^2}{\lambda+\lambda_j}\\&=\sum_{j=1}^n\frac{\left(q_j^T\sum_{k=1}^n\frac{q_k^Tg}{\lambda+\lambda_k}q_k\right)^2}{\lambda+\lambda_j}\\&=\sum_{j=1}^n\frac{(q_j^Tg)^2}{(\lambda+\lambda_j)^3}.\end{aligned} ∥ q ∥ 2 = ∥ R − T p ∥ 2 = p T R − 1 R − T p = p T ( B + λ I ) − 1 p = p T Q ( Λ + λ I ) − 1 Q T p = j = 1 ∑ n λ + λ j ( q j T p ) 2 = j = 1 ∑ n λ + λ j ( q j T ∑ k = 1 n λ + λ k q k T g q k ) 2 = j = 1 ∑ n ( λ + λ j ) 3 ( q j T g ) 2 . ϕ 2 ′ ( λ ) \phi_2'(\lambda) ϕ 2 ′ ( λ ) ϕ 2 ′ ( λ ) = ( 1 Δ − 1 ∥ p ∥ ) ′ = 1 ∥ p ∥ 2 d d λ ∥ p ∥ = 1 ∥ p ∥ 2 d d λ ( ∑ j = 1 n ( q j T g ) 2 ( λ + λ j ) 2 ) = 1 2 ∥ p ∥ 3 ∑ j = 1 n d d λ ( ( q j T g ) 2 ( λ + λ j ) 2 ) = − 1 ∥ p ∥ 3 ∑ j = 1 n ( q j T g ) 2 ( λ + λ j ) 3 = − ∥ q ∥ 2 ∥ p ∥ 3 . \begin{aligned}\phi_2'(\lambda)&=\left(\frac{1}{\Delta}-\frac{1}{\Vert p\Vert}\right)'\\&=\frac{1}{\Vert p\Vert^2}\frac{\mathrm{d}}{\mathrm{d}\lambda}\Vert p\Vert\\&=\frac{1}{\Vert p\Vert^2}\frac{\mathrm{d}}{\mathrm{d}\lambda}\left(\sqrt{\sum_{j=1}^n\frac{(q_j^Tg)^2}{(\lambda+\lambda_j)^2}}\right)\\&=\frac{1}{2\Vert p\Vert^3}\sum_{j=1}^n\frac{\mathrm{d}}{\mathrm{d}\lambda}\left(\frac{(q_j^Tg)^2}{(\lambda+\lambda_j)^2}\right)\\&=-\frac{1}{\Vert p\Vert^3}\sum_{j=1}^n\frac{(q_j^Tg)^2}{(\lambda+\lambda_j)^3}=-\frac{\Vert q\Vert^2}{\Vert p\Vert^3}.\end{aligned} ϕ 2 ′ ( λ ) = ( Δ 1 − ∥ p ∥ 1 ) ′ = ∥ p ∥ 2 1 d λ d ∥ p ∥ = ∥ p ∥ 2 1 d λ d ( j = 1 ∑ n ( λ + λ j ) 2 ( q j T g ) 2 ) = 2 ∥ p ∥ 3 1 j = 1 ∑ n d λ d ( ( λ + λ j ) 2 ( q j T g ) 2 ) = − ∥ p ∥ 3 1 j = 1 ∑ n ( λ + λ j ) 3 ( q j T g ) 2 = − ∥ p ∥ 3 ∥ q ∥ 2 . λ ( l + 1 ) = λ ( l ) − ϕ 2 ( λ ( l ) ) ϕ 2 ′ ( λ ( l ) ) = λ ( l ) + ∥ p ∥ 3 ∥ q ∥ 2 ∥ p ∥ − Δ ∥ p ∥ Δ = λ ( l ) + ∥ p ∥ 2 ∥ q ∥ 2 ∥ p ∥ − Δ Δ . \begin{aligned}\lambda^{(l+1)}&=\lambda^{(l)}-\frac{\phi_2(\lambda^{(l)})}{\phi_2'(\lambda^{(l)})}\\&=\lambda^{(l)}+\frac{\Vert p\Vert^3}{\Vert q\Vert^2}\frac{\Vert p\Vert-\Delta}{\Vert p\Vert\Delta}\\&=\lambda^{(l)}+\frac{\Vert p\Vert^2}{\Vert q\Vert^2}\frac{\Vert p\Vert -\Delta}{\Delta}.\end{aligned} λ ( l + 1 ) = λ ( l ) − ϕ 2 ′ ( λ ( l ) ) ϕ 2 ( λ ( l ) ) = λ ( l ) + ∥ q ∥ 2 ∥ p ∥ 3 ∥ p ∥ Δ ∥ p ∥ − Δ = λ ( l ) + ∥ q ∥ 2 ∥ p ∥ 2 Δ ∥ p ∥ − Δ .
算法2 (信赖域子问题算法)λ ( 0 ) , Δ > 0 \lambda^{(0)},\Delta>0 λ ( 0 ) , Δ > 0 l = 0 , 1 , 2 , … l=0,1,2,\ldots l = 0 , 1 , 2 , … 分 解 B + λ ( l ) I = R T R \quad\quad分解B+\lambda^{(l)}I=R^TR 分 解 B + λ ( l ) I = R T R 求 解 R T R p l = − g , R T q l = p l \quad\quad求解R^TRp_l=-g,R^Tq_l=p_l 求 解 R T R p l = − g , R T q l = p l 置 λ ( l + 1 ) = λ ( l ) + ∥ p l ∥ 2 ∥ q l ∥ 2 ∥ p l ∥ − Δ Δ \quad\quad置\lambda^{(l+1)}=\lambda^{(l)}+\frac{\Vert p_l\Vert^2}
{\Vert q_l\Vert^2}\frac{\Vert p_l\Vert -\Delta}{\Delta} 置 λ ( l + 1 ) = λ ( l ) + ∥ q l ∥ 2 ∥ p l ∥ 2 Δ ∥ p l ∥ − Δ
注:
算法2中并非单纯地使用牛顿法求根 , 而是在其中穿插了矩阵分解, 这里用于分解的矩阵也来源于上一步迭代产生的新λ \lambda λ
算法2中必须增加一些保护措施以保证算法的实用性 : 例如, 当λ ( l ) < − λ 1 \lambda^{(l)}<-\lambda_1 λ ( l ) < − λ 1 B + λ ( l ) I = R T R B+\lambda^{(l)}I=R^TR B + λ ( l ) I = R T R
算法2的主要计算量在于Cholesky分解. 此算法的实用版本并不等到λ \lambda λ λ \lambda λ . 事实证明这样的求解效果也是不错的.
上述方法也适用于最小特征值λ 1 \lambda_1 λ 1 0 > λ 1 = λ 2 = ⋯ 0>\lambda_1=\lambda_2=\cdots 0 > λ 1 = λ 2 = ⋯ Q 1 T g ≠ 0 Q_1^Tg\ne0 Q 1 T g ̸ = 0 Q 1 Q_1 Q 1 λ 1 \lambda_1 λ 1
算法2的收敛性将在后文给出说明.
当q 1 T g = 0 q_1^Tg=0 q 1 T g = 0 lim λ → − λ 1 ∥ p ( λ ) ∥ = + ∞ \lim_{\lambda\to-\lambda_1}\Vert p(\lambda)\Vert=+\infty λ → − λ 1 lim ∥ p ( λ ) ∥ = + ∞ ( − λ 1 , ∞ ) (-\lambda_1, \infty) ( − λ 1 , ∞ ) ∥ p ( λ ) ∥ = Δ \Vert p(\lambda)\Vert=\Delta ∥ p ( λ ) ∥ = Δ λ = − λ 1 \lambda=-\lambda_1 λ = − λ 1 λ j = λ 1 \lambda_j=\lambda_1 λ j = λ 1 p = ∑ j : λ j ≠ λ 1 q j T g λ + λ j q j p=\sum_{j:\lambda_j\ne\lambda_1}\frac{q_j^Tg}{\lambda+\lambda_j}q_j p = j : λ j ̸ = λ 1 ∑ λ + λ j q j T g q j ( B − λ 1 I ) (B-\lambda_1 I) ( B − λ 1 I ) z : ∥ z ∥ = 1 z:\Vert z\Vert=1 z : ∥ z ∥ = 1 ( B − λ 1 I ) z = 0 (B-\lambda_1I)z=0 ( B − λ 1 I ) z = 0 z z z B B B λ 1 \lambda_1 λ 1 Q Q Q q j T z = 0 , λ j ≠ λ 1 q_j^Tz=0,\lambda_j\ne\lambda_1 q j T z = 0 , λ j ̸ = λ 1 p = ∑ j : λ j ≠ λ 1 q j T g λ + λ j q j + τ z , p=\sum_{j:\lambda_j\ne\lambda_1}\frac{q_j^Tg}{\lambda+\lambda_j}q_j+\tau z, p = j : λ j ̸ = λ 1 ∑ λ + λ j q j T g q j + τ z , ∥ p ∥ 2 = ∑ j : λ j ≠ λ 1 ( q j T g ) 2 ( λ + λ j ) 2 + τ 2 . \Vert p\Vert^2=\sum_{j:\lambda_j\ne\lambda_1}\frac{\left(q_j^Tg\right)^2}{(\lambda+\lambda_j)^2}+\tau^2. ∥ p ∥ 2 = j : λ j ̸ = λ 1 ∑ ( λ + λ j ) 2 ( q j T g ) 2 + τ 2 . τ \tau τ ∥ p ∥ = Δ \Vert p\Vert=\Delta ∥ p ∥ = Δ p p p λ = − λ 1 \lambda=-\lambda_1 λ = − λ 1
引理 设m m m m ( p ) = g T p + 1 2 p T B p , m(p)=g^Tp+\frac{1}{2}p^TBp, m ( p ) = g T p + 2 1 p T B p , B B B
m m m B B B g g g B B B B B B p : B p = − g p:Bp=-g p : B p = − g m m m
m m m B B B
证明:
我们分充分性和必要性证明.
充分性. 由于g g g B B B p p p B p = − g Bp=-g B p = − g ∀ w ∈ R n \forall w\in\mathbb{R}^n ∀ w ∈ R n m ( p + w ) = g T ( p + w ) + 1 2 ( p + w ) T B ( p + w ) = ( g T p + 1 2 p T B p ) + g T w + ( B p ) T w + 1 2 w T B w = m ( p ) + 1 2 w T B w ≥ m ( p ) . \begin{aligned}m(p+w)&=g^T(p+w)+\frac{1}{2}(p+w)^TB(p+w)\\&=(g^Tp+\frac{1}{2}p^TBp)+g^Tw+(Bp)^Tw+\frac{1}{2}w^TBw\\&=m(p)+\frac{1}{2}w^TBw\\&\ge m(p).\end{aligned} m ( p + w ) = g T ( p + w ) + 2 1 ( p + w ) T B ( p + w ) = ( g T p + 2 1 p T B p ) + g T w + ( B p ) T w + 2 1 w T B w = m ( p ) + 2 1 w T B w ≥ m ( p ) .
必要性. 设p p p m m m ∇ m ( p ) = B p + g = 0 \nabla m(p)=Bp+g=0 ∇ m ( p ) = B p + g = 0 g g g B B B ∇ 2 m ( p ) = B \nabla^2m(p)=B ∇ 2 m ( p ) = B
充分性可由1的充分性中, w T B w > 0 , ∀ w ≠ 0 w^TBw>0,\forall w\ne0 w T B w > 0 , ∀ w ̸ = 0 B B B w ≠ 0 w\ne0 w ̸ = 0 B w = 0 Bw=0 B w = 0 m ( p ) = m ( p + w ) m(p)=m(p+w) m ( p ) = m ( p + w ) p p p m m m
下面证明定理1. 重述如下:
定理1 向量p ∗ p^* p ∗ min p ∈ R n m ( p ) = g T p + 1 2 p T B p , s . t . ∥ p ∥ ≤ Δ \min_{p\in\mathbb{R}^n}m(p)=g^Tp+\frac{1}{2}p^TBp,\quad\mathrm{s.t.}\Vert p\Vert\le\Delta p ∈ R n min m ( p ) = g T p + 2 1 p T B p , s . t . ∥ p ∥ ≤ Δ p ∗ p^* p ∗ λ ≥ 0 \lambda\ge0 λ ≥ 0 ( B + λ I ) p ∗ = − g , (B+\lambda I)p^*=-g, ( B + λ I ) p ∗ = − g , λ ( Δ − ∥ p ∗ ∥ ) = 0 , \lambda(\Delta-\Vert p^*\Vert)=0, λ ( Δ − ∥ p ∗ ∥ ) = 0 , ( B + λ I ) 半 正 定 . (B+\lambda I)半正定. ( B + λ I ) 半 正 定 . λ ≥ 0 \lambda\ge0 λ ≥ 0 p ∗ p^* p ∗ m ^ ( p ) = g T p + 1 2 p T ( B + λ I ) p = m ( p ) + λ 2 p T p \hat{m}(p)=g^Tp+\frac{1}{2}p^T(B+\lambda I)p=m(p)+\frac{\lambda}{2}p^Tp m ^ ( p ) = g T p + 2 1 p T ( B + λ I ) p = m ( p ) + 2 λ p T p m ^ ( p ) ≥ m ^ ( p ∗ ) , ∀ p \hat{m}(p)\ge\hat{m}(p^*),\forall p m ^ ( p ) ≥ m ^ ( p ∗ ) , ∀ p m ( p ) ≥ m ( p ∗ ) + λ 2 ( ( p ∗ ) T p ∗ − p T p ) . m(p)\ge m(p^*)+\frac{\lambda}{2}\left((p^*)^Tp^*-p^Tp\right). m ( p ) ≥ m ( p ∗ ) + 2 λ ( ( p ∗ ) T p ∗ − p T p ) . λ ( Δ − ∥ p ∗ ∥ ) = 0 \lambda(\Delta-\Vert p^*\Vert)=0 λ ( Δ − ∥ p ∗ ∥ ) = 0 λ ( Δ 2 − ( p ∗ ) T p ∗ ) = 0 \lambda(\Delta^2-(p^*)^Tp^*)=0 λ ( Δ 2 − ( p ∗ ) T p ∗ ) = 0 m ( p ) ≥ m ( p ∗ ) + λ 2 ( Δ 2 − p T p ) . m(p)\ge m(p^*)+\frac{\lambda}{2}(\Delta^2-p^Tp). m ( p ) ≥ m ( p ∗ ) + 2 λ ( Δ 2 − p T p ) . λ ≥ 0 \lambda\ge0 λ ≥ 0 m ( p ) ≥ m ( p ∗ ) , ∀ p : ∥ p ∥ ≤ Δ m(p)\ge m(p^*),\forall p:\Vert p\Vert\le\Delta m ( p ) ≥ m ( p ∗ ) , ∀ p : ∥ p ∥ ≤ Δ p ∗ p^* p ∗ m m m p ∗ p^* p ∗ m m m λ ≥ 0 \lambda\ge0 λ ≥ 0
若∥ p ∗ ∥ < Δ \Vert p^*\Vert<\Delta ∥ p ∗ ∥ < Δ p ∗ p^* p ∗ m m m ∇ m ( p ∗ ) = B p ∗ + g = 0 , ∇ 2 m ( p ∗ ) = B 半 正 定 , \nabla m(p^*)=Bp^*+g=0,\quad\nabla^2m(p^*)=B半正定, ∇ m ( p ∗ ) = B p ∗ + g = 0 , ∇ 2 m ( p ∗ ) = B 半 正 定 , λ = 0 \lambda=0 λ = 0
若∥ p ∗ ∥ = Δ \Vert p^*\Vert=\Delta ∥ p ∗ ∥ = Δ p ∗ p^* p ∗ min m ( p ) s . t . ∥ p ∥ = Δ \min m(p)\quad\mathrm{s.t.}\Vert p\Vert=\Delta min m ( p ) s . t . ∥ p ∥ = Δ L ( p , λ ) = m ( p ) + λ 2 ( p T p − Δ 2 ) , \mathcal{L}(p,\lambda)=m(p)+\frac{\lambda}{2}(p^Tp-\Delta^2), L ( p , λ ) = m ( p ) + 2 λ ( p T p − Δ 2 ) , λ \lambda λ p ∗ p^* p ∗ L \mathcal{L} L ∇ p L ( p ∗ , λ ) \nabla_p\mathcal{L}(p^*,\lambda) ∇ p L ( p ∗ , λ ) B p ∗ + g + λ p ∗ = 0 ⇒ ( B + λ I ) p ∗ = − g . Bp^*+g+\lambda p^*=0\Rightarrow (B+\lambda I)p^*=-g. B p ∗ + g + λ p ∗ = 0 ⇒ ( B + λ I ) p ∗ = − g . λ ≥ 0 \lambda\ge0 λ ≥ 0 m ( p ) ≥ m ( p ∗ ) , ∀ p : p T p = ( p ∗ ) T p ∗ = Δ 2 m(p)\ge m(p^*),\forall p:p^Tp=(p^*)^Tp^*=\Delta^2 m ( p ) ≥ m ( p ∗ ) , ∀ p : p T p = ( p ∗ ) T p ∗ = Δ 2 p p p m ( p ) ≥ m ( p ∗ ) + λ 2 ( ( p ∗ ) T p ∗ − p T p ) . m(p)\ge m(p^*)+\frac{\lambda}{2}\left((p^*)^Tp^*-p^Tp\right). m ( p ) ≥ m ( p ∗ ) + 2 λ ( ( p ∗ ) T p ∗ − p T p ) . ( B + λ I ) p ∗ = − g (B+\lambda I)p^*=-g ( B + λ I ) p ∗ = − g 1 2 ( p − p ∗ ) T ( B + λ I ) ( p − p ∗ ) ≥ 0. \frac{1}{2}(p-p^*)^T(B+\lambda I)(p-p^*)\ge0. 2 1 ( p − p ∗ ) T ( B + λ I ) ( p − p ∗ ) ≥ 0 . { w : w = ± p − p ∗ ∥ p − p ∗ ∥ , p : ∥ p ∥ = Δ } \left\{w:w=\pm\frac{p-p^*}{\Vert p-p^*\Vert},p:\Vert p\Vert=\Delta\right\} { w : w = ± ∥ p − p ∗ ∥ p − p ∗ , p : ∥ p ∥ = Δ } λ ≥ 0 \lambda\ge0 λ ≥ 0 p ∗ p^* p ∗ m ^ \hat{m} m ^ m ( p ) ≥ m ( p ∗ ) + λ 2 ( ( p ∗ ) T p ∗ − p T p ) , ∀ p : ∥ p ∥ ≤ Δ . m(p)\ge m(p^*)+\frac{\lambda}{2}\left((p^*)^Tp^*-p^Tp\right),\forall p:\Vert p\Vert\le\Delta. m ( p ) ≥ m ( p ∗ ) + 2 λ ( ( p ∗ ) T p ∗ − p T p ) , ∀ p : ∥ p ∥ ≤ Δ . λ < 0 \lambda<0 λ < 0 m ( p ) ≥ m ( p ∗ ) , ∀ p : ∥ p ∥ ≥ ∥ p ∗ ∥ = Δ . m(p)\ge m(p^*),\forall p:\Vert p\Vert\ge\Vert p^*\Vert=\Delta. m ( p ) ≥ m ( p ∗ ) , ∀ p : ∥ p ∥ ≥ ∥ p ∗ ∥ = Δ . p ∗ p^* p ∗ ∥ p ∥ ≤ Δ \Vert p\Vert\le\Delta ∥ p ∥ ≤ Δ m m m p ∗ p^* p ∗ m m m B p ∗ = − g , B 半 正 定 . Bp^*=-g,\quad B半正定. B p ∗ = − g , B 半 正 定 . λ = 0 \lambda=0 λ = 0
适用于小规模问题的算法2的注记中提到, 实用版本的算法2并不等到序列{ λ i } \{\lambda_i\} { λ i } m ( 0 ) − m ( p ) ≥ c 1 ( m ( 0 ) − m ( p ∗ ) ) , m(0)-m(p)\ge c_1(m(0)-m(p^*)), m ( 0 ) − m ( p ) ≥ c 1 ( m ( 0 ) − m ( p ∗ ) ) , ∥ p ∥ ≤ γ Δ , \Vert p\Vert\le\gamma\Delta, ∥ p ∥ ≤ γ Δ , p ∗ p^* p ∗ c 1 ∈ ( 0 , 1 ] , γ > 0 c_1\in(0,1], \gamma>0 c 1 ∈ ( 0 , 1 ] , γ > 0 m m m 注意这里并不需要知道p ∗ p^* p ∗ ) 这里的条件与先前的条件的一个主要的不同在于, 前者更好地利用了m ( ⋅ ) m(\cdot) m ( ⋅ ) 假若g = 0 g=0 g = 0 B B B x k x_k x k x k x_k x k .f f f B = ∇ 2 f ( x ) B=\nabla^2f(x) B = ∇ 2 f ( x ) 由非精确解算法产生的序列的聚点满足二阶必要条件 .
定理7 假设定理6的条件都成立, 另外f f f S S S B k = ∇ 2 f ( x k ) , ∀ k B_k=\nabla^2f(x_k),\forall k B k = ∇ 2 f ( x k ) , ∀ k p k p_k p k m ( 0 ) − m ( p ) ≥ c 1 ( m ( 0 ) − m ( p ∗ ) ) , m(0)-m(p)\ge c_1(m(0)-m(p^*)), m ( 0 ) − m ( p ) ≥ c 1 ( m ( 0 ) − m ( p ∗ ) ) , ∥ p ∥ ≤ γ Δ , \Vert p\Vert\le\gamma\Delta, ∥ p ∥ ≤ γ Δ , γ > 0 \gamma>0 γ > 0 lim k → ∞ ∥ g k ∥ = 0. \lim_{k\to\infty}\Vert g_k\Vert=0. k → ∞ lim ∥ g k ∥ = 0 . S S S x k x_k x k { x k } \{x_k\} { x k } S S S x ∗ x^* x ∗
前面已经提到信赖域牛顿法的全局收敛性, 下面来讨论其局部的收敛速度. 与牛顿法相关的算法要想要获得较快的收敛速度, 关键是在接近解的时候, 相关的限制不再对牛顿步起作用 . 具体地, 我们希望在靠近解的时候, 信赖域子问题的近似解老实地待在信赖域内部, 并愈来愈接近牛顿步. 满足后面这一性质的迭代步, 我们称之渐进相似于牛顿步 .信赖域的约束最终变得不积极从而算法达到超线性收敛速度 .
定理8 设f f f x ∗ x^* x ∗ { x k } \{x_k\} { x k } x ∗ x^* x ∗ k k k p k p_k p k ∥ p k N ∥ ≤ 1 2 Δ k \Vert p_k^N\Vert\le\frac{1}{2}\Delta_k ∥ p k N ∥ ≤ 2 1 Δ k p k p_k p k ∥ p k − p k N ∥ = o ( ∥ p k N ∥ ) . \Vert p_k-p_k^N\Vert=o(\Vert p_k^N\Vert). ∥ p k − p k N ∥ = o ( ∥ p k N ∥ ) . Δ k \Delta_k Δ k k k k { x k } \{x_k\} { x k } x ∗ x^* x ∗
证明: 我们证明对于充分大的k k k ∥ p k N ∥ < 1 2 Δ k \Vert p_k^N\Vert<\frac{1}{2}\Delta_k ∥ p k N ∥ < 2 1 Δ k ∵ ∥ p k ∥ ≤ ∥ p k N ∥ + o ( ∥ p k N ∥ ) ≤ 2 ∥ p k N ∥ < Δ k \because \Vert p_k\Vert\le\Vert p_k^N\Vert+o(\Vert p_k^N\Vert)\le2\Vert p_k^N\Vert<\Delta_k ∵ ∥ p k ∥ ≤ ∥ p k N ∥ + o ( ∥ p k N ∥ ) ≤ 2 ∥ p k N ∥ < Δ k k k k m k ( 0 ) − m k ( p k ) m_k(0)-m_k(p_k) m k ( 0 ) − m k ( p k ) k k k o ( ∥ p k N ∥ ) o(\Vert p_k^N\Vert) o ( ∥ p k N ∥ ) ∥ p k N ∥ \Vert p_k^N\Vert ∥ p k N ∥ ∥ p k N ∥ < 1 2 Δ k \Vert p_k^N\Vert<\frac{1}{2}\Delta_k ∥ p k N ∥ < 2 1 Δ k ∥ p k ∥ ≤ ∥ p k N ∥ + o ( ∥ p k N ∥ ) ≤ 2 ∥ p k N ∥ \Vert p_k\Vert\le\Vert p_k^N\Vert+o(\Vert p_k^N\Vert)\le2\Vert p_k^N\Vert ∥ p k ∥ ≤ ∥ p k N ∥ + o ( ∥ p k N ∥ ) ≤ 2 ∥ p k N ∥ ∥ p k N ∥ ≥ 1 2 Δ k \Vert p_k^N\Vert\ge\frac{1}{2}\Delta_k ∥ p k N ∥ ≥ 2 1 Δ k ∥ p k ∥ ≤ Δ k ≤ 2 ∥ p k N ∥ \Vert p_k\Vert\le\Delta_k\le2\Vert p_k^N\Vert ∥ p k ∥ ≤ Δ k ≤ 2 ∥ p k N ∥ ∥ p k ∥ ≤ 2 ∥ p k N ∥ ≤ 2 ∥ ∇ 2 f ( x k ) − 1 ∥ ∥ g k ∥ , \Vert p_k\Vert\le2\Vert p_k^N\Vert\le2\left\Vert\nabla^2f(x_k)^{-1}\right\Vert\Vert g_k\Vert, ∥ p k ∥ ≤ 2 ∥ p k N ∥ ≤ 2 ∥ ∥ ∇ 2 f ( x k ) − 1 ∥ ∥ ∥ g k ∥ , ∥ g k ∥ ≥ 1 2 ∥ p k ∥ / ∥ ∇ 2 f ( x k ) − 1 ∥ \Vert g_k\Vert\ge\frac{1}{2}\Vert p_k\Vert/\left\Vert\nabla^2f(x_k)^{-1}\right\Vert ∥ g k ∥ ≥ 2 1 ∥ p k ∥ / ∥ ∥ ∇ 2 f ( x k ) − 1 ∥ ∥ m k ( 0 ) − m k ( p k ) ≥ c 1 ∥ g k ∥ min ( Δ k , ∥ g k ∥ ∥ ∇ 2 f ( x k ) ∥ ) ≥ c 1 ∥ p k ∥ 2 ∥ ∇ 2 f ( x k ) − 1 ∥ min ( ∥ p k ∥ , ∥ p k ∥ 2 ∥ ∇ 2 f ( x k ) ∥ ∥ ∇ 2 f ( x k ) − 1 ∥ ) = c 1 ∥ p k ∥ 2 4 ∥ ∇ 2 f ( x k ) − 1 ∥ 2 ∥ ∇ 2 f ( x k ) ∥ . \begin{aligned}&m_k(0)-m_k(p_k)\\&\ge c_1\Vert g_k\Vert\min\left(\Delta_k,\frac{\Vert g_k\Vert}{\left\Vert\nabla^2f(x_k)\right\Vert}\right)\\&\ge c_1\frac{\Vert p_k\Vert}{2\left\Vert\nabla^2f(x_k)^{-1}\right\Vert}\min\left(\Vert p_k\Vert,\frac{\Vert p_k\Vert}{2\left\Vert\nabla^2f(x_k)\right\Vert\left\Vert\nabla^2f(x_k)^{-1}\right\Vert}\right)\\&=c_1\frac{\Vert p_k\Vert^2}{4\left\Vert\nabla^2f(x_k)^{-1}\right\Vert^2\left\Vert\nabla^2f(x_k)\right\Vert}.\end{aligned} m k ( 0 ) − m k ( p k ) ≥ c 1 ∥ g k ∥ min ( Δ k , ∥ ∇ 2 f ( x k ) ∥ ∥ g k ∥ ) ≥ c 1 2 ∥ ∇ 2 f ( x k ) − 1 ∥ ∥ p k ∥ min ( ∥ p k ∥ , 2 ∥ ∇ 2 f ( x k ) ∥ ∥ ∇ 2 f ( x k ) − 1 ∥ ∥ p k ∥ ) = c 1 4 ∥ ∇ 2 f ( x k ) − 1 ∥ 2 ∥ ∇ 2 f ( x k ) ∥ ∥ p k ∥ 2 . x k → x ∗ x_k\to x^* x k → x ∗ ∇ 2 f ( x ) \nabla^2f(x) ∇ 2 f ( x ) ∇ 2 f ( x ∗ ) \nabla^2f(x^*) ∇ 2 f ( x ∗ ) k k k c 1 4 ∥ ∇ 2 f ( x k ) − 1 ∥ 2 ∥ ∇ 2 f ( x k ) ∥ ≥ c 1 8 ∥ ∇ 2 f ( x ∗ ) − 1 ∥ 2 ∥ ∇ 2 f ( x ∗ ) ∥ ≜ c 3 , \frac{c_1}{4\left\Vert\nabla^2f(x_k)^{-1}\right\Vert^2\left\Vert\nabla^2f(x_k)\right\Vert}\ge\frac{c_1}{8\left\Vert\nabla^2f(x^*)^{-1}\right\Vert^2\left\Vert\nabla^2f(x^*)\right\Vert}\triangleq c_3, 4 ∥ ∇ 2 f ( x k ) − 1 ∥ 2 ∥ ∇ 2 f ( x k ) ∥ c 1 ≥ 8 ∥ ∇ 2 f ( x ∗ ) − 1 ∥ 2 ∥ ∇ 2 f ( x ∗ ) ∥ c 1 ≜ c 3 , c 3 > 0 c_3>0 c 3 > 0 m k ( 0 ) − m k ( p k ) ≥ c 3 ∥ p k ∥ 2 , k 充 分 大 . m_k(0)-m_k(p_k)\ge c_3\Vert p_k\Vert^2,\quad k充分大. m k ( 0 ) − m k ( p k ) ≥ c 3 ∥ p k ∥ 2 , k 充 分 大 . ∇ 2 f ( x ) \nabla^2f(x) ∇ 2 f ( x ) x ∗ x^* x ∗ ∣ ( f ( x k ) − f ( x k + p k ) ) − ( m k ( 0 ) − m k ( p k ) ) ∣ = ∣ 1 2 p k T ∇ 2 f ( x k ) p k − 1 2 ∫ 0 1 p k T ∇ 2 f ( x k + t p k ) p k d t ∣ ≤ L 4 ∥ p k ∥ 3 , \begin{aligned}&|(f(x_k)-f(x_k+p_k))-(m_k(0)-m_k(p_k))|\\&=\left|\frac{1}{2}p_k^T\nabla^2f(x_k)p_k-\frac{1}{2}\int_0^1p_k^T\nabla^2f(x_k+tp_k)p_k\,\mathrm{d}t\right|\\&\le\frac{L}{4}\Vert p_k\Vert^3,\end{aligned} ∣ ( f ( x k ) − f ( x k + p k ) ) − ( m k ( 0 ) − m k ( p k ) ) ∣ = ∣ ∣ ∣ ∣ 2 1 p k T ∇ 2 f ( x k ) p k − 2 1 ∫ 0 1 p k T ∇ 2 f ( x k + t p k ) p k d t ∣ ∣ ∣ ∣ ≤ 4 L ∥ p k ∥ 3 , L > 0 L>0 L > 0 ∇ 2 f ( ⋅ ) \nabla^2f(\cdot) ∇ 2 f ( ⋅ ) ρ k \rho_k ρ k k k k ∣ ρ k − 1 ∣ ≤ ∥ p k ∥ 3 ( L / 4 ) c 3 ∥ p k ∥ 2 = L 4 c 3 ∥ p k ∥ ≤ L 4 c 3 Δ k . |\rho_k-1|\le\frac{\Vert p_k\Vert^3(L/4)}{c_3\Vert p_k\Vert^2}=\frac{L}{4c_3}\Vert p_k\Vert\le\frac{L}{4c_3}\Delta_k. ∣ ρ k − 1 ∣ ≤ c 3 ∥ p k ∥ 2 ∥ p k ∥ 3 ( L / 4 ) = 4 c 3 L ∥ p k ∥ ≤ 4 c 3 L Δ k . ρ k < 1 4 \rho_k<\frac{1}{4} ρ k < 4 1 { Δ k } \{\Delta_k\} { Δ k } x k → x ∗ x_k\to x^* x k → x ∗ ∥ p k N ∥ → 0 ⇒ ∥ p k ∥ → 0 \Vert p_k^N\Vert\to0\Rightarrow\Vert p_k\Vert\to0 ∥ p k N ∥ → 0 ⇒ ∥ p k ∥ → 0 k k k ∥ p k N ∥ ≤ 1 2 Δ k \Vert p_k^N\Vert\le\frac{1}{2}\Delta_k ∥ p k N ∥ ≤ 2 1 Δ k 第三章 关于牛顿法的讨论, 我们有∥ x k + p k N − x ∗ ∥ = O ( ∥ x k − x ∗ ∥ ) , \Vert x_k+p_k^N-x^*\Vert=O(\Vert x_k-x^*\Vert), ∥ x k + p k N − x ∗ ∥ = O ( ∥ x k − x ∗ ∥ ) , ∥ p k N ∥ = O ( ∥ x k − x ∗ ∥ ) \Vert p_k^N\Vert=O(\Vert x_k-x^*\Vert) ∥ p k N ∥ = O ( ∥ x k − x ∗ ∥ ) ∥ x k + p k − x ∗ ∥ ≤ ∥ x k + p k N − x ∗ ∥ + ∥ p k N − p k ∥ = O ( ∥ x k − x ∗ ∥ 2 ) + o ( ∥ p k N ∥ ) = o ( ∥ x k − x ∗ ∥ ) . \begin{aligned}&\Vert x_k+p_k-x^*\Vert\\&\le\Vert x_k+p_k^N-x^*\Vert+\Vert p_k^N-p_k\Vert=O(\Vert x_k-x^*\Vert^2)+o(\Vert p_k^N\Vert)=o(\Vert x_k-x^*\Vert).\end{aligned} ∥ x k + p k − x ∗ ∥ ≤ ∥ x k + p k N − x ∗ ∥ + ∥ p k N − p k ∥ = O ( ∥ x k − x ∗ ∥ 2 ) + o ( ∥ p k N ∥ ) = o ( ∥ x k − x ∗ ∥ ) .
从牛顿法的讨论中立得: 若p k = p k N , k 充 分 大 p_k=p_k^N, k充分大 p k = p k N , k 充 分 大 { x k } \{x_k\} { x k } x ∗ x^* x ∗ 在定理8的假设下, B k = ∇ 2 f ( x k ) B_k=\nabla^2f(x_k) B k = ∇ 2 f ( x k ) .
在折线算法和二维子空间最小化算法中, 牛顿步p k N p_k^N p k N p k p_k p k p k N p_k^N p k N k k k p k = p k N p_k=p_k^N p k = p k N
在近似精确算法中, 若在算法2实施前先确认p k N p_k^N p k N p k N p_k^N p k N
正如我们在第二章 中提到过的, 数据的尺度将影响算法的效率. 当f f f ∥ D p ∥ ≤ Δ , \Vert Dp\Vert\le\Delta, ∥ D p ∥ ≤ Δ , Δ \Delta Δ min p ∈ R n m k ( p ) = f k + g k T p + 1 2 p T B k p , s . t . ∥ D p ∥ ≤ Δ k . \min_{p\in\mathbb{R}^n}m_k(p)=f_k+g_k^Tp+\frac{1}{2}p^TB_kp,\quad\mathrm{s.t.}\Vert Dp\Vert\le\Delta_k. p ∈ R n min m k ( p ) = f k + g k T p + 2 1 p T B k p , s . t . ∥ D p ∥ ≤ Δ k . f f f x x x i i i d i i d_{ii} d i i D D D D D D d i i d_{ii} d i i [ d l o , d h i ] [d_{lo},d_{hi}] [ d l o , d h i ] 0 < d l o < d h i < ∞ 0<d_{lo}<d_{hi}<\infty 0 < d l o < d h i < ∞
信赖域亦可用除欧式范数以外的范数定义, 例如∥ p ∥ 1 ≤ Δ k , ∥ p ∥ ∞ ≤ Δ k , \Vert p\Vert_1\le\Delta_k,\quad \Vert p\Vert_{\infty}\le\Delta_k, ∥ p ∥ 1 ≤ Δ k , ∥ p ∥ ∞ ≤ Δ k , ∥ D p ∥ 1 ≤ Δ k , ∥ D p ∥ ∞ ≤ Δ k . \Vert Dp\Vert_1\le\Delta_k,\quad \Vert Dp\Vert_{\infty}\le\Delta_k. ∥ D p ∥ 1 ≤ Δ k , ∥ D p ∥ ∞ ≤ Δ k . min x ∈ R n f ( x ) , s . t . x ≥ 0 , \min_{x\in\mathbb{R}^n}f(x),\quad\mathrm{s.t.}x\ge0, x ∈ R n min f ( x ) , s . t . x ≥ 0 , min p ∈ R n m k ( p ) = f k + g k T p + 1 2 p T B k p , s . t . x k + p ≥ 0 , ∥ p ∥ ≤ Δ k . \min_{p\in\mathbb{R}^n}m_k(p)=f_k+g_k^Tp+\frac{1}{2}p^TB_kp,\quad\mathrm{s.t.}x_k+p\ge0,\Vert p\Vert\le\Delta_k. p ∈ R n min m k ( p ) = f k + g k T p + 2 1 p T B k p , s . t . x k + p ≥ 0 , ∥ p ∥ ≤ Δ k . x k + p ≥ 0 , p ≥ − Δ k e , p ≤ Δ k e , x_k+p\ge0,\quad p\ge-\Delta_k e,\quad p\le\Delta_k e, x k + p ≥ 0 , p ≥ − Δ k e , p ≤ Δ k e , e e e B k B_k B k