将原问题替换为一系列子问题 , 在子问题中原本的约束被替换成加在目标函数上的项. 本章我们介绍属于此类的三种方法.
二次罚函数法the quadratic penalty method 是在目标函数上增加每个约束违反度平方的某个倍数. 这种方法比较简单、直观 . 尽管它有许多的缺陷 , 但实际中还是被经常使用 .
非光滑精确罚函数法the nonsmooth exact penalty methods 是用一个 (而不是一系列)无约束问题替代原本的约束问题. 使用这种罚函数, 我们可以仅利用一次无约束极小化找到解, 但随之而来, 非光滑性可能会制造一些麻烦. 此类的常见罚函数比如l 1 l_1 l 1
乘子法the methods of multipliers 或增广Lagrange函数法the augmented Lagrangian method 是另一种精确罚函数法, 其中显式估计了Lagrange乘子, 避免了二次罚函数带来的病态问题 .
对数障碍函数法log-barrier method 使用对数项来防止迭代点过于接近可行域边界. 此法是非线性规划内点法的部分基础. 我们放在第十九章讲述.
考虑使用单一个 函数代替约束优化问题, 其中那个函数由以下组成:
约束优化问题的目标函数 , 加上
对每个约束的一个附加项 , 其在当前点违反约束时为正, 否则取0.
许多方法都会通过定义一系列这样的罚函数求解问题, 其中罚项乘上了一个正系数(惩罚因子). 此系数越大, 我们对违反约束的现象越不能容忍, 从而越强制罚函数的极小点靠近约束问题的可行域 .
此类最简单的罚函数为二次罚函数(又称Courant罚函数) , 其中的罚项记为约束违反度的平方. 我们先在等式约束问题下讨论这一方法.min x f ( x ) , s u b j e c t  t o  c i ( x ) = 0 , i ∈ E . \min\limits_xf(x),\quad\mathrm{subject\,to\,}c_i(x)=0,\quad i\in\mathcal{E}. x min f ( x ) , s u b j e c t t o c i ( x ) = 0 , i ∈ E . Q ( x ; μ ) = d e f f ( x ) + μ 2 ∑ i ∈ E c i 2 ( x ) , Q(x;\mu)\xlongequal{def}f(x)+\frac{\mu}{2}\sum_{i\in\mathcal{E}}c_i^2(x), Q ( x ; μ ) d e f f ( x ) + 2 μ i ∈ E ∑ c i 2 ( x ) , μ > 0 \mu>0 μ > 0 惩罚因子(penalty parameter) . 当μ \mu μ ∞ \infty ∞ { μ k } : μ ↑ ∞ , k → ∞ \{\mu_k\}:\mu\uparrow\infty,k\to\infty { μ k } : μ ↑ ∞ , k → ∞ k k k Q ( x ; μ k ) Q(x;\mu_k) Q ( x ; μ k ) x k x_k x k x k x_k x k x k x_k x k x k − 1 , x k − 2 x_{k-1},x_{k-2} x k − 1 , x k − 2 初始的迭代点 . 对于合适的{ μ k } \{\mu_k\} { μ k }
例1 考虑以下等式约束问题min x 1 + x 2 , s u b j e c t  t o  x 1 2 + x 2 2 − 2 = 0 , \min x_1+x_2,\quad\mathrm{subject\,to\,}x_1^2+x_2^2-2=0, min x 1 + x 2 , s u b j e c t t o x 1 2 + x 2 2 − 2 = 0 , ( − 1 , − 1 ) T (-1,-1)^T ( − 1 , − 1 ) T Q ( x ; μ ) = x 1 + x 2 + μ 2 ( x 1 2 + x 2 2 − 2 ) 2 . Q(x;\mu)=x_1+x_2+\frac{\mu}{2}(x_1^2+x_2^2-2)^2. Q ( x ; μ ) = x 1 + x 2 + 2 μ ( x 1 2 + x 2 2 − 2 ) 2 . μ = 1 \mu=1 μ = 1 Q Q Q Q Q Q ( − 1.1 , − 1.1 ) T (-1.1,-1.1)^T ( − 1 . 1 , − 1 . 1 ) T ( 0.3 , 0.3 ) T (0.3,0.3)^T ( 0 . 3 , 0 . 3 ) T
而下图则对应μ = 10 \mu=10 μ = 1 0 x 1 2 + x 2 2 = 2 x_1^2+x_2^2=2 x 1 2 + x 2 2 = 2 Q Q Q ( 0 , 0 ) T (0,0)^T ( 0 , 0 ) T Q Q Q ∞ \infty ∞
但实际情形一般没有例1中那么良态. 对于一给定的μ \mu μ min − 5 x 1 2 + x 2 2 , s u b j e c t  t o  x 1 = 1 , \min -5x_1^2+x_2^2,\quad\mathrm{subject\,to\,}x_1=1, min − 5 x 1 2 + x 2 2 , s u b j e c t t o x 1 = 1 , ( 1 , 0 ) T (1,0)^T ( 1 , 0 ) T
对于一般的约束优化问题min x f ( x ) , s u b j e c t  t o  c i ( x ) = 0 , i ∈ E , c i ( x ) ≥ 0 , i ∈ I . \min\limits_xf(x),\quad\mathrm{subject\,to\,}c_i(x)=0,i\in\mathcal{E},\quad c_i(x)\ge0,i\in\mathcal{I}. x min f ( x ) , s u b j e c t t o c i ( x ) = 0 , i ∈ E , c i ( x ) ≥ 0 , i ∈ I . Q ( x ; μ ) = d e f f ( x ) + μ 2 ∑ i ∈ E c i 2 ( x ) + μ 2 ∑ i ∈ I ( [ c i ( x ) ] − ) 2 , Q(x;\mu)\xlongequal{def}f(x)+\frac{\mu}{2}\sum_{i\in\mathcal{E}}c_i^2(x)+\frac{\mu}{2}\sum_{i\in\mathcal{I}}\left([c_i(x)]^-\right)^2, Q ( x ; μ ) d e f f ( x ) + 2 μ i ∈ E ∑ c i 2 ( x ) + 2 μ i ∈ I ∑ ( [ c i ( x ) ] − ) 2 , [ y ] − = max ( − y , 0 ) [y]^-=\max(-y,0) [ y ] − = max ( − y , 0 ) 此时Q Q Q . 例如若有一个不等式约束为x 1 ≥ 0 x_1\ge0 x 1 ≥ 0 min ( 0 , x 1 ) 2 \min(0,x_1)^2 min ( 0 , x 1 ) 2 Q Q Q
基于二次罚函数的一般算法框架如下.
框架1 (Quadratic Penalty Method)μ 0 > 0 \mu_0>0 μ 0 > 0 { τ k } \{\tau_k\} { τ k } τ k → 0 \tau_k\to0 τ k → 0 x 0 s x_0^s x 0 s k = 0 , 1 , 2 , … k=0,1,2,\ldots k = 0 , 1 , 2 , … \quad\quad x k x_k x k Q ( ⋅ ; μ k ) Q(\cdot;\mu_k) Q ( ⋅ ; μ k ) x k s x_k^s x k s ∥ ∇ x Q ( x ; μ k ) ∥ ≤ τ k \Vert\nabla_xQ(x;\mu_k)\Vert\le\tau_k ∥ ∇ x Q ( x ; μ k ) ∥ ≤ τ k \quad\quad \quad\quad\quad\quad x k x_k x k \quad\quad \quad\quad μ k + 1 > μ k \mu_{k+1}>\mu_k μ k + 1 > μ k \quad\quad x k + 1 s x_{k+1}^s x k + 1 s
我们可以基于在每次迭代极小化罚函数的困难度自适应 地选取惩罚因子序列{ μ k } \{\mu_k\} { μ k }
当极小化Q ( x ; μ k ) Q(x;\mu_k) Q ( x ; μ k ) k k k μ k \mu_k μ k μ k + 1 \mu_{k+1} μ k + 1 μ k + 1 = 1.5 μ k \mu_{k+1}=1.5\mu_k μ k + 1 = 1 . 5 μ k
若我们比较容易就能找到Q ( x ; μ k ) Q(x;\mu_k) Q ( x ; μ k ) μ k + 1 = 10 μ k \mu_{k+1}=10\mu_k μ k + 1 = 1 0 μ k
框架1的收敛理论给予了非负序列{ τ k } \{\tau_k\} { τ k } τ k → 0 \tau_k\to0 τ k → 0
正如之前讨论过的, 我们无法保证∥ ∇ x Q ( x ; μ k ) ∥ ≤ τ k \Vert\nabla_xQ(x;\mu_k)\Vert\le\tau_k ∥ ∇ x Q ( x ; μ k ) ∥ ≤ τ k 当约束违反度下降不够快, 或者当迭代点趋向于发散时, 增大惩罚因子(或存储初始点)的防护措施 .
当只有等式约束时, Q ( x ; μ k ) Q(x;\mu_k) Q ( x ; μ k ) x k x_k x k μ k \mu_k μ k Q ( x ; μ k ) Q(x;\mu_k) Q ( x ; μ k ) 条件数变大 ), 除非我们使用特殊的手段计算搜索方向.
一方面, Hessian阵∇ x x 2 Q ( x ; μ k ) \nabla_{xx}^2Q(x;\mu_k) ∇ x x 2 Q ( x ; μ k ) μ k \mu_k μ k
在我们求解线性系统计算牛顿步时, 病态的∇ x x 2 Q ( x ; μ k ) \nabla_{xx}^2Q(x;\mu_k) ∇ x x 2 Q ( x ; μ k )
即使x x x Q ( ⋅ ; μ k ) Q(\cdot;\mu_k) Q ( ⋅ ; μ k ) Q ( x ; μ k ) Q(x;\mu_k) Q ( x ; μ k ) x x x Q Q Q Q ( x ; μ k ) Q(x;\mu_k) Q ( x ; μ k ) x k + 1 s x_{k+1}^s x k + 1 s x k + 1 s = x k x_{k+1}^s=x_k x k + 1 s = x k μ k + 1 \mu_{k+1} μ k + 1 μ k \mu_k μ k
我们在下面两个定理里介绍二次罚函数方法的一些收敛性质. 我们将讨论限制在等式约束问题.
对于第一个结果, 我们假设对每个μ k \mu_k μ k Q ( x ; μ k ) Q(x;\mu_k) Q ( x ; μ k )
定理1 设x k x_k x k Q ( x ; μ k ) Q(x;\mu_k) Q ( x ; μ k ) μ k ↑ ∞ \mu_k\uparrow\infty μ k ↑ ∞ { x k } \{x_k\} { x k }
证明: 令x ˉ \bar{x} x ˉ f ( x ˉ ) ≤ f ( x ) , ∀ x : c i ( x ) = 0 , i ∈ E . f(\bar{x})\le f(x),\quad\forall x:c_i(x)=0,i\in\mathcal{E}. f ( x ˉ ) ≤ f ( x ) , ∀ x : c i ( x ) = 0 , i ∈ E . k k k x k x_k x k Q ( ⋅ ; μ k ) Q(\cdot;\mu_k) Q ( ⋅ ; μ k ) Q ( x k ; μ k ) ≤ Q ( x ˉ ; μ k ) Q(x_k;\mu_k)\le Q(\bar{x};\mu_k) Q ( x k ; μ k ) ≤ Q ( x ˉ ; μ k ) f ( x k ) + μ k 2 ∑ i ∈ E c i 2 ( x k ) ≤ f ( x ˉ ) + μ k 2 ∑ i ∈ E c i 2 ( x ˉ ) = f ( x ˉ ) . f(x_k)+\frac{\mu_k}{2}\sum_{i\in\mathcal{E}}c_i^2(x_k)\le f(\bar{x})+\frac{\mu_k}{2}\sum_{i\in\mathcal{E}}c_i^2(\bar{x})=f(\bar{x}). f ( x k ) + 2 μ k i ∈ E ∑ c i 2 ( x k ) ≤ f ( x ˉ ) + 2 μ k i ∈ E ∑ c i 2 ( x ˉ ) = f ( x ˉ ) . ∑ i ∈ E c i 2 ( x k ) ≤ 2 μ k [ f ( x ˉ ) − f ( x k ) ] . \sum_{i\in\mathcal{E}}c_i^2(x_k)\le\frac{2}{\mu_k}[f(\bar{x})-f(x_k)]. i ∈ E ∑ c i 2 ( x k ) ≤ μ k 2 [ f ( x ˉ ) − f ( x k ) ] . x ∗ x^* x ∗ { x k } \{x_k\} { x k } { 1 , 2 , … , n , … } \{1,2,\ldots,n,\ldots\} { 1 , 2 , … , n , … } K \mathcal{K} K lim k → K x k = x ∗ . \lim_{k\to\mathcal{K}}x_k=x^*. k → K lim x k = x ∗ . k → ∞ , k ∈ K k\to\infty,k\in\mathcal{K} k → ∞ , k ∈ K ∑ i ∈ E c i 2 ( x ∗ ) = lim k ∈ K ∑ i ∈ E c i 2 ( x k ) ≤ lim k ∈ K 2 μ k [ f ( x ˉ ) − f ( x k ) ] = 0. \sum_{i\in\mathcal{E}}c_i^2(x^*)=\lim_{k\in\mathcal{K}}\sum_{i\in\mathcal{E}}c_i^2(x_k)\le\lim_{k\in\mathcal{K}}\frac{2}{\mu_k}[f(\bar{x})-f(x_k)]=0. i ∈ E ∑ c i 2 ( x ∗ ) = k ∈ K lim i ∈ E ∑ c i 2 ( x k ) ≤ k ∈ K lim μ k 2 [ f ( x ˉ ) − f ( x k ) ] = 0 . c i ( x ∗ ) = 0 , i ∈ E c_i(x^*)=0,i\in\mathcal{E} c i ( x ∗ ) = 0 , i ∈ E x ∗ x^* x ∗ f ( x ∗ ) ≤ f ( x ∗ ) + lim k ∈ K μ k 2 ∑ i ∈ E c i 2 ( x k ) ≤ f ( x ˉ ) . f(x^*)\le f(x^*)+\lim_{k\in\mathcal{K}}\frac{\mu_k}{2}\sum_{i\in\mathcal{E}}c_i^2(x_k)\le f(\bar{x}). f ( x ∗ ) ≤ f ( x ∗ ) + k ∈ K lim 2 μ k i ∈ E ∑ c i 2 ( x k ) ≤ f ( x ˉ ) . x ∗ x^* x ∗ x ˉ \bar{x} x ˉ x ∗ x^* x ∗
此结论对带不等式的问题也成立. 但由于需要我们求每个子问题的全局极小点, 因此此性质一般并不能成立 . 下一结论则允许我们不精确(但精确度要提升)极小化Q ( ⋅ ; μ k ) Q(\cdot;\mu_k) Q ( ⋅ ; μ k ) { x k } \{x_k\} { x k } − μ k c i ( x k ) -\mu_kc_i(x_k) − μ k c i ( x k ) λ i ∗ \lambda_i^* λ i ∗
为建立定理, 我们需乐观地 假设对所有k k k ∥ ∇ x Q ( x ; μ k ) ∥ ≤ τ k \Vert\nabla_xQ(x;\mu_k)\Vert\le\tau_k ∥ ∇ x Q ( x ; μ k ) ∥ ≤ τ k
定理2 设框架1中的容忍限序列和惩罚因子序列满足τ k → 0 , μ k ↑ ∞ \tau_k\to0,\mu_k\uparrow\infty τ k → 0 , μ k ↑ ∞ { x k } \{x_k\} { x k } x ∗ x^* x ∗ ∥ c ( x ) ∥ 2 \Vert c(x)\Vert^2 ∥ c ( x ) ∥ 2 x ∗ x^* x ∗ ∇ c i ( x ∗ ) \nabla c_i(x^*) ∇ c i ( x ∗ ) x ∗ x^* x ∗ x ∗ x^* x ∗ ∀ K : lim k ∈ K x k = x ∗ \forall\mathcal{K}:\lim_{k\in\mathcal{K}}x_k=x^* ∀ K : lim k ∈ K x k = x ∗ lim k ∈ K − μ k c i ( x k ) = λ i ∗ , ∀ i ∈ E , \lim_{k\in\mathcal{K}}-\mu_kc_i(x_k)=\lambda_i^*,\quad\forall i\in\mathcal{E}, k ∈ K lim − μ k c i ( x k ) = λ i ∗ , ∀ i ∈ E , λ ∗ \lambda^* λ ∗
证明: 对Q ( x ; μ k ) Q(x;\mu_k) Q ( x ; μ k ) ∇ x Q ( x k ; μ k ) = ∇ f ( x k ) + ∑ i ∈ E μ k c i ( x k ) ∇ c i ( x k ) , \nabla_xQ(x_k;\mu_k)=\nabla f(x_k)+\sum_{i\in\mathcal{E}}\mu_kc_i(x_k)\nabla c_i(x_k), ∇ x Q ( x k ; μ k ) = ∇ f ( x k ) + i ∈ E ∑ μ k c i ( x k ) ∇ c i ( x k ) , ∥ ∇ f ( x k ) + ∑ i ∈ E μ k c i ( x k ) ∇ c i ( x k ) ∥ ≤ τ k . \left\Vert\nabla f(x_k)+\sum_{i\in\mathcal{E}}\mu_kc_i(x_k)\nabla c_i(x_k)\right\Vert\le\tau_k. ∥ ∥ ∥ ∥ ∥ ∇ f ( x k ) + i ∈ E ∑ μ k c i ( x k ) ∇ c i ( x k ) ∥ ∥ ∥ ∥ ∥ ≤ τ k . ∥ ∑ i ∈ E c i ( x k ) ∇ c i ( x k ) ∥ ≤ 1 μ k [ τ k + ∥ ∇ f ( x k ) ∥ ] . \left\Vert\sum_{i\in\mathcal{E}}c_i(x_k)\nabla c_i(x_k)\right\Vert\le\frac{1}{\mu_k}[\tau_k+\Vert\nabla f(x_k)\Vert]. ∥ ∥ ∥ ∥ ∥ i ∈ E ∑ c i ( x k ) ∇ c i ( x k ) ∥ ∥ ∥ ∥ ∥ ≤ μ k 1 [ τ k + ∥ ∇ f ( x k ) ∥ ] . x ∗ x^* x ∗ K \mathcal{K} K lim k ∈ K x k = x ∗ \lim_{k\in\mathcal{K}}x_k=x^* lim k ∈ K x k = x ∗ k → ∞ , k ∈ K k\to\infty,k\in\mathcal{K} k → ∞ , k ∈ K ∑ i ∈ E c i ( x ∗ ) ∇ c i ( x ∗ ) = 0. \sum_{i\in\mathcal{E}}c_i(x^*)\nabla c_i(x^*)=0. i ∈ E ∑ c i ( x ∗ ) ∇ c i ( x ∗ ) = 0 . c i ( x ∗ ) ≠ 0 c_i(x^*)\ne0 c i ( x ∗ ) ̸ = 0 ∇ c i ( x ∗ ) \nabla c_i(x^*) ∇ c i ( x ∗ ) x ∗ x^* x ∗ ∥ c ( x ) ∥ 2 \Vert c(x)\Vert^2 ∥ c ( x ) ∥ 2
若∇ c i ( x ∗ ) \nabla c_i(x^*) ∇ c i ( x ∗ ) c i ( x ∗ ) = 0 , ∀ i ∈ E c_i(x^*)=0,\forall i\in\mathcal{E} c i ( x ∗ ) = 0 , ∀ i ∈ E x ∗ x^* x ∗ A A A − μ k c ( x k ) -\mu_kc(x_k) − μ k c ( x k ) λ k \lambda_k λ k A ( x k ) T λ k = ∇ f ( x k ) − ∇ x Q ( x k ; μ k ) , ∥ ∇ x Q ( x k ; μ k ) ∥ ≤ τ k . A(x_k)^T\lambda_k=\nabla f(x_k)-\nabla_xQ(x_k;\mu_k),\quad\Vert\nabla_xQ(x_k;\mu_k)\Vert\le\tau_k. A ( x k ) T λ k = ∇ f ( x k ) − ∇ x Q ( x k ; μ k ) , ∥ ∇ x Q ( x k ; μ k ) ∥ ≤ τ k . k ∈ K k\in\mathcal{K} k ∈ K A ( x k ) A(x_k) A ( x k ) A ( x k ) A ( x k ) T A(x_k)A(x_k)^T A ( x k ) A ( x k ) T A ( x k ) A(x_k) A ( x k ) λ k = [ A ( x k ) A ( x k ) T ] − 1 A ( x k ) [ ∇ f ( x k ) − ∇ x Q ( x k ; μ k ) ] . \lambda_k=\left[A(x_k)A(x_k)^T\right]^{-1}A(x_k)[\nabla f(x_k)-\nabla_xQ(x_k;\mu_k)]. λ k = [ A ( x k ) A ( x k ) T ] − 1 A ( x k ) [ ∇ f ( x k ) − ∇ x Q ( x k ; μ k ) ] . k → ∞ , k ∈ K k\to\infty,k\in\mathcal{K} k → ∞ , k ∈ K lim k ∈ K λ k = λ ∗ = [ A ( x ∗ ) A ( x ∗ ) T ] − 1 A ( x ∗ ) ∇ f ( x ∗ ) . \lim_{k\in\mathcal{K}}\lambda_k=\lambda^*=\left[A(x^*)A(x^*)^T\right]^{-1}A(x^*)\nabla f(x^*). k ∈ K lim λ k = λ ∗ = [ A ( x ∗ ) A ( x ∗ ) T ] − 1 A ( x ∗ ) ∇ f ( x ∗ ) . ∇ f ( x ∗ ) − A ( x ∗ ) T λ ∗ = 0 , \nabla f(x^*)-A(x^*)^T\lambda^*=0, ∇ f ( x ∗ ) − A ( x ∗ ) T λ ∗ = 0 , λ ∗ \lambda^* λ ∗ x ∗ x^* x ∗ λ ∗ \lambda^* λ ∗
需重复强调的是, 若聚点x ∗ x^* x ∗ ∥ c ( x ) ∥ 2 \Vert c(x)\Vert^2 ∥ c ( x ) ∥ 2 第十一章 中讨论非线性方程组的情形. 在原问题不可行时, 我们往往可以观察到二次罚函数算法收敛于∥ c ( x ) ∥ 2 \Vert c(x)\Vert^2 ∥ c ( x ) ∥ 2
若考虑不等式约束, 则x ∗ x^* x ∗ ∥ [ c ( x ) ] − ∥ 2 \Vert [c(x)]^-\Vert^2 ∥ [ c ( x ) ] − ∥ 2 [ c ( x ) ] − [c(x)]^- [ c ( x ) ] − [ c ( x ) ] i − = { c i ( x ) , i ∈ E , [ c i ( x ) ] − , i ∈ I . [c(x)]_i^-=\left\{\begin{array}{ll}c_i(x), & i\in\mathcal{E},\\ [c_i(x)]^-, & i\in\mathcal{I}.\end{array}\right. [ c ( x ) ] i − = { c i ( x ) , [ c i ( x ) ] − , i ∈ E , i ∈ I .
我们现在来验证Hessian阵∇ x x 2 Q ( x ; μ k ) \nabla^2_{xx}Q(x;\mu_k) ∇ x x 2 Q ( x ; μ k )
这里的Hessian阵为∇ x x 2 Q ( x ; μ k ) = ∇ 2 f ( x ) + ∑ i ∈ E μ k c i ( x ) ∇ 2 c i ( x ) + μ k A ( x ) T A ( x ) . \nabla^2_{xx}Q(x;\mu_k)=\nabla^2f(x)+\sum_{i\in\mathcal{E}}\mu_kc_i(x)\nabla^2c_i(x)+\mu_kA(x)^TA(x). ∇ x x 2 Q ( x ; μ k ) = ∇ 2 f ( x ) + i ∈ E ∑ μ k c i ( x ) ∇ 2 c i ( x ) + μ k A ( x ) T A ( x ) . x x x Q ( ⋅ ; μ k ) Q(\cdot;\mu_k) Q ( ⋅ ; μ k ) ∇ x x 2 Q ( x ; μ k ) ≈ ∇ x x 2 L ( x , λ ∗ ) + μ k A ( x ) T A ( x ) . \nabla^2_{xx}Q(x;\mu_k)\approx\nabla^2_{xx}\mathcal{L}(x,\lambda^*)+\mu_kA(x)^TA(x). ∇ x x 2 Q ( x ; μ k ) ≈ ∇ x x 2 L ( x , λ ∗ ) + μ k A ( x ) T A ( x ) . ∇ x x 2 Q ( x ; μ k ) \nabla^2_{xx}Q(x;\mu_k) ∇ x x 2 Q ( x ; μ k )
(Lagrange项)独立于μ k \mu_k μ k
秩为∣ E ∣ |\mathcal{E}| ∣ E ∣ μ k \mu_k μ k
由于约束的数量∣ E ∣ |\mathcal{E}| ∣ E ∣ n n n 某些特征值会趋近常数, 而其他的则与μ k \mu_k μ k . 因μ k → ∞ \mu_k\to\infty μ k → ∞ ∇ x x 2 Q ( x ; μ k ) \nabla^2_{xx}Q(x;\mu_k) ∇ x x 2 Q ( x ; μ k )
Hessian病态的一个后果是, 以下牛顿步的计算可能会不够精确:∇ x x 2 Q ( x ; μ k ) p = − ∇ x Q ( x ; μ k ) . \nabla^2_{xx}Q(x;\mu_k)p=-\nabla_xQ(x;\mu_k). ∇ x x 2 Q ( x ; μ k ) p = − ∇ x Q ( x ; μ k ) . p p p
不过, 我们可以重构 以上牛顿方程以避免病态. 引入一个新的变量ζ : = μ A ( x ) p \zeta:=\mu A(x)p ζ : = μ A ( x ) p p p p [ ∇ 2 f ( x ) + ∑ i ∈ E μ k c i ( x ) ∇ 2 c i ( x ) A ( x ) T A ( x ) − ( 1 / μ k ) I ] [ p ζ ] = [ − ∇ x Q ( x ; μ k ) 0 ] . \begin{bmatrix}\nabla^2f(x)+\sum\limits_{i\in\mathcal{E}}\mu_kc_i(x)\nabla^2c_i(x) & A(x)^T\\A(x) & -(1/\mu_k)I\end{bmatrix}\begin{bmatrix}p\\\zeta\end{bmatrix}=\begin{bmatrix}-\nabla_xQ(x;\mu_k)\\0\end{bmatrix}. [ ∇ 2 f ( x ) + i ∈ E ∑ μ k c i ( x ) ∇ 2 c i ( x ) A ( x ) A ( x ) T − ( 1 / μ k ) I ] [ p ζ ] = [ − ∇ x Q ( x ; μ k ) 0 ] . x x x x ∗ x^* x ∗ μ k \mu_k μ k μ k → ∞ \mu_k\to\infty μ k → ∞ [ ∇ x x 2 L ( x , λ ∗ ) A ( x ) T A ( x ) O ] , \begin{bmatrix}\nabla^2_{xx}\mathcal{L}(x,\lambda^*) & A(x)^T\\A(x) & O\end{bmatrix}, [ ∇ x x 2 L ( x , λ ∗ ) A ( x ) A ( x ) T O ] , ∇ x x 2 L ( x , λ ∗ ) \nabla^2_{xx}\mathcal{L}(x,\lambda^*) ∇ x x 2 L ( x , λ ∗ ) [ ∇ x x 2 L ( x , λ ∗ ) A ( x ) T A ( x ) O ] ∼ [ ∇ x x 2 L ( x , λ ∗ ) A ( x ) T O − A ( x ) [ ∇ x x 2 L ( x , λ ∗ ) ] − 1 A ( x ) T ] , \begin{bmatrix}\nabla^2_{xx}\mathcal{L}(x,\lambda^*) & A(x)^T\\A(x) & O\end{bmatrix}\sim\begin{bmatrix}\nabla^2_{xx}\mathcal{L}(x,\lambda^*) & A(x)^T\\O & -A(x)\left[\nabla^2_{xx}\mathcal{L}(x,\lambda^*)\right]^{-1}A(x)^T\end{bmatrix}, [ ∇ x x 2 L ( x , λ ∗ ) A ( x ) A ( x ) T O ] ∼ [ ∇ x x 2 L ( x , λ ∗ ) O A ( x ) T − A ( x ) [ ∇ x x 2 L ( x , λ ∗ ) ] − 1 A ( x ) T ] , 良态重构(well conditioned reformulation) . 不过, 由于∇ 2 f ( x ) + ∑ i ∈ E μ k c i ( x ) ∇ 2 c i ( x ) \nabla^2f(x)+\sum_{i\in\mathcal{E}}\mu_kc_i(x)\nabla^2c_i(x) ∇ 2 f ( x ) + ∑ i ∈ E μ k c i ( x ) ∇ 2 c i ( x ) ∇ x x 2 L ( x , λ ∗ ) \nabla_{xx}^2\mathcal{L}(x,\lambda^*) ∇ x x 2 L ( x , λ ∗ ) μ k c i ( x ) \mu_kc_i(x) μ k c i ( x ) − λ i ∗ -\lambda_i^* − λ i ∗ p p p Q ( ⋅ ; μ k ) Q(\cdot;\mu_k) Q ( ⋅ ; μ k )
如果要通过求解重构的问题计算迭代步, 我们需要求解一个n + ∣ E ∣ n+|\mathcal{E}| n + ∣ E ∣ n n n
当μ k \mu_k μ k SQP迭代步的正则化 , 其中的− ( 1 / μ k ) I -(1/\mu_k)I − ( 1 / μ k ) I A ( x ) A(x) A ( x )
当μ k \mu_k μ k 不能较好地满足约束的线性化 . 我们是不希望这种情况发生的, 因为这样一来迭代步就不能朝着可行性改进多少, 从而全局表现不够高效.
若{ μ k } \{\mu_k\} { μ k } 丧失线性化满足时获得超线性收敛速度的机会 . 具体讨论可见下一章.
总之, 重构系统可以视作无约束极小在二次惩罚函数Q ( ⋅ ; μ k ) Q(\cdot;\mu_k) Q ( ⋅ ; μ k )
有些罚函数是精确的(exact) , 即对于特定的惩罚因子, 仅做一次极小化即可得到非线性规划问题的精确解. 这一性质是很吸引人的, 因为它使得罚函数方法不依赖于惩罚因子的更新策略 . 第1节介绍的二次罚函数并不是精确的, 这是因为对任何μ \mu μ l 1 l_1 l 1 ϕ 1 ( x ; μ ) = f ( x ) + μ ∑ i ∈ E ∣ c i ( x ) ∣ + μ ∑ i ∈ I [ c i ( x ) ] − . \phi_1(x;\mu)=f(x)+\mu\sum_{i\in\mathcal{E}}|c_i(x)|+\mu\sum_{i\in\mathcal{I}}[c_i(x)]^-. ϕ 1 ( x ; μ ) = f ( x ) + μ i ∈ E ∑ ∣ c i ( x ) ∣ + μ i ∈ I ∑ [ c i ( x ) ] − . μ \mu μ l 1 l_1 l 1 ϕ 1 ( x ; μ ) \phi_1(x;\mu) ϕ 1 ( x ; μ ) x x x
下面的定理揭示了l 1 l_1 l 1
定理3 设x ∗ x^* x ∗ λ i ∗ , i ∈ E ∪ I \lambda_i^*,i\in\mathcal{E}\cup\mathcal{I} λ i ∗ , i ∈ E ∪ I x ∗ x^* x ∗ ϕ 1 ( x ; μ ) , ∀ μ > μ ∗ \phi_1(x;\mu),\forall\mu>\mu^* ϕ 1 ( x ; μ ) , ∀ μ > μ ∗ μ ∗ = ∥ λ ∗ ∥ ∞ = max i ∈ E ∪ I ∣ λ i ∗ ∣ . \mu^*=\Vert\lambda^*\Vert_{\infty}=\max_{i\in\mathcal{E}\cup\mathcal{I}}|\lambda_i^*|. μ ∗ = ∥ λ ∗ ∥ ∞ = i ∈ E ∪ I max ∣ λ i ∗ ∣ . x ∗ x^* x ∗ μ > μ ∗ \mu>\mu^* μ > μ ∗ x ∗ x^* x ∗ ϕ 1 ( x ; μ ) \phi_1(x;\mu) ϕ 1 ( x ; μ )
证明可见 Han S P和Mangasarian O L1
粗略地说, 在非线性规划的解x ∗ x^* x ∗ ϕ 1 ( x ∗ ; μ ) = f ( x ∗ ) \phi_1(x^*;\mu)=f(x^*) ϕ 1 ( x ∗ ; μ ) = f ( x ∗ ) ϕ ( ⋅ ; μ ) \phi(\cdot;\mu) ϕ ( ⋅ ; μ ) x ∗ x^* x ∗
例2 考虑如下单变量问题:min x , s u b j e c t  t o  x ≥ 1 , \min x,\quad\mathrm{subject\,to\,}x\ge1, min x , s u b j e c t t o x ≥ 1 , x ∗ = 1 x^*=1 x ∗ = 1 ϕ 1 ( x ; μ ) = x + μ [ x − 1 ] − = { ( 1 − μ ) x + μ , x ≤ 1 , x x > 1. \phi_1(x;\mu)=x+\mu[x-1]^-=\left\{\begin{array}{ll}(1-\mu)x+\mu, & x\le1,\\x & x>1.\end{array}\right. ϕ 1 ( x ; μ ) = x + μ [ x − 1 ] − = { ( 1 − μ ) x + μ , x x ≤ 1 , x > 1 . μ > 1 \mu>1 μ > 1 x ∗ = 1 x^*=1 x ∗ = 1 μ < 1 \mu<1 μ < 1
由于罚函数法通过直接极小化罚函数起作用, 所以我们需要刻画ϕ 1 \phi_1 ϕ 1 ϕ 1 \phi_1 ϕ 1 方向导数 D ( ϕ 1 ( x ; μ ) ; p ) D(\phi_1(x;\mu);p) D ( ϕ 1 ( x ; μ ) ; p )
定义1 一点x ^ ∈ R n \hat{x}\in\mathbb{R}^n x ^ ∈ R n ϕ 1 ( x ; μ ) \phi_1(x;\mu) ϕ 1 ( x ; μ ) D ( ϕ 1 ( x ^ ; μ ) ; p ) ≥ 0 , ∀ p ∈ R n . D(\phi_1(\hat{x};\mu);p)\ge0,\quad\forall p\in\mathbb{R}^n. D ( ϕ 1 ( x ^ ; μ ) ; p ) ≥ 0 , ∀ p ∈ R n . x ^ \hat{x} x ^ 不可行度量(the measure of infeasibility) h ( x ) = ∑ i ∈ E ∣ c i ( x ) ∣ + ∑ i ∈ I [ c i ( x ) ] − h(x)=\sum_{i\in\mathcal{E}}|c_i(x)|+\sum_{i\in\mathcal{I}}[c_i(x)]^- h ( x ) = i ∈ E ∑ ∣ c i ( x ) ∣ + i ∈ I ∑ [ c i ( x ) ] − D ( h ( x ^ ) ; p ) ≥ 0 , ∀ p ∈ R n D(h(\hat{x});p)\ge0,\forall p\in\mathbb{R}^n D ( h ( x ^ ) ; p ) ≥ 0 , ∀ p ∈ R n h h h 不可行稳定点(infeasible stationary point) .
对于例2中的函数, 我们在x ∗ = 1 x^*=1 x ∗ = 1 D ( ϕ 1 ( x ∗ ; μ ) ; p ) = { p , p ≥ 0 , ( 1 − μ ) p , p < 0. D(\phi_1(x^*;\mu);p)=\left\{\begin{array}{ll}p, & p\ge0,\\(1-\mu)p, & p<0.\end{array}\right. D ( ϕ 1 ( x ∗ ; μ ) ; p ) = { p , ( 1 − μ ) p , p ≥ 0 , p < 0 . μ > 1 \mu>1 μ > 1 D ( ϕ 1 ( x ∗ ; μ ) ; p ) ≥ 0 , ∀ p ∈ R D(\phi_1(x^*;\mu);p)\ge0,\forall p\in\mathbb{R} D ( ϕ 1 ( x ∗ ; μ ) ; p ) ≥ 0 , ∀ p ∈ R
下面的结论则在一定程度上弥补了定理3的不足, 即证明了反方向: 在一定条件下, ϕ 1 ( x ; μ ) \phi_1(x;\mu) ϕ 1 ( x ; μ )
定理4 设x ^ \hat{x} x ^ ϕ 1 ( x ; μ ) , ∀ μ > μ ^ > 0 \phi_1(x;\mu),\forall\mu>\hat{\mu}>0 ϕ 1 ( x ; μ ) , ∀ μ > μ ^ > 0 x ^ \hat{x} x ^
证明: 若x ^ \hat{x} x ^ D ( ϕ 1 ( x ^ ; μ ) ; p ) = ∇ f ( x ^ ) T p + μ ∑ i ∈ E ∣ ∇ c i ( x ^ ) T p ∣ + μ ∑ i ∈ I ∩ A ( x ^ ) [ ∇ c i ( x ^ ) T p ] − , ? ? ? D(\phi_1(\hat{x};\mu);p)=\nabla f(\hat{x})^Tp+\mu\sum_{i\in\mathcal{E}}\left|\nabla c_i(\hat{x})^Tp\right|+\mu\sum_{i\in\mathcal{I}\cap\mathcal{A}(\hat{x})}\left[\nabla c_i(\hat{x})^Tp\right]^-,??? D ( ϕ 1 ( x ^ ; μ ) ; p ) = ∇ f ( x ^ ) T p + μ i ∈ E ∑ ∣ ∣ ∇ c i ( x ^ ) T p ∣ ∣ + μ i ∈ I ∩ A ( x ^ ) ∑ [ ∇ c i ( x ^ ) T p ] − , ? ? ? A ( x ^ ) \mathcal{A}(\hat{x}) A ( x ^ ) x ^ \hat{x} x ^ F ( x ^ ) \mathcal{F}(\hat{x}) F ( x ^ ) p p p F ( x ^ ) \mathcal{F}(\hat{x}) F ( x ^ ) ∣ ∇ c i ( x ^ ) T p ∣ + ∑ i ∈ I ∩ A ( x ^ ) [ ∇ c i ( x ^ ) T p ] − = 0 , \left|\nabla c_i(\hat{x})^Tp\right|+\sum_{i\in\mathcal{I}\cap\mathcal{A}(\hat{x})}\left[\nabla c_i(\hat{x})^Tp\right]^-=0, ∣ ∣ ∇ c i ( x ^ ) T p ∣ ∣ + i ∈ I ∩ A ( x ^ ) ∑ [ ∇ c i ( x ^ ) T p ] − = 0 , 0 ≤ D ( ϕ 1 ( x ^ ; μ ) ; p ) = ∇ f ( x ^ ) T p , ∀ p ∈ F ( x ^ ) . 0\le D(\phi_1(\hat{x};\mu);p)=\nabla f(\hat{x})^Tp,\quad\forall p\in\mathcal{F}(\hat{x}). 0 ≤ D ( ϕ 1 ( x ^ ; μ ) ; p ) = ∇ f ( x ^ ) T p , ∀ p ∈ F ( x ^ ) . λ ^ i : λ ^ i ≥ 0 , i ∈ I ∩ A ( x ^ ) \hat{\lambda}_i:\hat{\lambda}_i\ge0,i\in\mathcal{I}\cap\mathcal{A}(\hat{x}) λ ^ i : λ ^ i ≥ 0 , i ∈ I ∩ A ( x ^ ) ∇ f ( x ^ ) = ∑ i ∈ A ( x ^ ) λ ^ i ∇ c i ( x ^ ) . \nabla f(\hat{x})=\sum_{i\in\mathcal{A}(\hat{x})}\hat{\lambda}_i\nabla c_i(\hat{x}). ∇ f ( x ^ ) = i ∈ A ( x ^ ) ∑ λ ^ i ∇ c i ( x ^ ) . x ^ \hat{x} x ^
若x ^ \hat{x} x ^ p p p D ( h ( x ^ ) ; p ) < 0 D(h(\hat{x});p)<0 D ( h ( x ^ ) ; p ) < 0 0 ≤ D ( ϕ 1 ( x ^ ; μ ) ; p ) = ∇ f ( x ^ ) T p + μ D ( h ( x ^ ) ; p ) , ∀ μ 充 分 大 , 0\le D(\phi_1(\hat{x};\mu);p)=\nabla f(\hat{x})^Tp+\mu D(h(\hat{x});p),\forall\mu充分大, 0 ≤ D ( ϕ 1 ( x ^ ; μ ) ; p ) = ∇ f ( x ^ ) T p + μ D ( h ( x ^ ) ; p ) , ∀ μ 充 分 大 , ∇ f ( x ^ ) T p → ∞ \nabla f(\hat{x})^Tp\to\infty ∇ f ( x ^ ) T p → ∞
例3 再考虑例1中的问题, 这里使用l 1 l_1 l 1 ϕ 1 ( x ; μ ) = x 1 + x 2 + μ ∣ x 1 2 + x 2 2 − 2 ∣ . \phi_1(x;\mu)=x_1+x_2+\mu|x_1^2+x_2^2-2|. ϕ 1 ( x ; μ ) = x 1 + x 2 + μ ∣ x 1 2 + x 2 2 − 2 ∣ . ϕ ( x ; 2 ) \phi(x;2) ϕ ( x ; 2 ) x ∗ = ( − 1 , − 1 ) T x^*=(-1,-1)^T x ∗ = ( − 1 , − 1 ) T
事实上由定理3, 我们知道对于所有的μ > ∣ λ ∗ ∣ = 0.5 \mu>|\lambda^*|=0.5 μ > ∣ λ ∗ ∣ = 0 . 5 ϕ 1 ( x ; μ ) \phi_1(x;\mu) ϕ 1 ( x ; μ ) x ∗ x^* x ∗ l 1 l_1 l 1
这些结论为基于l 1 l_1 l 1
框架2 (Classical l 1 l_1 l 1 μ 0 > 0 \mu_0>0 μ 0 > 0 τ > 0 \tau>0 τ > 0 x 0 s x_0^s x 0 s k = 0 , 1 , 2 , … k=0,1,2,\ldots k = 0 , 1 , 2 , … \quad\quad x k x_k x k ϕ 1 ( x ; μ k ) \phi_1(x;\mu_k) ϕ 1 ( x ; μ k ) x k s x_k^s x k s \quad\quad h ( x k ) ≤ τ h(x_k)\le\tau h ( x k ) ≤ τ \quad\quad\quad\quad x k x_k x k \quad\quad \quad\quad μ k + 1 > μ k \mu_{k+1}>\mu_k μ k + 1 > μ k \quad\quad x k + 1 s x_{k+1}^s x k + 1 s
这其中对于ϕ 1 ( x ; μ k ) \phi_1(x;\mu_k) ϕ 1 ( x ; μ k ) 若使用ϕ 1 ( x ; μ k ) \phi_1(x;\mu_k) ϕ 1 ( x ; μ k ) , 事情就变得好理解多了. 我们将在下一小节介绍, 这有点类似于逐步二次规划算法.
更新惩罚因子μ k \mu_k μ k μ 0 \mu_0 μ 0 x ∗ x^* x ∗ ϕ 1 ( x ; μ k ) \phi_1(x;\mu_k) ϕ 1 ( x ; μ k ) x k s x_k^s x k s μ k \mu_k μ k
l 1 l_1 l 1
如前所述, ϕ 1 ( x ; μ ) \phi_1(x;\mu) ϕ 1 ( x ; μ ) x : c i ( x ) = 0 , ∃ i ∈ E ∪ I x:c_i(x)=0,\exists i\in\mathcal{E}\cup\mathcal{I} x : c i ( x ) = 0 , ∃ i ∈ E ∪ I ϕ 1 ( x ; μ ) \phi_1(x;\mu) ϕ 1 ( x ; μ ) c i c_i c i f f f q ( p ; μ ) = f ( x ) + ∇ f ( x ) T p + 1 2 p T W p + μ ∑ i ∈ E ∣ c i ( x ) + ∇ c i ( x ) T p ∣ + μ ∑ i ∈ I [ c i ( x ) + ∇ c i ( x ) T p ] − , q(p;\mu)=f(x)+\nabla f(x)^Tp+\frac{1}{2}p^TWp+\mu\sum_{i\in\mathcal{E}}|c_i(x)+\nabla c_i(x)^Tp|+\mu\sum_{i\in\mathcal{I}}[c_i(x)+\nabla c_i(x)^Tp]^-, q ( p ; μ ) = f ( x ) + ∇ f ( x ) T p + 2 1 p T W p + μ i ∈ E ∑ ∣ c i ( x ) + ∇ c i ( x ) T p ∣ + μ i ∈ I ∑ [ c i ( x ) + ∇ c i ( x ) T p ] − , W W W f f f c i , i ∈ E ∪ I c_i,i\in\mathcal{E}\cup\mathcal{I} c i , i ∈ E ∪ I q ( p ; μ ) q(p;\mu) q ( p ; μ ) min p , r , s , t f ( x ) + 1 2 p T W p + ∇ f ( x ) T p + μ ∑ i ∈ E ( r i + s i ) + μ ∑ i ∈ I t i s u b j e c t  t o ∇ c i ( x ) T p + c i ( x ) = r i − s i , i ∈ E , ∇ c i ( x ) T p + c i ( x ) ≥ − t i , i ∈ I , r , s , t ≥ 0. \begin{array}{rl}\min\limits_{p,r,s,t} & f(x)+\frac{1}{2}p^TWp+\nabla f(x)^Tp+\mu\sum\limits_{i\in\mathcal{E}}(r_i+s_i)+\mu\sum\limits_{i\in\mathcal{I}}t_i\\\mathrm{subject\,to} & \nabla c_i(x)^Tp+c_i(x)=r_i-s_i,\quad i\in\mathcal{E},\\& \nabla c_i(x)^Tp+c_i(x)\ge-t_i,\quad i\in\mathcal{I},\\ & r,s,t\ge0.\end{array} p , r , s , t min s u b j e c t t o f ( x ) + 2 1 p T W p + ∇ f ( x ) T p + μ i ∈ E ∑ ( r i + s i ) + μ i ∈ I ∑ t i ∇ c i ( x ) T p + c i ( x ) = r i − s i , i ∈ E , ∇ c i ( x ) T p + c i ( x ) ≥ − t i , i ∈ I , r , s , t ≥ 0 . 标准的二次规划求解器 求解. 即使再添加一个盒型(box-shaped)信赖域约束 ∥ p ∥ ∞ ≤ Δ \Vert p\Vert_{\infty}\le\Delta ∥ p ∥ ∞ ≤ Δ ϕ 1 \phi_1 ϕ 1
选取和更新惩罚因子μ k \mu_k μ k μ k > ∥ λ k ∥ ∞ \mu_k>\Vert\lambda_k\Vert_{\infty} μ k > ∥ λ k ∥ ∞ λ k \lambda_k λ k x k x_k x k μ k \mu_k μ k
因选取合适μ k \mu_k μ k
在选取初始点x k + 1 s x_{k+1}^s x k + 1 s μ k \mu_k μ k x k + 1 s x_{k+1}^s x k + 1 s ϕ 1 ( x ; μ k ) \phi_1(x;\mu_k) ϕ 1 ( x ; μ k ) x k x_k x k
精确非光滑罚函数可以定义为除了l 1 l_1 l 1 ϕ ( x ; μ ) = f ( x ) + μ ∥ c E ( x ) ∥ + μ ∥ [ c I ( x ) ] − ∥ , \phi(x;\mu)=f(x)+\mu\Vert c_{\mathcal{E}}(x)\Vert+\mu\Vert[c_{\mathcal{I}}(x)]^-\Vert, ϕ ( x ; μ ) = f ( x ) + μ ∥ c E ( x ) ∥ + μ ∥ [ c I ( x ) ] − ∥ , ∥ ⋅ ∥ \Vert\cdot\Vert ∥ ⋅ ∥ c E , c I c_{\mathcal{E}},c_{\mathcal{I}} c E , c I h ( x ) = ∥ c E ( x ) ∥ + ∥ [ c I ( x ) ] − ∥ h(x)=\Vert c_{\mathcal{E}}(x)\Vert+\Vert[c_{\mathcal{I}}(x)]^-\Vert h ( x ) = ∥ c E ( x ) ∥ + ∥ [ c I ( x ) ] − ∥ 一一对应 的, 证明可见Han S P和Mangasarian O L2 l 1 , l ∞ , l 2 l_1,l_{\infty},l_2 l 1 , l ∞ , l 2 l ∞ l_{\infty} l ∞
对l 1 l_1 l 1 μ ≥ μ ∗ = ∥ λ ∗ ∥ D , \mu\ge\mu^*=\Vert\lambda^*\Vert_D, μ ≥ μ ∗ = ∥ λ ∗ ∥ D , ∥ ⋅ ∥ D \Vert\cdot\Vert_D ∥ ⋅ ∥ D ∥ ⋅ ∥ \Vert\cdot\Vert ∥ ⋅ ∥ 对偶范数 . 而定理4则无需修改直接可应用.
下说明此类 的罚函数若是精确的必是非光滑的. 为说明简单, 我们仅考虑只有一个等式约束c 1 ( x ) = 0 c_1(x)=0 c 1 ( x ) = 0 ϕ ( x ; μ ) = f ( x ) + μ h ( c i ( x ) ) , \phi(x;\mu)=f(x)+\mu h(c_i(x)), ϕ ( x ; μ ) = f ( x ) + μ h ( c i ( x ) ) , h : R → R h:\mathbb{R}\to\mathbb{R} h : R → R h ( y ) ≥ 0 , ∀ y ∈ R ; h ( 0 ) = 0 h(y)\ge0,\forall y\in\mathbb{R};h(0)=0 h ( y ) ≥ 0 , ∀ y ∈ R ; h ( 0 ) = 0 h h h h h h 0 0 0 ∇ h ( 0 ) = 0 \nabla h(0)=0 ∇ h ( 0 ) = 0 x ∗ x^* x ∗ c i ( x ∗ ) = 0 ⇒ ∇ h ( c 1 ( x ∗ ) ) = 0 c_i(x^*)=0\Rightarrow\nabla h(c_1(x^*))=0 c i ( x ∗ ) = 0 ⇒ ∇ h ( c 1 ( x ∗ ) ) = 0 x ∗ x^* x ∗ ϕ ( x ; μ ) \phi(x;\mu) ϕ ( x ; μ ) μ \mu μ 0 = ∇ ϕ ( x ∗ ; μ ) = ∇ f ( x ∗ ) + μ ∇ c i ( x ∗ ) ∇ h ( c 1 ( x ∗ ) ) = ∇ f ( x ∗ ) . 0=\nabla\phi(x^*;\mu)=\nabla f(x^*)+\mu\nabla c_i(x^*)\nabla h(c_1(x^*))=\nabla f(x^*). 0 = ∇ ϕ ( x ∗ ; μ ) = ∇ f ( x ∗ ) + μ ∇ c i ( x ∗ ) ∇ h ( c 1 ( x ∗ ) ) = ∇ f ( x ∗ ) . f f f h h h ϕ ( ⋅ ; μ ) \phi(\cdot;\mu) ϕ ( ⋅ ; μ )
非光滑罚函数也可用作其他机制下计算迭代步时的价值函数 . 具体可见第十五章第四节的一般性讨论和第十八和第十九章中的具体实施.
下面我们讨论一种被称为乘子法the method of multipliers 或增广Lagrange函数法augmented Lagragian methods 的方法. 此法与第1节中的二次罚函数法相关, 但它通过显式引入Lagrange乘子估计 至目标函数, 减小了问题的病态程度 . 引入显式Lagrange乘子估计后的函数就被称为增广Lagrange函数(augmented Lagrangian function) . 与第2节中讨论的罚函数相反, 增广Lagrange函数极大地保留了光滑性 , 因此具体实施可调用标准的无约束或带界约束的优化软件求解.
我们首先考虑等式约束问题. 二次罚函数Q ( x ; μ ) Q(x;\mu) Q ( x ; μ ) Q ( x ; μ k ) Q(x;\mu_k) Q ( x ; μ k ) x k x_k x k 并不满足可行性条件 c i ( x ) = 0 , i ∈ E c_i(x)=0,i\in\mathcal{E} c i ( x ) = 0 , i ∈ E c i ( x k ) ≈ − λ i ∗ / μ k , ∀ i ∈ E . c_i(x_k)\approx-\lambda_i^*/\mu_k,\quad\forall i\in\mathcal{E}. c i ( x k ) ≈ − λ i ∗ / μ k , ∀ i ∈ E . c i ( x k ) → 0 , μ k → ∞ c_i(x_k)\to0,\mu_k\to\infty c i ( x k ) → 0 , μ k → ∞ 修改函数Q ( x ; μ k ) Q(x;\mu_k) Q ( x ; μ k ) 来避免这种系统上的偏差, 即使得近似极小点无需μ k \mu_k μ k c i ( x ) = 0 c_i(x)=0 c i ( x ) = 0
增广Lagrange函数L A ( x , λ ; μ ) \mathcal{L}_A(x,\lambda;\mu) L A ( x , λ ; μ ) λ \lambda λ L A ( x , λ ; μ ) = d e f f ( x ) − ∑ i ∈ E λ i c i ( x ) + μ 2 ∑ i ∈ E c i 2 ( x ) . \mathcal{L}_A(x,\lambda;\mu)\xlongequal{def}f(x)-\sum_{i\in\mathcal{E}}\lambda_ic_i(x)+\frac{\mu}{2}\sum_{i\in\mathcal{E}}c_i^2(x). L A ( x , λ ; μ ) d e f f ( x ) − i ∈ E ∑ λ i c i ( x ) + 2 μ i ∈ E ∑ c i 2 ( x ) . λ \lambda λ Lagrange函数和二次罚函数的结合 .
下面我们设计一种对x x x μ \mu μ μ k > 0 \mu_k>0 μ k > 0 λ \lambda λ λ k \lambda^k λ k x k x_k x k L A ( x , λ k ; μ k ) \mathcal{L}_A(x,\lambda^k;\mu_k) L A ( x , λ k ; μ k ) 0 = ∇ x L A ( x k , λ k ; μ k ) = ∇ f ( x k ) − ∑ i ∈ E [ λ i k − μ k c i ( x k ) ] ∇ c i ( x k ) . 0=\nabla_x\mathcal{L}_A(x_k,\lambda^k;\mu_k)=\nabla f(x_k)-\sum_{i\in\mathcal{E}}[\lambda_i^k-\mu_kc_i(x_k)]\nabla c_i(x_k). 0 = ∇ x L A ( x k , λ k ; μ k ) = ∇ f ( x k ) − i ∈ E ∑ [ λ i k − μ k c i ( x k ) ] ∇ c i ( x k ) . ∇ c i ( x k ) \nabla c_i(x_k) ∇ c i ( x k ) λ i ∗ ≈ λ i k − μ k c i ( x k ) , ∀ i ∈ E . \lambda_i^*\approx\lambda_i^k-\mu_kc_i(x_k),\quad\forall i\in\mathcal{E}. λ i ∗ ≈ λ i k − μ k c i ( x k ) , ∀ i ∈ E . c i ( x k ) ≈ − 1 μ k ( λ i ∗ − λ i k ) , ∀ i ∈ E , c_i(x_k)\approx-\frac{1}{\mu_k}(\lambda_i^*-\lambda_i^k),\quad\forall i\in\mathcal{E}, c i ( x k ) ≈ − μ k 1 ( λ i ∗ − λ i k ) , ∀ i ∈ E , λ k \lambda^k λ k λ ∗ \lambda^* λ ∗ x k x_k x k ( 1 / μ k ) (1/\mu_k) ( 1 / μ k ) λ k \lambda^k λ k λ i k + 1 = λ i k − μ k c i ( x k ) , ∀ i ∈ E . \lambda_i^{k+1}=\lambda_i^k-\mu_kc_i(x_k),\quad\forall i\in\mathcal{E}. λ i k + 1 = λ i k − μ k c i ( x k ) , ∀ i ∈ E .
于是我们得到下述算法框架.
框架3 (Augmented Lagrangian Method-Equality Constraints)μ 0 > 0 \mu_0>0 μ 0 > 0 τ 0 > 0 \tau_0>0 τ 0 > 0 x 0 s x_0^s x 0 s λ 0 \lambda^0 λ 0 k = 0 , 1 , 2 , … k=0,1,2,\ldots k = 0 , 1 , 2 , … \quad\quad x k x_k x k L A ( ⋅ , λ k ; μ k ) \mathcal{L}_A(\cdot,\lambda^k;\mu_k) L A ( ⋅ , λ k ; μ k ) x k s x_k^s x k s ∥ ∇ x L A ( x k , λ k ; μ k ) ∥ ≤ τ k \Vert\nabla_x\mathcal{L}_A(x_k,\lambda^k;\mu_k)\Vert\le\tau_k ∥ ∇ x L A ( x k , λ k ; μ k ) ∥ ≤ τ k \quad\quad \quad\quad\quad\quad x k x_k x k \quad\quad \quad\quad λ k + 1 \lambda^{k+1} λ k + 1 \quad\quad μ k + 1 ≥ μ k \mu_{k+1}\ge\mu_k μ k + 1 ≥ μ k \quad\quad x k + 1 s = x k x_{k+1}^s=x_k x k + 1 s = x k \quad\quad τ k + 1 \tau_{k+1} τ k + 1
我们通过一个例子来阐释此法可以无需将μ \mu μ ∞ \infty ∞ x k + 1 s x_{k+1}^s x k + 1 s τ k \tau_k τ k ∑ i ∈ E ∣ c ( x k ) ∣ \sum_{i\in\mathcal{E}}|c(x_k)| ∑ i ∈ E ∣ c ( x k ) ∣ μ \mu μ
例4 再次考虑例1中的问题, 其增广Lagrange函数为L A ( x , λ ; μ ) = x 1 + x 2 − λ ( x 1 2 + x 2 2 − 2 ) + μ 2 ( x 1 2 + x 2 2 − 2 ) 2 . \mathcal{L}_A(x,\lambda;\mu)=x_1+x_2-\lambda(x_1^2+x_2^2-2)+\frac{\mu}{2}(x_1^2+x_2^2-2)^2. L A ( x , λ ; μ ) = x 1 + x 2 − λ ( x 1 2 + x 2 2 − 2 ) + 2 μ ( x 1 2 + x 2 2 − 2 ) 2 . x ∗ = ( − 1 , − 1 ) T x^*=(-1,-1)^T x ∗ = ( − 1 , − 1 ) T λ ∗ = − 0.5 \lambda^*=-0.5 λ ∗ = − 0 . 5 k k k μ k = 1 \mu_k=1 μ k = 1 λ k = − 0.4 \lambda^k=-0.4 λ k = − 0 . 4 μ k = 1 \mu_k=1 μ k = 1
注意到等高线之间的跨度说明此问题的条件数与二次罚函数Q ( x ; 1 ) Q(x;1) Q ( x ; 1 ) x k ≈ ( − 1.02 , − 1.02 ) T x_k\approx(-1.02,-1.02)^T x k ≈ ( − 1 . 0 2 , − 1 . 0 2 ) T x ∗ = ( − 1 , − 1 ) T x^*=(-1,-1)^T x ∗ = ( − 1 , − 1 ) T L A ( x , λ ; μ ) \mathcal{L}_A(x,\lambda;\mu) L A ( x , λ ; μ )
下面我们证明关于增广Lagrange函数使用和等式约束问题乘子法的两条结论.
第一条证实, 若我们有精确Lagrange乘子λ ∗ \lambda^* λ ∗ x ∗ x^* x ∗ L A ( x , λ ∗ ; μ ) , μ \mathcal{L}_A(x,\lambda^*;\mu),\mu L A ( x , λ ∗ ; μ ) , μ λ ∗ \lambda^* λ ∗ 只要λ \lambda λ λ ∗ \lambda^* λ ∗ μ \mu μ L A ( x , λ ; μ ) \mathcal{L}_A(x,\lambda;\mu) L A ( x , λ ; μ ) x ∗ x^* x ∗ .
定理5 令x ∗ x^* x ∗ λ = λ ∗ \lambda=\lambda^* λ = λ ∗ μ ˉ \bar{\mu} μ ˉ ∀ μ ≥ μ ˉ \forall\mu\ge\bar{\mu} ∀ μ ≥ μ ˉ x ∗ x^* x ∗ L A ( x , λ ∗ ; μ ) \mathcal{L}_A(x,\lambda^*;\mu) L A ( x , λ ∗ ; μ )
证明: 我们证明x ∗ x^* x ∗ μ \mu μ L A ( x , λ ∗ ; μ ) \mathcal{L}_A(x,\lambda^*;\mu) L A ( x , λ ∗ ; μ ) ∇ x L A ( x ∗ , λ ∗ ; μ ) = 0 , ∇ x x 2 L A ( x ∗ , λ ∗ ; μ ) 正 定 . \nabla_x\mathcal{L}_A(x^*,\lambda^*;\mu)=0,\quad\nabla^2_{xx}\mathcal{L}_A(x^*,\lambda^*;\mu)正定. ∇ x L A ( x ∗ , λ ∗ ; μ ) = 0 , ∇ x x 2 L A ( x ∗ , λ ∗ ; μ ) 正 定 . x ∗ x^* x ∗ ∇ x L ( x ∗ , λ ∗ ) = 0 , c i ( x ∗ ) = 0 , ∀ i ∈ E \nabla_x\mathcal{L}(x^*,\lambda^*)=0,c_i(x^*)=0,\forall i\in\mathcal{E} ∇ x L ( x ∗ , λ ∗ ) = 0 , c i ( x ∗ ) = 0 , ∀ i ∈ E ∇ x L A ( x ∗ , λ ∗ ; μ ) = ∇ f ( x ∗ ) − ∑ i ∈ E [ λ i ∗ − μ c i ( x ∗ ) ] ∇ c i ( x ∗ ) = ∇ f ( x ∗ ) − ∑ i ∈ E λ i ∗ ∇ c i ( x ∗ ) = ∇ x L ( x ∗ , λ ∗ ) = 0 , \begin{aligned}\nabla_x\mathcal{L}_A(x^*,\lambda^*;\mu)&=\nabla f(x^*)-\sum_{i\in\mathcal{E}}[\lambda_i^*-\mu c_i(x^*)]\nabla c_i(x^*)\\&=\nabla f(x^*)-\sum_{i\in\mathcal{E}}\lambda_i^*\nabla c_i(x^*)=\nabla_x\mathcal{L}(x^*,\lambda^*)=0,\end{aligned} ∇ x L A ( x ∗ , λ ∗ ; μ ) = ∇ f ( x ∗ ) − i ∈ E ∑ [ λ i ∗ − μ c i ( x ∗ ) ] ∇ c i ( x ∗ ) = ∇ f ( x ∗ ) − i ∈ E ∑ λ i ∗ ∇ c i ( x ∗ ) = ∇ x L ( x ∗ , λ ∗ ) = 0 , μ \mu μ A A A ∇ x x 2 L A ( x ∗ , λ ∗ ; μ ) = ∇ x x 2 L ( x ∗ , λ ∗ ) + μ A T A . \nabla^2_{xx}\mathcal{L}_A(x^*,\lambda^*;\mu)=\nabla^2_{xx}\mathcal{L}(x^*,\lambda^*)+\mu A^TA. ∇ x x 2 L A ( x ∗ , λ ∗ ; μ ) = ∇ x x 2 L ( x ∗ , λ ∗ ) + μ A T A . μ \mu μ k ≥ 1 k\ge1 k ≥ 1 w k : ∥ w k ∥ = 1 w_k:\Vert w_k\Vert=1 w k : ∥ w k ∥ = 1 0 ≥ w k T ∇ x x 2 L A ( x ∗ , λ ∗ ; k ) w k = w k T ∇ x x 2 L ( x ∗ , λ ∗ ) w k + k ∥ A w k ∥ 2 2 , 0\ge w_k^T\nabla^2_{xx}\mathcal{L}_A(x^*,\lambda^*;k)w_k=w_k^T\nabla^2_{xx}\mathcal{L}(x^*,\lambda^*)w_k+k\Vert Aw_k\Vert^2_2, 0 ≥ w k T ∇ x x 2 L A ( x ∗ , λ ∗ ; k ) w k = w k T ∇ x x 2 L ( x ∗ , λ ∗ ) w k + k ∥ A w k ∥ 2 2 , ∥ A w k ∥ 2 2 ≤ − ( 1 / k ) w k T ∇ x x 2 L ( x ∗ , λ ∗ ) w k → 0 , k → ∞ . \Vert Aw_k\Vert_2^2\le-(1/k)w_k^T\nabla^2_{xx}\mathcal{L}(x^*,\lambda^*)w_k\to0,\quad k\to\infty. ∥ A w k ∥ 2 2 ≤ − ( 1 / k ) w k T ∇ x x 2 L ( x ∗ , λ ∗ ) w k → 0 , k → ∞ . { w k } \{w_k\} { w k } w w w A w = 0 Aw=0 A w = 0 w ∈ F ( x ∗ ) w\in\mathcal{F}(x^*) w ∈ F ( x ∗ ) w k T ∇ x x 2 L ( x ∗ , λ ∗ ) w k ≤ − k ∥ A w k ∥ 2 2 ≤ 0 , w_k^T\nabla^2_{xx}\mathcal{L}(x^*,\lambda^*)w_k\le-k\Vert Aw_k\Vert_2^2\le0, w k T ∇ x x 2 L ( x ∗ , λ ∗ ) w k ≤ − k ∥ A w k ∥ 2 2 ≤ 0 , w T ∇ x x 2 L ( x ∗ , λ ∗ ) w ≤ 0 w^T\nabla^2_{xx}\mathcal{L}(x^*,\lambda^*)w\le0 w T ∇ x x 2 L ( x ∗ , λ ∗ ) w ≤ 0
第二条结论则立意于更实际的情形λ ≠ λ ∗ \lambda\ne\lambda^* λ ̸ = λ ∗ L A ( x , λ ; μ ) \mathcal{L}_A(x,\lambda;\mu) L A ( x , λ ; μ ) x ∗ x^* x ∗ x k x_k x k λ k + 1 \lambda^{k+1} λ k + 1 x ∗ , λ ∗ x^*,\lambda^* x ∗ , λ ∗
定理6 设定理5的条件在x ∗ , λ ∗ x^*,\lambda^* x ∗ , λ ∗ μ ˉ \bar{\mu} μ ˉ δ , ϵ , M \delta,\epsilon,M δ , ϵ , M
对所有满足∥ λ k − λ ∗ ∥ ≤ μ k δ , μ k ≥ μ ˉ \Vert\lambda^k-\lambda^*\Vert\le\mu_k\delta,\quad\mu_k\ge\bar{\mu} ∥ λ k − λ ∗ ∥ ≤ μ k δ , μ k ≥ μ ˉ λ k , μ k \lambda^k,\mu_k λ k , μ k min x L A ( x , λ k ; μ k ) , s u b j e c t  t o  ∥ x − x ∗ ∥ ≤ ϵ \min_x\mathcal{L}_A(x,\lambda^k;\mu_k),\quad\mathrm{subject\,to\,}\Vert x-x^*\Vert\le\epsilon x min L A ( x , λ k ; μ k ) , s u b j e c t t o ∥ x − x ∗ ∥ ≤ ϵ x k x_k x k ∥ x k − x ∗ ∥ ≤ M ∥ λ k − λ ∗ ∥ / μ k . \Vert x_k-x^*\Vert\le M\Vert\lambda^k-\lambda^*\Vert/\mu_k. ∥ x k − x ∗ ∥ ≤ M ∥ λ k − λ ∗ ∥ / μ k .
对所有满足∥ λ k − λ ∗ ∥ ≤ μ k δ , μ k ≥ μ ˉ \Vert\lambda^k-\lambda^*\Vert\le\mu_k\delta,\quad\mu_k\ge\bar{\mu} ∥ λ k − λ ∗ ∥ ≤ μ k δ , μ k ≥ μ ˉ λ k , μ k \lambda^k,\mu_k λ k , μ k ∥ λ k + 1 − λ ∗ ∥ ≤ M ∥ λ k − λ ∗ ∥ / μ k , \Vert\lambda^{k+1}-\lambda^*\Vert\le M\Vert\lambda^k-\lambda^*\Vert/\mu_k, ∥ λ k + 1 − λ ∗ ∥ ≤ M ∥ λ k − λ ∗ ∥ / μ k , λ k + 1 \lambda^{k+1} λ k + 1 λ i k + 1 = λ i k − μ k c i ( x k ) , ∀ i ∈ E \lambda_i^{k+1}=\lambda_i^k-\mu_kc_i(x_k),\quad\forall i\in\mathcal{E} λ i k + 1 = λ i k − μ k c i ( x k ) , ∀ i ∈ E
对所有满足∥ λ k − λ ∗ ∥ ≤ μ k δ , μ k ≥ μ ˉ \Vert\lambda^k-\lambda^*\Vert\le\mu_k\delta,\quad\mu_k\ge\bar{\mu} ∥ λ k − λ ∗ ∥ ≤ μ k δ , μ k ≥ μ ˉ λ k , μ k \lambda^k,\mu_k λ k , μ k ∇ x x 2 L A ( x k , λ k ; μ k ) \nabla^2_{xx}\mathcal{L}_A(x_k,\lambda^k;\mu_k) ∇ x x 2 L A ( x k , λ k ; μ k ) ∇ c i ( x k ) , i ∈ E \nabla c_i(x_k),i\in\mathcal{E} ∇ c i ( x k ) , i ∈ E
证明: 以下以μ , λ , λ ~ \mu,\lambda,\tilde\lambda μ , λ , λ ~ μ k , λ k , λ k + 1 \mu_k,\lambda^k,\lambda^{k+1} μ k , λ k , λ k + 1 μ > 0 \mu>0 μ > 0 ( x , λ ~ , λ , μ ) (x,\tilde\lambda,\lambda,\mu) ( x , λ ~ , λ , μ ) ∇ f ( x ) − ∇ h ( x ) λ ~ = 0 , h ( x ) + ( λ − λ ~ ) / μ = 0 , \nabla f(x)-\nabla h(x)\tilde\lambda=0,\quad h(x)+(\lambda-\tilde\lambda)/\mu=0, ∇ f ( x ) − ∇ h ( x ) λ ~ = 0 , h ( x ) + ( λ − λ ~ ) / μ = 0 , h ( x ) = [ c 1 ( x ) c 2 ( x ) ⋮ c m ( x ) ] , ∇ h ( x ) = [ ∇ c 1 ( x ) ∇ c 2 ( x ) ⋯ ∇ c m ( x ) ] . h(x)=\begin{bmatrix}c_1(x)\\c_2(x)\\\vdots\\c_m(x)\end{bmatrix},\quad \nabla h(x)=\begin{bmatrix}\nabla c_1(x) & \nabla c_2(x) & \cdots & \nabla c_m(x)\end{bmatrix}. h ( x ) = ⎣ ⎢ ⎢ ⎢ ⎡ c 1 ( x ) c 2 ( x ) ⋮ c m ( x ) ⎦ ⎥ ⎥ ⎥ ⎤ , ∇ h ( x ) = [ ∇ c 1 ( x ) ∇ c 2 ( x ) ⋯ ∇ c m ( x ) ] . t ∈ R m , γ ∈ R t\in\mathbb{R}^m,\gamma\in\mathbb{R} t ∈ R m , γ ∈ R t = ( λ − λ ∗ ) / μ , γ = 1 / μ , t=(\lambda-\lambda^*)/\mu,\quad\gamma=1/\mu, t = ( λ − λ ∗ ) / μ , γ = 1 / μ , ∇ f ( x ) − ∇ h ( x ) λ ~ = 0 , h ( x ) + t + γ λ ∗ − γ λ ~ = 0. \nabla f(x)-\nabla h(x)\tilde\lambda=0,\quad h(x)+t+\gamma\lambda^*-\gamma\tilde\lambda=0. ∇ f ( x ) − ∇ h ( x ) λ ~ = 0 , h ( x ) + t + γ λ ∗ − γ λ ~ = 0 . t = 0 , γ ∈ [ 0 , 1 / μ ˉ ] t=0,\gamma\in[0,1/\bar{\mu}] t = 0 , γ ∈ [ 0 , 1 / μ ˉ ] x = x ∗ , λ ~ = λ ∗ x=x^*,\tilde\lambda=\lambda^* x = x ∗ , λ ~ = λ ∗ ( x , λ ~ ) (x,\tilde\lambda) ( x , λ ~ ) [ ∇ x x 2 L ( x ∗ , λ ∗ ) ∇ h ( x ∗ ) − ∇ h ( x ∗ ) T − γ I ] . \begin{bmatrix}\nabla^2_{xx}\mathcal{L}(x^*,\lambda^*) & \nabla h(x^*)\\-\nabla h(x^*)^T & -\gamma I\end{bmatrix}. [ ∇ x x 2 L ( x ∗ , λ ∗ ) − ∇ h ( x ∗ ) T ∇ h ( x ∗ ) − γ I ] . γ ∈ [ 0 , 1 / μ ˉ ] \gamma\in[0,1/\bar\mu] γ ∈ [ 0 , 1 / μ ˉ ] γ = 0 \gamma=0 γ = 0 ∀ γ ∈ ( 0 , 1 / μ ˉ ] \forall\gamma\in(0,1/\bar\mu] ∀ γ ∈ ( 0 , 1 / μ ˉ ] z ∈ R n , w ∈ R m z\in\mathbb{R}^n,w\in\mathbb{R}^m z ∈ R n , w ∈ R m [ ∇ x x 2 L ( x ∗ , λ ∗ ) ∇ h ( x ∗ ) − ∇ h ( x ∗ ) T − γ I ] [ z w ] = 0 \begin{bmatrix}\nabla^2_{xx}\mathcal{L}(x^*,\lambda^*) & \nabla h(x^*)\\-\nabla h(x^*)^T & -\gamma I\end{bmatrix}\begin{bmatrix}z\\w\end{bmatrix}=0 [ ∇ x x 2 L ( x ∗ , λ ∗ ) − ∇ h ( x ∗ ) T ∇ h ( x ∗ ) − γ I ] [ z w ] = 0 ∇ x x 2 L ( x ∗ , λ ∗ ) z + ∇ h ( x ∗ ) w = 0 , − ∇ h ( x ∗ ) T z − γ w = 0. \begin{aligned}\nabla^2_{xx}\mathcal{L}(x^*,\lambda^*)z+\nabla h(x^*)w&=0,\\-\nabla h(x^*)^Tz-\gamma w&=0.\end{aligned} ∇ x x 2 L ( x ∗ , λ ∗ ) z + ∇ h ( x ∗ ) w − ∇ h ( x ∗ ) T z − γ w = 0 , = 0 . w w w [ ∇ x x 2 L ( x ∗ , λ ∗ ) − ( 1 / γ ) ∇ h ( x ∗ ) ∇ h ( x ∗ ) T ] z = 0. [\nabla^2_{xx}\mathcal{L}(x^*,\lambda^*)-(1/\gamma)\nabla h(x^*)\nabla h(x^*)^T]z=0. [ ∇ x x 2 L ( x ∗ , λ ∗ ) − ( 1 / γ ) ∇ h ( x ∗ ) ∇ h ( x ∗ ) T ] z = 0 . γ = 1 / μ , μ ≥ μ ˉ \gamma=1/\mu,\mu\ge\bar\mu γ = 1 / μ , μ ≥ μ ˉ ∇ x x 2 L A ( x ∗ , λ ∗ ; μ ) z = 0 \nabla^2_{xx}\mathcal{L}_A(x^*,\lambda^*;\mu)z=0 ∇ x x 2 L A ( x ∗ , λ ∗ ; μ ) z = 0 ∇ x x 2 L A ( x ∗ , λ ∗ ; μ ) > 0 , μ ≥ μ ˉ \nabla^2_{xx}\mathcal{L}_A(x^*,\lambda^*;\mu)>0,\mu\ge\bar\mu ∇ x x 2 L A ( x ∗ , λ ∗ ; μ ) > 0 , μ ≥ μ ˉ z = 0 ⇒ w = 0 z=0\Rightarrow w=0 z = 0 ⇒ w = 0 γ ∈ [ 0 , 1 / μ ˉ ] \gamma\in[0,1/\bar\mu] γ ∈ [ 0 , 1 / μ ˉ ] ϵ > 0 , δ > 0 \epsilon>0,\delta>0 ϵ > 0 , δ > 0 S ( K ; δ ) S(K;\delta) S ( K ; δ ) K = { ( 0 , γ ) ∣ γ ∈ [ 0 , 1 / μ ˉ ] } K=\{(0,\gamma)\mid\gamma\in[0,1/\bar\mu]\} K = { ( 0 , γ ) ∣ γ ∈ [ 0 , 1 / μ ˉ ] } x ^ ( t , γ ) , λ ^ ( t , γ ) \hat x(t,\gamma),\hat\lambda(t,\gamma) x ^ ( t , γ ) , λ ^ ( t , γ ) ( ∣ x ^ ( t , γ ) − x ∗ ∣ 2 + ∣ λ ^ ( t , γ ) − λ ∗ ∣ 2 ) 1 / 2 < ϵ , ∀ ( t , γ ) ∈ S ( K ; δ ) , \left(|\hat{x}(t,\gamma)-x^*|^2+|\hat{\lambda}(t,\gamma)-\lambda^*|^2\right)^{1/2}<\epsilon,\forall(t,\gamma)\in S(K;\delta), ( ∣ x ^ ( t , γ ) − x ∗ ∣ 2 + ∣ λ ^ ( t , γ ) − λ ∗ ∣ 2 ) 1 / 2 < ϵ , ∀ ( t , γ ) ∈ S ( K ; δ ) , ∇ f [ x ^ ( t , γ ) ] − ∇ h [ x ^ ( t , γ ) ] λ ^ ( t , γ ) = 0 , h [ x ^ ( t , γ ) ] + t + γ λ ∗ − γ λ ^ ( t , γ ) = 0. \begin{aligned}\nabla f[\hat{x}(t,\gamma)]-\nabla h[\hat{x}(t,\gamma)]\hat{\lambda}(t,\gamma)&=0,\\h[\hat{x}(t,\gamma)]+t+\gamma\lambda^*-\gamma\hat\lambda(t,\gamma)&=0.\end{aligned} ∇ f [ x ^ ( t , γ ) ] − ∇ h [ x ^ ( t , γ ) ] λ ^ ( t , γ ) h [ x ^ ( t , γ ) ] + t + γ λ ∗ − γ λ ^ ( t , γ ) = 0 , = 0 . δ , ϵ \delta,\epsilon δ , ϵ ∇ h [ x ^ ( t , γ ) ] \nabla h[\hat{x}(t,\gamma)] ∇ h [ x ^ ( t , γ ) ] m m m ∇ x x 2 L [ x ^ ( t , γ ) , λ ^ ( t , γ ) ] − μ ∇ h [ x ^ ( t , γ ) ] ∇ h [ x ^ ( t , γ ) ] T > 0 , ∀ ( t , γ ) ∈ S ( K ; δ ) , μ ≥ μ ˉ . \nabla^2_{xx}\mathcal{L}[\hat{x}(t,\gamma),\hat{\lambda}(t,\gamma)]-\mu\nabla h[\hat x(t,\gamma)]\nabla h[\hat x(t,\gamma)]^T>0,\forall (t,\gamma)\in S(K;\delta),\mu\ge\bar\mu. ∇ x x 2 L [ x ^ ( t , γ ) , λ ^ ( t , γ ) ] − μ ∇ h [ x ^ ( t , γ ) ] ∇ h [ x ^ ( t , γ ) ] T > 0 , ∀ ( t , γ ) ∈ S ( K ; δ ) , μ ≥ μ ˉ . μ ≥ μ ˉ , ∥ λ − λ ∗ ∥ ≤ δ μ \mu\ge\bar\mu,\Vert\lambda-\lambda^*\Vert\le\delta\mu μ ≥ μ ˉ , ∥ λ − λ ∗ ∥ ≤ δ μ x ( λ , μ ) = x ^ ( λ − λ ∗ μ , 1 μ ) , λ ~ ( λ , μ ) = λ ^ ( λ − λ ∗ μ , 1 μ ) . x(\lambda,\mu)=\hat{x}\left(\frac{\lambda-\lambda^*}{\mu},\frac{1}{\mu}\right),\quad\tilde\lambda(\lambda,\mu)=\hat\lambda\left(\frac{\lambda-\lambda^*}{\mu},\frac{1}{\mu}\right). x ( λ , μ ) = x ^ ( μ λ − λ ∗ , μ 1 ) , λ ~ ( λ , μ ) = λ ^ ( μ λ − λ ∗ , μ 1 ) . ∥ λ − λ ∗ ∥ ≤ μ δ , μ ≥ μ ˉ \Vert\lambda-\lambda^*\Vert\le\mu\delta,\quad\mu\ge\bar{\mu} ∥ λ − λ ∗ ∥ ≤ μ δ , μ ≥ μ ˉ λ k , μ k \lambda^k,\mu_k λ k , μ k ∇ f [ x ( λ , μ ) ] − ∇ h [ x ( λ , c ) ] λ ~ ( λ , μ ) = 0 , λ ~ ( λ , μ ) = λ − μ h [ x ( λ , μ ) ] , ∇ x x 2 L [ x ( λ , μ ) , λ ~ ( λ , μ ) ] − μ ∇ h [ x ( λ , μ ) ] ∇ h [ x ( λ , μ ) ] T = ∇ x x 2 L A [ x ( λ , μ ) , λ ] > 0. \begin{aligned}\nabla f[x(\lambda,\mu)]-\nabla h[x(\lambda,c)]\tilde\lambda(\lambda,\mu)&=0,\\\tilde\lambda(\lambda,\mu)&=\lambda-\mu h[x(\lambda,\mu)],\\\nabla^2_{xx}\mathcal{L}[x(\lambda,\mu),\tilde\lambda(\lambda,\mu)]-\mu\nabla h[x(\lambda,\mu)]\nabla h[x(\lambda,\mu)]^T=\nabla^2_{xx}\mathcal{L}_A[x(\lambda,\mu),\lambda]&>0.\end{aligned} ∇ f [ x ( λ , μ ) ] − ∇ h [ x ( λ , c ) ] λ ~ ( λ , μ ) λ ~ ( λ , μ ) ∇ x x 2 L [ x ( λ , μ ) , λ ~ ( λ , μ ) ] − μ ∇ h [ x ( λ , μ ) ] ∇ h [ x ( λ , μ ) ] T = ∇ x x 2 L A [ x ( λ , μ ) , λ ] = 0 , = λ − μ h [ x ( λ , μ ) ] , > 0 . ∇ f [ x ^ ( t , γ ) ] − ∇ h [ x ^ ( t , γ ) ] λ ^ ( t , γ ) = 0 , h [ x ^ ( t , γ ) ] + t + γ λ ∗ − γ λ ^ ( t , γ ) = 0. \begin{aligned}\nabla f[\hat{x}(t,\gamma)]-\nabla h[\hat{x}(t,\gamma)]\hat{\lambda}(t,\gamma)&=0,\\h[\hat{x}(t,\gamma)]+t+\gamma\lambda^*-\gamma\hat\lambda(t,\gamma)&=0.\end{aligned} ∇ f [ x ^ ( t , γ ) ] − ∇ h [ x ^ ( t , γ ) ] λ ^ ( t , γ ) h [ x ^ ( t , γ ) ] + t + γ λ ∗ − γ λ ^ ( t , γ ) = 0 , = 0 . t , γ t,\gamma t , γ [ ∇ t x ^ ( t , γ ) T ∇ γ x ^ ( t , γ ) T ∇ t λ ^ ( t , γ ) T ∇ γ λ ^ ( t , γ ) T ] = A ( t , γ ) [ O O − I λ ^ ( t , γ ) − λ ∗ ] , \begin{bmatrix}\nabla_t\hat x(t,\gamma)^T & \nabla_{\gamma}\hat x(t,\gamma)^T\\\nabla_t\hat\lambda(t,\gamma)^T & \nabla_{\gamma}\hat\lambda(t,\gamma)^T\end{bmatrix}=A(t,\gamma)\begin{bmatrix}O & O\\-I & \hat\lambda(t,\gamma)-\lambda^*\end{bmatrix}, [ ∇ t x ^ ( t , γ ) T ∇ t λ ^ ( t , γ ) T ∇ γ x ^ ( t , γ ) T ∇ γ λ ^ ( t , γ ) T ] = A ( t , γ ) [ O − I O λ ^ ( t , γ ) − λ ∗ ] , A ( t , γ ) = [ ∇ x x 2 L [ x ^ ( t , γ ) , λ ^ ( t , γ ) ] − ∇ h [ x ^ ( t , γ ) ] ∇ h [ x ^ ( t , γ ) ] T − γ I ] − 1 . A(t,\gamma)=\begin{bmatrix}\nabla^2_{xx}\mathcal{L}[\hat x(t,\gamma),\hat\lambda(t,\gamma)] &- \nabla h[\hat x(t,\gamma)]\\\nabla h[\hat x(t,\gamma)]^T & -\gamma I\end{bmatrix}^{-1}. A ( t , γ ) = [ ∇ x x 2 L [ x ^ ( t , γ ) , λ ^ ( t , γ ) ] ∇ h [ x ^ ( t , γ ) ] T − ∇ h [ x ^ ( t , γ ) ] − γ I ] − 1 . ∀ ( t , γ ) : ∣ t ∣ < δ , γ ∈ [ 0 , 1 / μ ˉ ] \forall(t,\gamma):|t|<\delta,\gamma\in[0,1/\bar\mu] ∀ ( t , γ ) : ∣ t ∣ < δ , γ ∈ [ 0 , 1 / μ ˉ ] [ x ^ ( t , γ ) − x ∗ λ ^ ( t , γ ) − λ ∗ ] = [ x ^ ( t , γ ) − x ^ ( 0 , 0 ) λ ^ ( t , γ ) − λ ^ ( 0 , 0 ) ] = ∫ 0 1 A ( ζ t , ζ γ ) [ O O − I λ ^ ( ζ t , ζ γ ) − λ ∗ ] [ t γ ] d ζ . \begin{aligned}\begin{bmatrix}\hat x(t,\gamma)-x^*\\\hat\lambda(t,\gamma)-\lambda^*\end{bmatrix}&=\begin{bmatrix}\hat x(t,\gamma)-\hat x(0,0)\\\hat\lambda(t,\gamma)-\hat\lambda(0,0)\end{bmatrix}\\&=\int_0^1A(\zeta t,\zeta\gamma)\begin{bmatrix}O & O\\-I & \hat\lambda(\zeta t,\zeta\gamma)-\lambda^*\end{bmatrix}\begin{bmatrix}t\\\gamma\end{bmatrix}\,\mathrm{d}\zeta.\end{aligned} [ x ^ ( t , γ ) − x ∗ λ ^ ( t , γ ) − λ ∗ ] = [ x ^ ( t , γ ) − x ^ ( 0 , 0 ) λ ^ ( t , γ ) − λ ^ ( 0 , 0 ) ] = ∫ 0 1 A ( ζ t , ζ γ ) [ O − I O λ ^ ( ζ t , ζ γ ) − λ ∗ ] [ t γ ] d ζ . γ ∈ [ 0 , 1 / μ ˉ ] \gamma\in[0,1/\bar\mu] γ ∈ [ 0 , 1 / μ ˉ ] δ \delta δ A ( t , γ ) A(t,\gamma) A ( t , γ ) { ( t , γ ) ∣ ∣ t ∣ < δ , γ ∈ [ 0 , 1 / μ ˉ ] } \{(t,\gamma)\mid|t|<\delta,\gamma\in[0,1/\bar\mu]\} { ( t , γ ) ∣ ∣ t ∣ < δ , γ ∈ [ 0 , 1 / μ ˉ ] } M ˉ \bar M M ˉ ∥ A ( t , γ ) ∥ ≤ M ˉ , ∀ ∣ t ∣ < δ , γ ∈ [ 0 , 1 / μ ˉ ] \Vert A(t,\gamma)\Vert\le\bar M,\forall|t|<\delta,\gamma\in[0,1/\bar\mu] ∥ A ( t , γ ) ∥ ≤ M ˉ , ∀ ∣ t ∣ < δ , γ ∈ [ 0 , 1 / μ ˉ ] δ \delta δ M ˉ δ < 1 \bar M\delta<1 M ˉ δ < 1 ( ∣ x ^ ( t , γ ) − x ∗ ∣ 2 + ∣ λ ^ ( t , γ ) − λ ∗ ∣ 2 ) 1 / 2 ≤ M ˉ ( ∣ t ∣ + max 0 ≤ ζ ≤ 1 ∣ λ ^ ( ζ t , ζ γ ) − λ ∗ ∣ γ ) . \begin{aligned}&\left(|\hat x(t,\gamma)-x^*|^2+|\hat\lambda(t,\gamma)-\lambda^*|^2\right)^{1/2}&\le\bar M\left(|t|+\max_{0\le\zeta\le1}|\hat\lambda(\zeta t,\zeta\gamma)-\lambda^*|\gamma\right).\end{aligned} ( ∣ x ^ ( t , γ ) − x ∗ ∣ 2 + ∣ λ ^ ( t , γ ) − λ ∗ ∣ 2 ) 1 / 2 ≤ M ˉ ( ∣ t ∣ + 0 ≤ ζ ≤ 1 max ∣ λ ^ ( ζ t , ζ γ ) − λ ∗ ∣ γ ) . ∀ ( t , γ ) : ∣ t ∣ < δ , γ ∈ [ 0 , 1 / μ ˉ ] , γ < δ \forall(t,\gamma):|t|<\delta,\gamma\in[0,1/\bar\mu],\gamma<\delta ∀ ( t , γ ) : ∣ t ∣ < δ , γ ∈ [ 0 , 1 / μ ˉ ] , γ < δ ∣ λ ^ ( t , γ ) − λ ∗ ∣ ≤ M ˉ ∣ t ∣ + M ˉ γ max 0 ≤ ζ ≤ 1 ∣ λ ^ ( ζ t , ζ γ ) − λ ∗ ∣ . |\hat\lambda(t,\gamma)-\lambda^*|\le\bar M|t|+\bar M\gamma\max_{0\le\zeta\le1}|\hat\lambda(\zeta t,\zeta\gamma)-\lambda^*|. ∣ λ ^ ( t , γ ) − λ ∗ ∣ ≤ M ˉ ∣ t ∣ + M ˉ γ 0 ≤ ζ ≤ 1 max ∣ λ ^ ( ζ t , ζ γ ) − λ ∗ ∣ . t , γ t,\gamma t , γ ζ t , ζ γ , ζ ∈ [ 0 , 1 ] \zeta t,\zeta\gamma,\zeta\in[0,1] ζ t , ζ γ , ζ ∈ [ 0 , 1 ] max 0 ≤ ζ ≤ 1 ∣ λ ^ ( ζ t , ζ γ ) − λ ∗ ) ∣ ≤ M ˉ 1 − M ˉ γ ∣ t ∣ . \max_{0\le\zeta\le1}|\hat\lambda(\zeta t,\zeta\gamma)-\lambda^*)|\le\frac{\bar M}{1-\bar M\gamma}|t|. 0 ≤ ζ ≤ 1 max ∣ λ ^ ( ζ t , ζ γ ) − λ ∗ ) ∣ ≤ 1 − M ˉ γ M ˉ ∣ t ∣ . ∀ ( t , γ ) : ∣ t ∣ < δ , γ ∈ [ 0 , 1 / μ ˉ ] , γ < δ \forall(t,\gamma):|t|<\delta,\gamma\in[0,1/\bar\mu],\gamma<\delta ∀ ( t , γ ) : ∣ t ∣ < δ , γ ∈ [ 0 , 1 / μ ˉ ] , γ < δ ( ∣ x ^ ( t , γ ) − x ∗ ∣ 2 + ∣ λ ^ ( t , , γ ) − λ ∗ ∣ 2 ) 1 / 2 ≤ ( μ + μ 2 γ 1 − μ γ ) ∣ t ∣ ≤ M ˉ 1 − M ˉ δ ∣ t ∣ . \left(|\hat x(t,\gamma)-x^*|^2+|\hat\lambda(t,,\gamma)-\lambda^*|^2\right)^{1/2}\le\left(\mu+\frac{\mu^2\gamma}{1-\mu\gamma}\right)|t|\le\frac{\bar M}{1-\bar M\delta}|t|. ( ∣ x ^ ( t , γ ) − x ∗ ∣ 2 + ∣ λ ^ ( t , , γ ) − λ ∗ ∣ 2 ) 1 / 2 ≤ ( μ + 1 − μ γ μ 2 γ ) ∣ t ∣ ≤ 1 − M ˉ δ M ˉ ∣ t ∣ . δ \delta δ ( ∣ x ^ ( t , γ ) − x ∗ ∣ 2 + ∣ λ ^ ( t , , γ ) − λ ∗ ∣ 2 ) 1 / 2 ≤ 2 M ˉ ∣ t ∣ . \left(|\hat x(t,\gamma)-x^*|^2+|\hat\lambda(t,,\gamma)-\lambda^*|^2\right)^{1/2}\le2\bar M|t|. ( ∣ x ^ ( t , γ ) − x ∗ ∣ 2 + ∣ λ ^ ( t , , γ ) − λ ∗ ∣ 2 ) 1 / 2 ≤ 2 M ˉ ∣ t ∣ . t t t x ( λ , μ ) = x ^ ( t , γ ) , λ ~ ( λ , μ ) = λ ^ ( t , γ ) x(\lambda,\mu)=\hat x(t,\gamma),\tilde\lambda(\lambda,\mu)=\hat\lambda(t,\gamma) x ( λ , μ ) = x ^ ( t , γ ) , λ ~ ( λ , μ ) = λ ^ ( t , γ ) ∀ ( λ , μ ) : ∣ λ − λ ∗ ∣ < δ μ , μ > max { μ ˉ , 1 / δ } \forall(\lambda,\mu):|\lambda-\lambda^*|<\delta\mu,\mu>\max\{\bar\mu,1/\delta\} ∀ ( λ , μ ) : ∣ λ − λ ∗ ∣ < δ μ , μ > max { μ ˉ , 1 / δ } ∣ x ( λ , μ ) − x ∗ ∣ ≤ 2 M ˉ ∣ λ − λ ∗ ∣ / μ , ∣ λ ~ ( λ , μ ) − λ ∗ ∣ ≤ 2 M ˉ ∣ λ − λ ∗ ∣ / μ . |x(\lambda,\mu)-x^*|\le2\bar M|\lambda-\lambda^*|/\mu,\quad|\tilde\lambda(\lambda,\mu)-\lambda^*|\le2\bar M|\lambda-\lambda^*|/\mu. ∣ x ( λ , μ ) − x ∗ ∣ ≤ 2 M ˉ ∣ λ − λ ∗ ∣ / μ , ∣ λ ~ ( λ , μ ) − λ ∗ ∣ ≤ 2 M ˉ ∣ λ − λ ∗ ∣ / μ . M = 2 M ˉ M=2\bar M M = 2 M ˉ ∀ ( λ , μ ) : ∣ λ − λ ∗ ∣ < δ μ , μ > max { μ ˉ , 1 / δ } \forall(\lambda,\mu):|\lambda-\lambda^*|<\delta\mu,\mu>\max\{\bar\mu,1/\delta\} ∀ ( λ , μ ) : ∣ λ − λ ∗ ∣ < δ μ , μ > max { μ ˉ , 1 / δ } x ( ⋅ , ⋅ ) x(\cdot,\cdot) x ( ⋅ , ⋅ ) M M M ∀ ( λ , μ ) : ∣ λ − λ ∗ ∣ < δ μ , ≤ ˉ μ ≤ max { μ ˉ , 1 / δ } \forall(\lambda,\mu):|\lambda-\lambda^*|<\delta\mu,\bar\le\mu\le\max\{\bar\mu,1/\delta\} ∀ ( λ , μ ) : ∣ λ − λ ∗ ∣ < δ μ , ≤ ˉ μ ≤ max { μ ˉ , 1 / δ }
此定理解释了增广Lagrange函数法的一些显著的性质.
∥ x k − x ∗ ∥ ≤ M ∥ λ k − λ ∗ ∥ / μ k \Vert x_k-x^*\Vert\le M\Vert\lambda^k-\lambda^*\Vert/\mu_k ∥ x k − x ∗ ∥ ≤ M ∥ λ k − λ ∗ ∥ / μ k λ k \lambda^k λ k μ k \mu_k μ k x k x_k x k x ∗ x^* x ∗ x k x_k x k μ k \mu_k μ k
∥ λ k + 1 − λ ∗ ∥ ≤ M ∥ λ k − λ ∗ ∥ / μ k \Vert\lambda^{k+1}-\lambda^*\Vert\le M\Vert\lambda^k-\lambda^*\Vert/\mu_k ∥ λ k + 1 − λ ∗ ∥ ≤ M ∥ λ k − λ ∗ ∥ / μ k μ k \mu_k μ k
无约束极小的二阶充分条件在第k k k
本节我们讨论实用的增广Lagrange函数法, 特别地, 是要处理带不等式约束 的情形. 我们分别讨论三种方式——界约束重构、约束线性化重构和无约束重构 .
给定一般的非线性规划问题, 我们可以增加松弛变量s i s_i s i c i ( x ) − s i = 0 , s i ≥ 0 , ∀ i ∈ I . c_i(x)-s_i=0,\quad s_i\ge0,\quad\forall i\in\mathcal{I}. c i ( x ) − s i = 0 , s i ≥ 0 , ∀ i ∈ I . l ≤ x ≤ u l\le x\le u l ≤ x ≤ u min x ∈ R n f ( x ) , s u b j e c t  t o  c i ( x ) = 0 , i = 1 , 2 , … , m , l ≤ x ≤ u . \min\limits_{x\in\mathbb{R}^n}f(x),\quad\mathrm{subject\,to\,}c_i(x)=0,i=1,2,\ldots,m,l\le x\le u. x ∈ R n min f ( x ) , s u b j e c t t o c i ( x ) = 0 , i = 1 , 2 , … , m , l ≤ x ≤ u . l l l − ∞ -\infty − ∞ u u u
界约束重构的增广Lagrange函数法the bound-constrained Lagrangian methods, BCL methods 只包含等式约束, 即L A ( x , λ ; μ ) = f ( x ) − ∑ i = 1 m λ i c i ( x ) + μ 2 ∑ i = 1 m c i 2 ( x ) . \mathcal{L}_A(x,\lambda;\mu)=f(x)-\sum_{i=1}^m\lambda_ic_i(x)+\frac{\mu}{2}\sum_{i=1}^mc_i^2(x). L A ( x , λ ; μ ) = f ( x ) − i = 1 ∑ m λ i c i ( x ) + 2 μ i = 1 ∑ m c i 2 ( x ) . min x L A ( x , λ ; μ ) , s u b j e c t  t o  ≤ x ≤ u . \min_x\mathcal{L}_A(x,\lambda;\mu),\quad\mathrm{subject\,to\,}\le x\le u. x min L A ( x , λ ; μ ) , s u b j e c t t o ≤ x ≤ u . λ \lambda λ μ \mu μ
一种求解带界约束重构非线性规划的技术是非线性投影梯度法 , 可见下一章. 考虑子问题的KKT条件, 我们会发现x x x x − P ( x − ∇ x L A ( x , λ ; μ ) , l , u ) = 0 , x-P(x-\nabla_x\mathcal{L}_A(x,\lambda;\mu),l,u)=0, x − P ( x − ∇ x L A ( x , λ ; μ ) , l , u ) = 0 , P ( g , l , u ) P(g,l,u) P ( g , l , u ) g ∈ R n g\in\mathbb{R}^n g ∈ R n [ l , u ] [l,u] [ l , u ] P ( g , l , u ) i = { l i , g i ≤ l i , g i , g i ∈ ( l i , u i ) , u i , g i ≥ u i , i = 1 , 2 , … , n . P(g,l,u)_i=\left\{\begin{array}{ll}l_i, & g_i\le l_i,\\g_i, & g_i\in(l_i,u_i),\\u_i, & g_i\ge u_i,\end{array}\right.i=1,2,\ldots,n. P ( g , l , u ) i = ⎩ ⎨ ⎧ l i , g i , u i , g i ≤ l i , g i ∈ ( l i , u i ) , g i ≥ u i , i = 1 , 2 , … , n .
算法4 (Bound-Constrained Lagrangian Method)x 0 x_0 x 0 λ 0 \lambda^0 λ 0 η ∗ \eta_* η ∗ w ∗ w_* w ∗ μ 0 = 10 , w 0 = 1 μ 0 , η 0 = 1 / μ 0 0.1 \mu_0=10,w_0=1\mu_0,\eta_0=1/\mu_0^{0.1} μ 0 = 1 0 , w 0 = 1 μ 0 , η 0 = 1 / μ 0 0 . 1 k = 0 , 1 , 2 , … k=0,1,2,\ldots k = 0 , 1 , 2 , … \quad\quad x k x_k x k ∥ x k − P ( x k − ∇ x L A ( x k , λ k ; μ k ) , l , u ) ∥ ≤ w k ; \Vert x_k-P(x_k-\nabla_x\mathcal{L}_A(x_k,\lambda^k;\mu_k),l,u)\Vert\le w_k; ∥ x k − P ( x k − ∇ x L A ( x k , λ k ; μ k ) , l , u ) ∥ ≤ w k ; \quad\quad ∥ c ( x k ) ∥ ≤ η k \Vert c(x_k)\Vert\le\eta_k ∥ c ( x k ) ∥ ≤ η k \quad\quad\quad\quad test for convergence )\quad\quad\quad\quad ∥ c ( x k ) ∥ ≤ η ∗ \Vert c(x_k)\Vert\le\eta_* ∥ c ( x k ) ∥ ≤ η ∗ ∥ x k − P ( x k − ∇ x L A ( x k , λ k ; μ k ) , l , u ) ∥ ≤ w ∗ \Vert x_k-P(x_k-\nabla_x\mathcal{L}_A(x_k,\lambda^k;\mu_k),l,u)\Vert\le w_* ∥ x k − P ( x k − ∇ x L A ( x k , λ k ; μ k ) , l , u ) ∥ ≤ w ∗ \quad\quad\quad\quad\quad\quad x k x_k x k \quad\quad\quad\quad \quad\quad\quad\quad update multipliers, tighten tolerances )\quad\quad\quad\quad λ k + 1 = λ k − μ k c ( x k ) \lambda^{k+1}=\lambda^k-\mu_kc(x_k) λ k + 1 = λ k − μ k c ( x k ) \quad\quad\quad\quad μ k + 1 = μ k \mu_{k+1}=\mu_k μ k + 1 = μ k \quad\quad\quad\quad η k + 1 = η k / μ k + 1 0.9 \eta_{k+1}=\eta_k/\mu_{k+1}^{0.9} η k + 1 = η k / μ k + 1 0 . 9 \quad\quad\quad\quad w k + 1 = w k / μ k + 1 w_{k+1}=w_k/\mu_{k+1} w k + 1 = w k / μ k + 1 \quad\quad \quad\quad\quad\quad increase penalty parameter, tighten tolerances )\quad\quad\quad\quad λ k + 1 = λ k \lambda^{k+1}=\lambda^k λ k + 1 = λ k \quad\quad\quad\quad μ k + 1 = 100 μ k \mu_{k+1}=100\mu_k μ k + 1 = 1 0 0 μ k \quad\quad\quad\quad η k + 1 = 1 / μ k + 1 0.1 \eta_{k+1}=1/\mu_{k+1}^{0.1} η k + 1 = 1 / μ k + 1 0 . 1 \quad\quad\quad\quad w k + 1 = 1 / μ k + 1 w_{k+1}=1/\mu_{k+1} w k + 1 = 1 / μ k + 1 \quad\quad
算法主要分岔位于近似求解子问题后, 算法将验证约束违反是否充分下降, 即验证条件∥ c ( x k ) ∥ ≤ η k . \Vert c(x_k)\Vert\le\eta_k. ∥ c ( x k ) ∥ ≤ η k . w k , η k w_k,\eta_k w k , η k
算法4中出现的常数0.1,0.9,100在某种程度上可以任取. 软件LANCELOT使用带信赖域的投影梯度法 求解界约束非线性子问题. 此时, 投影梯度法构建增广Lagrange函数L A \mathcal{L}_A L A 二次模型 , 并通过近似求解以下信赖域问题得到迭代步d d d min d 1 2 d T [ ∇ x x 2 L ( x k , λ k ) + μ k A k T A k ] d + ∇ x L A ( x k , λ k ; μ k ) T d s u b j e c t  t o l ≤ x k + d ≤ u , ∥ d ∥ ∞ ≤ Δ , \begin{array}{rl}\min\limits_d & \frac{1}{2}d^T\left[\nabla^2_{xx}\mathcal{L}(x_k,\lambda^k)+\mu_kA_k^TA_k\right]d+\nabla_x\mathcal{L}_A(x_k,\lambda^k;\mu_k)^Td\\\mathrm{subject\,to} & l\le x_k+d\le u,\quad\Vert d\Vert_{\infty}\le\Delta,\end{array} d min s u b j e c t t o 2 1 d T [ ∇ x x 2 L ( x k , λ k ) + μ k A k T A k ] d + ∇ x L A ( x k , λ k ; μ k ) T d l ≤ x k + d ≤ u , ∥ d ∥ ∞ ≤ Δ , A k = A ( x k ) A_k=A(x_k) A k = A ( x k ) Δ \Delta Δ − Δ e ≤ d ≤ Δ e -\Delta e\le d\le\Delta e − Δ e ≤ d ≤ Δ e
第一阶段, 做投影梯度线搜索以决定d d d
第二阶段, 使用CG极小化子问题, 其中不再牵涉第一阶段中被固定的分量.
这有点类似于第十六章 二次规划的投影梯度法. 第二阶段相当于在可行多面体的某个面上做的极小化. 这一算法的好处在于, 无需计算KKT矩阵或者约束Jacobi矩阵A k A_k A k
需要说明的是, Lagrange函数的Hessian阵∇ x x 2 L ( x k , λ k ) \nabla^2_{xx}\mathcal{L}(x_k,\lambda^k) ∇ x x 2 L ( x k , λ k ) 部分可分结构 , 以高效计算Lagrange函数的Hessian阵或拟牛顿近似矩阵, 可见第七章 .
约束线性化重构的增广Lagrange函数法linearly constrained Lagragian methods, LCL methods 的主要想法是, 在线性化 原约束的限制下, 极小化Lagrange函数(或增广Lagrange函数), 产生迭代步. 对于上一小节中的一般非线性规划min x ∈ R n f ( x ) , s u b j e c t  t o  c i ( x ) = 0 , i = 1 , 2 , … , m , l ≤ x ≤ u , \min_{x\in\mathbb{R}^n}f(x),\quad\mathrm{subject\,to\,}c_i(x)=0,i=1,2,\ldots,m,l\le x\le u, x ∈ R n min f ( x ) , s u b j e c t t o c i ( x ) = 0 , i = 1 , 2 , … , m , l ≤ x ≤ u , min x F k ( x ) s u b j e c t  t o c ( x k ) + A k ( x − x k ) = 0 , l ≤ x ≤ u . \begin{array}{rl}\min\limits_x & F_k(x)\\\mathrm{subject\,to} & c(x_k)+A_k(x-x_k)=0,\quad l\le x\le u.\end{array} x min s u b j e c t t o F k ( x ) c ( x k ) + A k ( x − x k ) = 0 , l ≤ x ≤ u . F k ( x ) F_k(x) F k ( x ) F k ( x ) = f ( x ) − ∑ i = 1 m λ i k c ˉ i k ( x ) , F_k(x)=f(x)-\sum_{i=1}^m\lambda_i^k\bar{c}_i^k(x), F k ( x ) = f ( x ) − i = 1 ∑ m λ i k c ˉ i k ( x ) , λ k \lambda^k λ k c ˉ i k ( x ) \bar{c}_i^k(x) c ˉ i k ( x ) c i ( x ) c_i(x) c i ( x ) x k x_k x k 差函数 , 即c ˉ i k ( x ) = c i ( x ) − c i ( x k ) − ∇ c i ( x k ) T ( x − x k ) . \bar{c}_i^k(x)=c_i(x)-c_i(x_k)-\nabla c_i(x_k)^T(x-x_k). c ˉ i k ( x ) = c i ( x ) − c i ( x k ) − ∇ c i ( x k ) T ( x − x k ) . x k x_k x k x ∗ x^* x ∗ 对应于子问题中等式约束的Lagrange乘子将收敛于最优乘子 . 因此, 我们可设λ k \lambda^k λ k
现在LCL方法定义F k F_k F k F k ( x ) = f ( x ) − ∑ i = 1 m λ i k c ˉ i k ( x ) + μ 2 ∑ i = 1 m [ c ˉ i k ( x ) ] 2 . F_k(x)=f(x)-\sum_{i=1}^m\lambda_i^k\bar{c}_i^k(x)+\frac{\mu}{2}\sum_{i=1}^m[\bar c_i^k(x)]^2. F k ( x ) = f ( x ) − i = 1 ∑ m λ i k c ˉ i k ( x ) + 2 μ i = 1 ∑ m [ c ˉ i k ( x ) ] 2 . 更好的全局收敛效果 .
可以看到, 上述F k F_k F k c i ( x ) c_i(x) c i ( x ) c ˉ i k ( x ) \bar c_i^k(x) c ˉ i k ( x ) 仅捕捉c i c_i c i . 线性化约束重构子问题与原本的增广Lagrange函数子问题的不同则在于, 前者要求新迭代点x x x c ˉ i k \bar c_i^k c ˉ i k c i c_i c i μ \mu μ
因c ˉ i k ( x ) \bar c_i^k(x) c ˉ i k ( x ) x = x k x=x_k x = x k ∇ F k ( x k ) = ∇ f ( x k ) \nabla F_k(x_k)=\nabla f(x_k) ∇ F k ( x k ) = ∇ f ( x k ) F k F_k F k
MINOS包使用增广形式的模型函数, 用既约投影梯度法求解线性化子问题, 其中对F k F_k F k
利用近邻点proximal point 方法, 我们可以得到带不等式约束问题的无约束化重构增广Lagrange函数子问题. 为说明简便, 假设问题无等式约束, 于是我们可以将原问题等价地写作无约束优化问题min x ∈ R n F ( x ) , \min\limits_{x\in\mathbb{R}^n}F(x), x ∈ R n min F ( x ) , F ( x ) = max λ ≥ 0 { f ( x ) − ∑ i ∈ I λ i c i ( x ) } = { f ( x ) , x  i s  f e a s i b l e , ∞ , o t h e r w i s e . F(x)=\max_{\lambda\ge0}\left\{f(x)-\sum_{i\in\mathcal{I}}\lambda_ic_i(x)\right\}=\left\{\begin{array}{ll}f(x), & x\,is\,feasible,\\\infty, & otherwise.\end{array}\right. F ( x ) = λ ≥ 0 max { f ( x ) − i ∈ I ∑ λ i c i ( x ) } = { f ( x ) , ∞ , x i s f e a s i b l e , o t h e r w i s e . F F F F F F F ^ ( x ; λ k , μ k ) \hat F(x;\lambda^k,\mu_k) F ^ ( x ; λ k , μ k ) F F F μ k \mu_k μ k λ k \lambda^k λ k F ^ ( x ; λ k , μ k ) = max λ ≥ 0 { f ( x ) − ∑ i ∈ I λ i c i ( x ) − 1 2 μ k ∑ i ∈ I ( λ i − λ i k ) 2 } . \hat F(x;\lambda^k,\mu_k)=\max_{\lambda\ge0}\left\{f(x)-\sum_{i\in\mathcal{I}}\lambda_ic_i(x)-\frac{1}{2\mu_k}\sum_{i\in\mathcal{I}}(\lambda_i-\lambda_i^k)^2\right\}. F ^ ( x ; λ k , μ k ) = λ ≥ 0 max { f ( x ) − i ∈ I ∑ λ i c i ( x ) − 2 μ k 1 i ∈ I ∑ ( λ i − λ i k ) 2 } . λ \lambda λ λ k \lambda^k λ k 它会使新的乘子λ \lambda λ λ k \lambda^k λ k . 因F ^ \hat F F ^ λ \lambda λ 显式解 λ i = { 0 , − c i ( x ) + λ i k / μ k ≤ 0 , λ i k − μ k c i ( x ) , o t h e r w i s e . \lambda_i=\left\{\begin{array}{ll}0, & -c_i(x)+\lambda_i^k/\mu_k\le0,\\\lambda_i^k-\mu_kc_i(x), & otherwise.\end{array}\right. λ i = { 0 , λ i k − μ k c i ( x ) , − c i ( x ) + λ i k / μ k ≤ 0 , o t h e r w i s e . F ^ \hat F F ^ F ^ ( x ; λ k , μ k ) = f ( x ) + ∑ i ∈ I ψ ( c i ( x ) , λ i k ; μ k ) , \hat F(x;\lambda^k,\mu_k)=f(x)+\sum_{i\in\mathcal{I}}\psi(c_i(x),\lambda_i^k;\mu_k), F ^ ( x ; λ k , μ k ) = f ( x ) + i ∈ I ∑ ψ ( c i ( x ) , λ i k ; μ k ) , ψ \psi ψ ψ ( t , σ ; μ ) = d e f { − σ t + μ 2 t 2 , t − σ / μ ≤ 0 , − 1 2 μ σ 2 , o t h e r w i s e . \psi(t,\sigma;\mu)\xlongequal{def}\left\{\begin{array}{ll}-\sigma t+\frac{\mu}{2}t^2, & t-\sigma/\mu\le0,\\-\frac{1}{2\mu}\sigma^2, & otherwise.\end{array}\right. ψ ( t , σ ; μ ) d e f { − σ t + 2 μ t 2 , − 2 μ 1 σ 2 , t − σ / μ ≤ 0 , o t h e r w i s e . x x x F ^ ( x ; λ k , μ k ) \hat F(x;\lambda^k,\mu_k) F ^ ( x ; λ k , μ k ) x k x_k x k F F F L A \mathcal{L}_A L A 向不等式约束情形的推广 . 不过, 由于其性质还未得到验证, 此无约束化重构并不如界约束重构和约束线性化重构一般, 具有常用软件包的内嵌.
在约束数目较小时, 人们通常使用二次罚函数法 . 事实上, 有时仅需对一个较大的μ \mu μ Q ( x ; μ ) Q(x;\mu) Q ( x ; μ ) μ \mu μ
尽管二次罚函数更直观、简洁, 但第3节与第4节的增广Lagrange函数法通常更受青睐 . 子问题的求解通常并不太难, 且乘子估计的引入避免了病态子问题的出现. 不过二次罚函数仍然作为许多算法正则化(regularization)的一种重要手段 , 譬如SQP.
一般情形的l 1 l_1 l 1 l 1 l_1 l 1 l 1 l_1 l 1
近些年l 1 l_1 l 1 被成功用于求解一些困难的问题 , 例如带互补约束的数学规划(mathematical programs with complementarity constraints, MPCCs), 其中约束不满足标准的约束规范. 将这些问题约束吸收为罚项而不是线性化, 之后后再使用诸如SQP或内点法, 我们能够扩展这些其他方法的应用面 . 软件包SNOPT在SQP算法中使用l 1 l_1 l 1 防护策略 , 以防二次模型不可行或无界, 亦或有无界乘子.
增广Lagrange函数法由于其简洁性, 已受多年关注. MINOS和LANCELOT则是实施增广Lagrange函数法最好的软件包 . 它们也适用于大型非线性规划问题. 一般层面上, MINOS的LCL和LANCELOT的BCL有共同的性质. 但它们在计算迭代步的子问题和求解子问题的技术上具有较大差异:
MINOS使用既约空间法处理线性化约束, 并且使用(稠密)拟牛顿近似代替Lagrange函数的Hessian. 因此, MINOS对具有较小自由度的问题是最合适的 .
LANCELOT则更适用于约束较少的情形 . 正如第4节所说, LANCELOT无需约束Jacobi矩阵A A A
PENNON软件包也基于增广Lagrange函数法, 它在处理半定矩阵约束时较有优势 .
界约束重构和无约束重构Lagrange函数法的缺陷是, 它们增加了约束的平方, 让问题变得更复杂 . 这时, 我们只有通过极小化增广Lagrange函数才能获得可行性的改善 . 相反, LCL则在约束上计算牛顿类的迭代步, 稳步改善可行性 . 因此, 不难预见, 在带线性约束的问题上MINOS比LANCELOT更具优势 .
从第3节的增广Lagrange函数也可构造光滑精确罚函数(smooth exact penalty functions) , 但这些就过于复杂了. 例如我们提过等式约束的Fletcher光滑精确罚函数:ϕ F ( x ; μ ) = f ( x ) − λ ( x ) T c ( x ) + μ 2 ∑ i ∈ E c i ( x ) 2 . \phi_F(x;\mu)=f(x)-\lambda(x)^Tc(x)+\frac{\mu}{2}\sum_{i\in\mathcal{E}}c_i(x)^2. ϕ F ( x ; μ ) = f ( x ) − λ ( x ) T c ( x ) + 2 μ i ∈ E ∑ c i ( x ) 2 . λ ( x ) \lambda(x) λ ( x ) λ ( x ) = [ A ( x ) A ( x ) T ] − 1 A ( x ) ∇ f ( x ) = ( A T ) † ∇ f ( x ) . \lambda(x)=[A(x)A(x)^T]^{-1}A(x)\nabla f(x)=\left(A^T\right)^{\dagger}\nabla f(x). λ ( x ) = [ A ( x ) A ( x ) T ] − 1 A ( x ) ∇ f ( x ) = ( A T ) † ∇ f ( x ) . ϕ F \phi_F ϕ F ϕ F \phi_F ϕ F λ ( x ) \lambda(x) λ ( x ) A ( x ) A(x) A ( x ) λ ( x ) \lambda(x) λ ( x ) A ( x ) A(x) A ( x ) λ \lambda λ
Han S P , Mangasarian O L . Exact penalty functions in nonlinear programming[J]. Mathematical Programming, 1979, 17(1):251-269. ↩︎
Han S P , Mangasarian O L . Exact penalty functions in nonlinear programming[J]. Mathematical Programming, 1979, 17(1):251-269. ↩︎