Chapter11: Interior-point methods
11.1 Inequality constrained minimization problems
In this chapter we discuss interior-point methods for solving convex optimization problems that include inequality constraints,
m
i
n
i
m
i
z
e
f
0
(
x
)
s
u
b
j
e
c
t
t
o
f
i
(
x
)
≤
0
,
i
=
1
,
.
.
.
m
A
x
=
b
\begin{aligned} {\rm minimize} \ \ \ \ & f_0(x)\\ {\rm subject \ to} \ \ \ \ & f_i(x)\leq0, \ \ i=1,...m \\ & Ax=b \end{aligned}
minimize subject to f0(x)fi(x)≤0, i=1,...mAx=bwhere
f
0
,
.
.
.
,
f
m
:
R
n
→
R
f_0 , . . . , f_m : \mathbf{R}^n → \mathbf{R}
f0,...,fm:Rn→R are convex and twice continuously differentiable, and
A
∈
R
p
×
n
A ∈ \mathbf{R}^{p×n}
A∈Rp×n with
r
a
n
k
A
=
p
<
n
\mathbf{rank} \ A = p < n
rank A=p<n.
11.2 Logarithmic barrier function and central path
Our first step is to rewrite the problem, making the inequality constraints implicit in the objective:
m
i
n
i
m
i
z
e
f
0
(
x
)
+
∑
i
=
1
m
I
−
(
f
i
(
x
)
)
s
u
b
j
e
c
t
t
o
A
x
=
b
\begin{aligned} {\rm minimize} \ \ \ \ & f_0(x) + \sum_{i=1}^mI_-(f_i(x))\\ {\rm subject \ to} \ \ \ \ & Ax=b \\ \end{aligned}
minimize subject to f0(x)+i=1∑mI−(fi(x))Ax=bwhere
I
−
I_-
I− is the indicator function:
I
−
(
u
)
=
{
0
u
<
0
∞
u
≥
0
I_-(u) = \left\{ \begin{array}{cc} 0 & u<0 \\ \infty & u\geq0 \end{array} \right.
I−(u)={0∞u<0u≥0
11.2.1 Logarithmic barrier
The basic idea of the barrier method is to approximate the indicator function
I
−
I_−
I− by the function
I
^
−
(
u
)
=
−
1
t
log
(
−
u
)
,
d
o
m
I
^
−
=
−
R
+
+
\hat{I}_-(u)=-\frac{1}{t}\log{(-u)}, \ \ \ \ \mathbf{dom} \ \hat{I}_-=-\mathbf{R}_{++}
I^−(u)=−t1log(−u), dom I^−=−R++The objective here is convex, since
−
1
t
log
(
−
u
)
-\frac{1}{t}\log{(-u)}
−t1log(−u) is convex and increasing in
u
u
u, and differentiable.
The function
ϕ
(
x
)
=
−
∑
i
=
1
m
log
(
−
f
i
(
x
)
)
\phi(x)=-\sum_{i=1}^m\log {(-f_i(x))}
ϕ(x)=−i=1∑mlog(−fi(x))
with
d
o
m
ϕ
=
{
x
∈
R
n
∣
f
i
(
x
)
<
0
,
i
=
1
,
.
.
.
,
m
}
\mathbf{dom} \ \phi = \{x ∈ \mathbf{R}^n | f_i(x) < 0, i = 1,...,m\}
dom ϕ={x∈Rn∣fi(x)<0,i=1,...,m}, is called the logarithmic barrier or log barrier for the problem.
11.2.2 Central path
We now consider in more detail the minimization problem. It will simplify notation later on if we multiply the objective by
t
t
t, and consider the equivalent problem
m
i
n
i
m
i
z
e
t
f
0
(
x
)
+
ϕ
(
x
)
s
u
b
j
e
c
t
t
o
A
x
=
b
\begin{aligned} {\rm minimize} \ \ \ \ & tf_0(x)+\phi(x)\\ {\rm subject \ to} \ \ \ \ & Ax=b \end{aligned}
minimize subject to tf0(x)+ϕ(x)Ax=bFor
t
>
0
t > 0
t>0 we define
x
⋆
(
t
)
x^⋆(t)
x⋆(t) as the solution. The central path associated with problem is defined as the set of points
x
⋆
(
t
)
,
t
>
0
x^⋆(t), t > 0
x⋆(t),t>0, which we call the central points.
11.3 The barrier method
We have seen that the point
x
⋆
(
t
)
x^⋆(t)
x⋆(t) is
m
/
t
m/t
m/t-suboptimal, and that a certificate of this accuracy is provided by the dual feasible pair
λ
⋆
(
t
)
λ^⋆(t)
λ⋆(t),
ν
⋆
(
t
)
\nu^⋆(t)
ν⋆(t). This suggests a very straightforward method for solving the original problem with a guaranteed specified accuracy
ϵ
\epsilon
ϵ: We simply take
t
=
m
ϵ
t = \frac{m}{\epsilon}
t=ϵm and solve the equality constrained
m
i
n
i
m
i
z
e
m
ϵ
f
0
(
x
)
+
ϕ
(
x
)
s
u
b
j
e
c
t
t
o
A
x
=
b
\begin{aligned} {\rm minimize} \ \ \ \ & \frac{m}{\epsilon}f_0(x) + \phi(x)\\ {\rm subject \ to} \ \ \ \ & Ax=b \\ \end{aligned}
minimize subject to ϵmf0(x)+ϕ(x)Ax=b
11.3.1 The barrier method
We compute
x
⋆
(
t
)
x^⋆(t)
x⋆(t) for a sequence of increasing values of
t
t
t, until
t
≥
m
ϵ
t ≥ \frac{m}{\epsilon}
t≥ϵm, which guarantees that we have an
ϵ
\epsilon
ϵ-suboptimal solution of the original problem.
11.3.2 Examples
11.3.3 Convergence analysis
11.3.4 Newton step for modified KKT equations
Consider the modified KKT equations
∇
f
0
(
x
)
+
∑
i
=
1
m
λ
i
f
i
(
x
)
+
A
T
ν
=
0
−
λ
i
f
i
(
x
)
=
1
t
,
i
=
1
,
.
.
.
,
m
A
x
=
b
\begin{aligned} \nabla f_0(x)+\sum_{i=1}^{m}\lambda_if_i(x)+A^T\nu &= 0 \\ -\lambda_if_i(x) &= \frac{1}{t}, \ \ i=1,...,m \\ Ax &= b \end{aligned}
∇f0(x)+i=1∑mλifi(x)+ATν−λifi(x)Ax=0=t1, i=1,...,m=bUse
λ
i
=
1
t
f
i
(
x
)
\lambda_i=\frac{1}{tf_i(x)}
λi=tfi(x)1, then we get
∇
f
0
(
x
)
+
∑
i
=
1
m
1
−
t
f
i
(
x
)
∇
f
i
(
x
)
+
A
T
ν
=
0
,
A
x
=
b
\nabla f_0(x)+\sum_{i=1}^{m}\frac{1}{-tf_i(x)}\nabla f_i(x)+A^T\nu=0, \ \ \ \ Ax=b
∇f0(x)+i=1∑m−tfi(x)1∇fi(x)+ATν=0, Ax=bFor
v
v
v small, we use the Taylor approximation to approximate
∇
f
0
(
x
)
+
∑
i
=
1
m
λ
i
f
i
(
x
)
\nabla f_0(x)+\sum_{i=1}^{m}\lambda_if_i(x)
∇f0(x)+∑i=1mλifi(x), then we get
H
v
+
A
T
ν
=
−
g
,
A
v
=
0
,
Hv + A^T ν = −g, \ \ Av = 0,
Hv+ATν=−g, Av=0,where
H
=
∇
2
f
0
(
x
)
+
∑
i
=
1
m
1
−
t
f
i
(
x
)
∇
2
f
i
(
x
)
+
∑
i
=
1
m
1
t
f
i
2
(
x
)
∇
f
i
(
x
)
∇
f
i
(
x
)
T
g
=
∇
f
0
(
x
)
+
∑
i
=
1
m
1
−
t
f
i
(
x
)
∇
f
i
(
x
)
\begin{aligned} H &= \nabla^2f_0(x) + \sum_{i=1}^{m}\frac{1}{-tf_i(x)}\nabla^2 f_i(x)+\sum_{i=1}^m\frac{1}{tf_i^2(x)}\nabla f_i(x)\nabla f_i(x)^T \\ g &= \nabla f_0(x)+\sum_{i=1}^{m}\frac{1}{-tf_i(x)}\nabla f_i(x) \end{aligned}
Hg=∇2f0(x)+i=1∑m−tfi(x)1∇2fi(x)+i=1∑mtfi2(x)1∇fi(x)∇fi(x)T=∇f0(x)+i=1∑m−tfi(x)1∇fi(x)Observe that
H
=
∇
2
f
0
(
x
)
+
1
t
∇
2
ϕ
(
x
)
,
g
=
∇
f
0
(
x
)
+
1
t
∇
ϕ
(
x
)
,
H = ∇^2f_0(x) + \frac{1}{t}∇^2\phi(x), \ \ g = ∇f_0(x) + \frac{1}{t}∇\phi(x),
H=∇2f0(x)+t1∇2ϕ(x), g=∇f0(x)+t1∇ϕ(x),We get that
v
=
∆
x
n
t
,
ν
=
1
t
ν
n
t
v = ∆x_{\rm nt}, \ \ ν = \frac{1}{t}ν_{\rm nt}
v=∆xnt, ν=t1νnt
11.4 Feasibility and phase I methods
When such a point is not known, the barrier method is preceded by a preliminary stage, called phase I, in which a strictly feasible point is computed (or the constraints are found to be infeasible). The strictly feasible point found during phase I is then used as the starting point for the barrier method, which is called the phase II stage.
11.4.1 Basic phase I method
Our goal is to find a strictly feasible solution of these inequalities and equalities, or determine that none exists. To do this we form the following optimization problem:
m
i
n
i
m
i
z
e
s
s
u
b
j
e
c
t
t
o
f
i
(
x
)
≤
s
,
i
=
1
,
.
.
.
m
A
x
=
b
\begin{aligned} {\rm minimize} \ \ \ \ & s\\ {\rm subject \ to} \ \ \ \ & f_i(x)\leq s, \ \ i=1,...m \\ & Ax=b \end{aligned}
minimize subject to sfi(x)≤s, i=1,...mAx=b
11.4.2 Phase I via infeasible start Newton method
We can also carry out the phase I stage using an infeasible start Newton method, applied to a modified version of the original problem. First express the problem in such form:
m
i
n
i
m
i
z
e
f
0
(
x
)
s
u
b
j
e
c
t
t
o
f
i
(
x
)
≤
s
,
i
=
1
,
.
.
.
m
A
x
=
b
,
s
=
0
\begin{aligned} {\rm minimize} \ \ \ \ & f_0(x)\\ {\rm subject \ to} \ \ \ \ & f_i(x)\leq s, \ \ i=1,...m \\ & Ax=b, \ \ \ \ s=0 \end{aligned}
minimize subject to f0(x)fi(x)≤s, i=1,...mAx=b, s=0
Provided the problem is strictly feasible, the infeasible start Newton method will eventually take an undamped step, and thereafter we will have
s
=
0
s = 0
s=0, i.e.,
x
x
x strictly feasible.