LADN: Local Adversarial Disentangling Network for Facial Makeup and De-Makeup（ICCV19）

3. LADN

3.1. Problem Formulation

定义domain，$X\subset \mathbb{R}^{H\times W\times 3}$X⊂RH×W×3为before-makeup faces，$Y\subset \mathbb{R}^{H\times W\times 3}$Y⊂RH×W×3为after-makeup faces

数据集包括$\left \{ x_i \right \}_{i=1,\cdots M}, x_i\in X${xi​}i=1,⋯M​,xi​∈X以及$\left \{ y_i \right \}_{i=1,\cdots N}, y_i\in Y${yi​}i=1,⋯N​,yi​∈Y

goal：学习化妆的mapping function $\Phi_Y: x_i, y_j\rightarrow\tilde{y}_i$ΦY​:xi​,yj​→y~​i​，以及卸妆的mapping function $\Phi_X: y_j\rightarrow\tilde{x}_j$ΦX​:yj​→x~j​
值得注意的是，makeup的过程是需要reference image作为condition，而makeup removal的过程不需要condition

3.2. Network Architecture

核心idea：separate the makeup style latent variable from non-makup features (identity, facial structure, head pose, etc.) and generate new images through recombination of these latent variables.
因此本文借鉴了一个disentanglement framework DRIT

定义：attribute space $A$A that captures the makeup style latent、content space $S$S which includes the non-makeup features

网络：2个domain的content encoders $\left \{ E_X^c, E_Y^c \right \}${EXc​,EYc​}，style encoders $\left \{ E_X^a, E_Y^a \right \}${EXa​,EYa​}（上标为$a$a，其实就是attribute encoders），generators $\left \{ G_X, G_Y \right \}${GX​,GY​}

使用Encoder网络分别对$x_i$xi​和$y_j$yj​提取attribute and content features
$E_X^a(x_i)=A_i \quad E_Y^a(y_j)=A_j \\ E_X^c(x_i)=C_i \quad E_X^c(y_j)=C_j$EXa​(xi​)=Ai​EYa​(yj​)=Aj​EXc​(xi​)=Ci​EXc​(yj​)=Cj​
然后送入generators生成de-makeup result $\tilde{x}_j$x~j​和makeup transfer result $\tilde{y}_i$y~​i​
$G_X\left ( A_i, C_j \right )=\tilde{x}_j \quad G_Y\left ( A_j, C_i \right )=\tilde{y}_i \qquad(1)$GX​(Ai​,Cj​)=x~j​GY​(Aj​,Ci​)=y~​i​(1)

The encoders and decoders are designed with a U-Net structure, The latent variables $A$A, $C$C are concatenated at the bottleneck and skip connections are used between the content encoder and generator. This structure can help retain more identity details from the source in the generated image.

对于2个domain，设置2个判别器$\left \{ D_X, D_Y \right \}${DX​,DY​}，从而有adversarial loss $L_{domain}^{adv}=L_X^{adv}+L_Y^{adv}$Ldomainadv​=LXadv​+LYadv​
$\begin{aligned} &L_X^{adv}=\mathbb{E}_{x\sim P_X}\left [ \log D_X(x) \right ] + \mathbb{E}_{\tilde{x}\sim G_X}\left [ \log\left ( 1-D_X\left ( \tilde{x} \right ) \right ) \right ] \\ &L_Y^{adv}=\mathbb{E}_{y\sim P_Y}\left [ \log D_Y(y) \right ] + \mathbb{E}_{\tilde{y}\sim G_Y}\left [ \log\left ( 1-D_Y\left ( \tilde{y} \right ) \right ) \right ] \qquad(2) \end{aligned}$​LXadv​=Ex∼PX​​[logDX​(x)]+Ex~∼GX​​[log(1−DX​(x~))]LYadv​=Ey∼PY​​[logDY​(y)]+Ey~​∼GY​​[log(1−DY​(y~​))](2)​

3.3. Local Style Discriminator

对于每一个unpaired的样本$\left ( x_i, y_j \right )$(xi​,yj​)，人工生成一个synthetic ground truth $W\left ( x_i, y_j \right )$W(xi​,yj​)，方法是warping and blending $y_j$yj​ onto $x_i$xi​ according
to their facial landmarks

当然warping result是有artifacts的，需要使用网络来fix

注：这个生成的代价是否太大了，假设$X$X有1k幅图像，$Y$Y有1k幅图像，那么两两交叉，需要生成100万幅图像

好处是：Although the synthetic results cannot serve as the real ground truth of the final results, they can provide guidance to the makeup transfer network on what the generated results should look like.