Glow: Generative Flow with Invertible 1x1 Convolutions

Glow: Generative Flow with Invertible 1×$\times$1 Convolutions

Diederik P. Kingma, Prafulla Dhariwal

Abstract

flow-based generative models: tractability of the exact log-likelihood, tractability of exact latent-variable inference, and parallelizability of both training and synthesis
Glow: a simple type of generative flow using an invertible 1 ×$\times$ 1 convolution
capable of efficient realistic-looking synthesis and manipulation of large images
code:https://github.com/openai/glow

Introduction

two major unsolved problems of machine learning:
(1) data-efficiency: the ability to learn from few datapoints, like humans;
(2) generalization: robustness to changes of the task or its context

generative models:
(1) learning realistic world models
(2) learning meaningful features of the input while requiring little or no human supervision or labeling

generative model⎧⎩⎨⎪⎪⎪⎪likelihood-based methods⎧⎩⎨autoregressive modelVAEflow-based generative modelGAN$$\text{generative model}\{\begin{array}{l}\text{likelihood-based methods}\{\begin{array}{l}\text{autoregressive model}\\ \text{VAE}\\ \text{flow-based generative model}\end{array}\\ \text{GAN}\end{array}$$

merits of flow-based generative models:
1. Exact latent-variable inference and log-likelihood evaluation
2. Efficient inference and efficient synthesis
3. Useful latent space for downstream tasks
4. Significant potential for memory savings

Background: Flow-based Generative Models

x$x$: high dimensional random vector with unknown true distribution p(x)$p(x)$
D$D$: i.i.d dataset from p(x)$p(x)$
pθ(x)$p_{\theta }(x)$: model
log-likelihood objective(the expected compression cost):

minθL(D)=E(−logpθ(x))=1|D|∑x∈D−logpθ(x)$$\underset{\theta }{min}L(D)=E(-\log p_{\theta }(x))=\frac{1}{|D|}\sum _{x\in D}-\log p_{\theta }(x)$$

z$z$: latent variable with tractable/simple pdf pθ(z)$p_{\theta }(z)$, e.g. N(μ,Σ)$N(\mu ,\mathrm{\Sigma })$
x=gθ(z)$x=g_{\theta }(z)$, gθ$g_{\theta }$ is bijective, g−1θ=fθ$g_{\theta }^{-1}=f_{\theta }$
f=f1∘f2∘...∘fK,h1=f1(x),h2=f2(h1),...,z=fK(hK−1)$f=f_1\circ f_2\circ ...\circ f_K,h_1=f_1(x),h_2=f_2(h_1),...,z=f_K(h_{K-1})$
flow: x↔f1h1↔f2h2...↔fKz$x\stackrel{f_1}{\leftrightarrow }{h}_1\stackrel{f_2}{\leftrightarrow }{h}_2...\stackrel{f_K}{\leftrightarrow }z$
根据随机变量函数的概率密度公式，有

logpθ(x)=logpθ(z)+∑i=1Klog|det(∂fi∂hi−1)|$$\log p_{\theta }(x)=\log p_{\theta }(z)+\sum _{i=1}^K\log |det(\frac{\mathrm{\partial }{f}_i}{\mathrm{\partial }{h}_{i-1}})|$$

若∂fi∂hi−1$\frac{\mathrm{\partial }{f}_i}{\mathrm{\partial }{h}_{i-1}}$为三角矩阵，则det(∂fi∂hi−1)=∏diag(∂fi∂hi−1)$det(\frac{\mathrm{\partial }{f}_i}{\mathrm{\partial }{h}_{i-1}})=\prod \text{diag}(\frac{\mathrm{\partial }{f}_i}{\mathrm{\partial }{h}_{i-1}})$

Proposed Generative Flow

提出的新flow结构如下图所示，其中每一步flow包括一个actnorm, invertible 1×$\times$1 convolution和affine coupling layer，最终得到(b)中的multi-scale architecture，其中K$K$为flow深度，L$L$为level数

Actnorm: scale and bias layer with data dependent initialization

activation normalizaton, performs an affine transformation of the activations using a scale and bias parameter per channel
initialize: the post-actnorm activations per-channel have zero mean and unit variance given an initial minibatch of data (data dependent initialization)

Invertible 1 ×$\times$ 1 convolution

h∈Rh×w×c,W∈R1×1×c×c$h\in R^{h\times w\times c},W\in R^{1\times 1\times c\times c}$

log|det(∂conv2d(h;W)∂h)|=hwlog|det(W)|$$\log |det(\frac{\mathrm{\partial }\text{conv2d}(h;W)}{\mathrm{\partial }h})|=hw\log |det(W)|$$

det(W):O(c3)$det(W):O(c^3)$
initialize the weights W$W$ as a random rotation matrix, having a log-determinant of 0

W=PL(U+diag(s))$$W=PL(U+\text{diag}(s))$$

P$P$: permutation matrix
L$L$: lower triangular matrix with ones on the diagonal
U$U$: upper triangular matrix with zeros on the diagonal
s$s$: vector

log|det(W)|=sum(log|s|)$$\log |det(W)|=\text{sum}(\log |s|)$$

Affine Coupling Layers

A powerful reversible transformation where the forward function, the reverse function and the logdeterminant are computationally efficient
Zero initialization: initialize the last convolution of each NN() with zeros, such that each affine coupling layer initially performs an identity function
Split and concatenation: split() splits h$h$ the input tensor into two halves along the channel dimension, concat() concatenation into a single tensor
Permutation: ensures that after sufficient steps of flow, each dimensions can affect every other dimension

Related Work

GAN

Quantitative Experiments

NN(): conv3×3×512$3\times 3\times 512$+relu+conv1×1×512$1\times 1\times 512$+relu+conv1×1×3$1\times 1\times 3$
K=32,L=3$K=32,L=3$
3种permutation方法: a reversing operation as described in the RealNVP, a fixed random permutation, and our invertible 1×1$1\times 1$ convolution

Qualitative Experiments

K=32,L=6$K=32,L=6$
sampling from a reduced-temperature model often results in higher-quality samples, pθ,T(x)∝pT2θ(x)$p_{\theta ,T}(x)\propto p_{\theta }^{T^2}(x)$
Synthesis and Interpolation: take a pair of real images, encode them with the encoder, and linearly interpolate between the latents to obtain samples. extremely high quality
Semantic Manipulation: calculate the average latent vector zpos$z_{pos}$ for images with the attribute and zneg$z_{neg}$ for images without, use the difference (zpos−zneg)$(z_{pos}-z_{neg})$ as a direction for manipulating. 需要标注信息

Conclusion

first likelihood-based model in the literature that can efficiently synthesize high-resolution natural images
比auto encoder不知道高到哪去了