NICE- Non-linear Independent Components Estimation: Insights and Implementation in Keras

Keras implementation can be found here.

Flow-based deep generative models have not gained much attention in the research community when compared to GANs or VAEs. This post discusses a flow-based model called NICE, its advantages over the other generative models and finally an implementation in Keras.

While VAEs use an encoder that finds only an approximation of the latent variable corresponding to a datapoint, GANs doesnt even have an encoder to infer latents. In flow-based models, the latent variables can be infered exactly without any approximation. Flow-based models make use of reversible architecture (which will be explained below) which enables accurate inference, in addition to providing optimization over the exact log-likelihood of the data instead of a lower bound of it.

Flow-based generative models

In flow-based approaches the generative process for a datapoint $\mathbf{x}$ is defined as:

$$\begin{gathered} \mathbf{z}\sim p(\mathbf{z}) \\ \mathbf{x}=\mathbf{g}(\mathbf{z}) \end{gathered}$$

where $\mathbf{z}$ is the latent variable with some tractable density $p(\mathbf{z})$ such as standard normal: $p(\mathbf{z})=\mathcal{N}(\mathbf{z};\mathbf{0},\mathbf{I})$. The key aspect in flow-based approaches is that the function $\mathbf{g}$ is an invertible function (also called bijective) such that for every datapoint $\mathbf{x}$, the latent variable can be inferred by $\mathbf{z}=\mathbf{f}(\mathbf{x})={\mathbf{g}}^{-1}(\mathbf{x})$.

The function $\mathbf{f}$ which maps the datapoint to the corresponding latent-variable is realised using a deep neural network. In addition, notice that $\mathbf{g}$ is the inverse of $\mathbf{f}$, therefore the neural network architecture should be carefully defined such that it is reversible or invertible. Therefore, we focus on functions where $\mathbf{f}$ (and $\mathbf{g}$) is composed of a sequence of invertible transformations: $\mathbf{f}={\mathbf{f}}_{\mathbf{1}}\circ {\mathbf{f}}_{\mathbf{2}}\circ \cdots \circ {\mathbf{f}}_{\mathbf{K}}$, such that the mapping from $\mathbf{x}$ to $\mathbf{z}$ can be written as: $$\mathbf{x}\stackrel{{\mathbf{f}}_{\mathbf{1}}}{⟷}{\mathbf{h}}_{\mathbf{1}}\stackrel{{\mathbf{f}}_{\mathbf{1}}}{⟷}{\mathbf{h}}_{\mathbf{2}}\cdots \stackrel{{\mathbf{f}}_{\mathbf{K}}}{⟷}\mathbf{z}.$$ Such a sequence of invertible transformations is also called normalizing flow.

Given an observed data variable $\mathbf{x}\in \mathcal{X}$, a simple probability distribution $p(\mathbf{z})$ with $\mathbf{z}\in \mathcal{Z}$, and a bijective function $\mathbf{f}:\mathcal{X}\to \mathcal{Z}$ (with $\mathbf{g}={\mathbf{f}}^{-1}$), the change of variable formula defines a model distribution on $\mathcal{X}$ by $$\begin{array}{rl}p(\mathbf{x})& =p(\mathbf{z})|det\left({\displaystyle \frac{\partial \mathbf{z}}{\partial \mathbf{x}}}\right)|\\ \log p(\mathbf{x})& =\log p(\mathbf{z})+\log |det\left({\displaystyle \frac{\partial \mathbf{z}}{\partial \mathbf{x}}}\right)|\\ & =\log p(\mathbf{z})+\sum _{i=1,j=i-1}^K\log |det\left({\displaystyle \frac{\partial {\mathbf{h}}_{\mathbf{i}}}{\partial {\mathbf{h}}_{\mathbf{j}}}}\right)|\end{array}$$ with ${\mathbf{h}}_{\mathbf{0}}=\mathbf{x}$ and ${\mathbf{h}}_{\mathbf{K}}=\mathbf{z}$.

The determinant of the Jacobian matrix $\partial {\mathbf{h}}_{\mathbf{i}}/\partial {\mathbf{h}}_{\mathbf{j}}$ is the change in density when ${\mathbf{h}}_{\mathbf{j}}$ is transformed to ${\mathbf{h}}_{\mathbf{i}}$ under the transformation ${\mathbf{f}}_{\mathbf{i}}$. Thus, flow-based models require to compute this determinant as well. The second key aspect is to design the functions ${\mathbf{f}}_{\mathbf{i}}$ such that the determinant of its Jacobian is easy to compute.

Thus, flow-based models require two important design choices on ${\mathbf{f}}_{\mathbf{i}}$:

1. Have reversible architecture
2. Design transformations whose determinant of Jacobians are easy to compute

To satisfy these two requirements, the trick is to choose the transformations whose Jacobian is a triangular matrix, such that their determinnant can be simply computed as the product of its diagonal elements. Thus, $$\log |det\left({\displaystyle \frac{\partial {\mathbf{h}}_{\mathbf{i}}}{\partial {\mathbf{h}}_{\mathbf{j}}}}\right)|=\sum \log \left|\text{diag}\left({\displaystyle \frac{\partial {\mathbf{h}}_{\mathbf{i}}}{\partial {\mathbf{h}}_{\mathbf{j}}}}\right)\right|$$ where, $\text{diag}(\cdot )$ takes the diagonal of the Jacobian matrix.

These models are trained (i.e., training the neural nets $\mathbf{f}$) such that the negative log-likelihood of $\mathbf{z}$ is minimized with respect to some prior distribution (more on this below).

To generate data, we can sample from the prior distribution $p(\mathbf{z})$ and do the inverse operation.

NICE: Non-linear Independent Components Estimation

As mentioned before, the two key main aspects of flow-based approaches is easy determinant of the Jacobian and easy inverse. In NICE, the input data is split into two blocks $\mathbf{x}\to ({\mathbf{x}}_{\mathbf{1}},{\mathbf{x}}_{\mathbf{2}})$ (say, even and odd indices). Then apply block transformation from $({\mathbf{x}}_{\mathbf{1}},{\mathbf{x}}_{\mathbf{2}})$ to $({\mathbf{h}}_{\mathbf{1}}^{\mathbf{1}},{\mathbf{h}}_{\mathbf{1}}^{\mathbf{2}})$ of the form: $$\begin{array}{rl}{\mathbf{h}}_{\mathbf{1}}^{\mathbf{1}}& ={\mathbf{x}}_{\mathbf{1}}\\ {\mathbf{h}}_{\mathbf{1}}^{\mathbf{2}}& ={\mathbf{x}}_{\mathbf{2}}+\mathbf{m}({\mathbf{x}}_{\mathbf{1}})\end{array}$$ where, $\mathbf{m}$ is an arbitrarily complex function (neural net). They call this transformation as an Affine Coupling layer. This transformation satisfies the two design choices:

• The inverse can be easily computed as: $$\begin{array}{rl}{\mathbf{x}}_{\mathbf{1}}& ={\mathbf{h}}_{\mathbf{1}}^{\mathbf{1}}\\ {\mathbf{x}}_{\mathbf{2}}& ={\mathbf{h}}_{\mathbf{1}}^{\mathbf{2}}-\mathbf{m}({\mathbf{h}}_{\mathbf{1}}^{\mathbf{1}})\end{array}$$

• The transformation $\mathbf{f}$ is (where $d$ and $e$ are the dimensions of ${\mathbf{x}}_{\mathbf{1}}$ and ${\mathbf{x}}_{\mathbf{2}}$) $$\begin{array}{r}{\mathbf{h}}_{\mathbf{1}}=\left[\begin{array}{c}{\mathbf{h}}_{\mathbf{1}}^{\mathbf{1}}\\ {\mathbf{h}}_{\mathbf{1}}^{\mathbf{2}}\end{array}\right]=\left[\begin{array}{cc}{\mathbf{I}}_{\mathbf{d}}& 0\\ \mathbf{m}(\cdot )& {\mathbf{I}}_{\mathbf{e}}\end{array}\right]\left[\begin{array}{c}{\mathbf{x}}_{\mathbf{1}}\\ {\mathbf{x}}_{\mathbf{2}}\end{array}\right]\end{array}$$ resulting in a Jacobian matrix $$\begin{array}{rl}{\displaystyle \frac{\partial \mathbf{f}}{\partial \mathbf{x}}}& =\left[\begin{array}{cc}{\displaystyle \frac{\partial {\mathbf{h}}_{\mathbf{1}}^{\mathbf{1}}}{\partial {\mathbf{x}}_{\mathbf{1}}}}& {\displaystyle \frac{\partial {\mathbf{h}}_{\mathbf{1}}^{\mathbf{1}}}{\partial {\mathbf{x}}_{\mathbf{2}}}}\\ {\displaystyle \frac{\partial {\mathbf{h}}_{\mathbf{1}}^{\mathbf{2}}}{\partial {\mathbf{x}}_{\mathbf{1}}}}& {\displaystyle \frac{\partial {\mathbf{h}}_{\mathbf{1}}^{\mathbf{2}}}{\partial {\mathbf{x}}_{\mathbf{2}}}}\end{array}\right]\\ & =\left[\begin{array}{cc}{\mathbf{I}}_{\mathbf{d}}& 0\\ {\displaystyle \frac{\partial \mathbf{m}(\cdot )}{\partial {\mathbf{x}}_{\mathbf{1}}}}& {\mathbf{I}}_{\mathbf{e}}\end{array}\right]\end{array}$$ whose determinant is unity. Notice that such a design not only enables easy compuatation of the determinant, but also lets us choose arbitrarily complex $\mathbf{m}(\cdot )$ since we dont have to compute its derivative to obtain the determinant.

Similarly, inverse operation from $\mathbf{z}$ to $\mathbf{x}$ also results in a unit Jacobian determinant. Thus generating data also is easy with the NICE model.

In the NICE model, since all the transformations are volume preserving (unit Jacobian determinant), the resulting transformation will have equal weight over all dimensions, which is not desirable in practical applications. To address this, NICE also includes a scaling layer at the output that scales every dimension by a trainable weight $S_i$. This allows the model to give more weight on some dimensions and less on others.

Thus the nice criterion becomes maximizing the log-likelihood of the data distribution: $$\begin{array}{rl}\log p(\mathbf{x})& =\log p(\mathbf{z})+\log |det\left({\displaystyle \frac{\partial \mathbf{z}}{\partial \mathbf{x}}}\right)|\\ & =\log p(\mathbf{z})+\sum _{i=1}^D\log (|S_i|)\end{array}$$

Further, NICE model assumes that the prior distribution is factorial: $p(\mathbf{z})=\prod _{i=1}^{D}p({\mathbf{z}}_{\mathbf{i}})$. The training criterion for NICE is to maximize its log-likelihood or minimize the negative log-likelihood: $\mathcal{L}=-\log p(\mathbf{x})$

• For standard gaussian:
  $\mathcal{L}=\sum _{i=1}^{D}0.5\cdot ({{\mathbf{z}}_{\mathbf{i}}}^2+\log 2\pi )-\log (|S_i|)$
• For standard logistic:
  $\mathcal{L}=\sum _{i=1}^D\log (1+\exp ({\mathbf{z}}_{\mathbf{i}}))+\log (1+\exp (\mathbf{-}{\mathbf{z}}_{\mathbf{i}}))-\log (|S_i|)$

As we stack multiple affine coupling layers to obtain more complex transformations. Since the transformation leaves one part of the data leaves unchanged, we can alternate the role of each part in subsequent coupling layers. Typically, 4 coupling layers are used so that all dimensions influence the one another. The scaling layer is paramterised exponentially $\exp (S_i)$ to have positive scaling.

In NICE the forward model maps the datapoint to the latent space and is trained to minimize the negative log-likelihood with respect to some prior distribution. And for inference, we sample from the prior distribution to get $\mathbf{h}$ and new datapoints $\mathbf{x}$ can be generated using the inverse flow. For a 4 layer architecture, the forward and inverse flow equations are,

Implementation Notes

As mentioned above, the NICE model is trained to minimize the negative log-likelihood with respect to a standard Gaussian or logistic distribution.

• Getting the models correct: The main design challenge was to get the inverse model correct with its weights tied to the corresponding forward model. We can verify if the inverse model is correct even before training the model. Just take a test image from MNIST and pass it through the forward model. Then use the resulting output as input to the inverse model. If the inverse model is created with correct weights, it should yield the test image.

• Getting NaNs: For implementing the logistic loss $\log (1+\exp ({\mathbf{z}}_{\mathbf{i}}))+\log (1+\exp (\mathbf{-}{\mathbf{z}}_{\mathbf{i}}))$, I initially used Keras backend as:

import keras.backend as K     
logistic_negloglikelihood = K.sum ( K.log (1 + K.exp(z)) +  K.log(1 + K.exp(-z)), axis=1 )

However, it was resulting in NaN after a few epochs. Then this was replaced with softplus function which computes $\log (1+\exp (x))$ and the NaN issue was (magically!) gone.

logistic_negloglikelihood = K.sum ( K.softplus(z) + K.softplus(-z), axis=1 )

• Initialization: Initially, I was using the default glorot_uniform initialization of keras which was not resulting in the log-likelihoods given in the paper. Dinh’s original implementation in Theano used an initialization from a uniform distribution in $[-0.01,0.01]$. Using the same initialization for the Dense layers resulted in better log-likelihoods.

• Batch-size: Dinh used Adam optimizer with a learning rate $0.001$ and batch size of 256. In my experiments, I observed that increasing the batch size from $256$ to $2048$ consistently yielded better log-likelihoods.

• Clip the outputs during generating data: MNIST data is rescaled to the range $[0,1]$ for training. But during inference, the output may not be always in $[0,1]$. Therefore, we apply clipping on the generated outputs to bring them to the desired range.