Related Work & Experiments

This section covers Sections 4 and 5 of the paper: connections to other algorithms and experimental validation.

Related Work

Wake-Sleep Algorithm

The VAE shares structural similarities with the Wake-Sleep algorithm (Hinton et al., 1995):

Key difference: Wake-Sleep uses the recognition model’s outputs as targets for the generative model. AEVB uses the variational bound which takes the recognition model as input.

The Wake-Sleep gradient estimator can have high variance since it “weights the approximate posterior $q_\phi$ by how well it ‘explains’ the data.”

Stochastic Variational Inference (SVI)

Hoffman et al. (2013) developed Stochastic Variational Inference for conjugate-exponential family models:

• Exploits conjugacy for closed-form updates
• Uses natural gradients
• Requires model-specific derivations

AEVB advantage: Works with any differentiable model, not just conjugate families. Uses standard backpropagation — no model-specific math required.

REINFORCE-Based Methods

The REINFORCE gradient estimator (Williams, 1992):

$$\nabla_\phi \mathbb{E}{q\phi}[f(\mathbf{z})] = \mathbb{E}{q\phi}[f(\mathbf{z}) \nabla_\phi \log q_\phi(\mathbf{z})]$$

Problems:

• Very high variance
• Requires many samples per gradient estimate
• Control variates help but add complexity

Reparameterization advantage: Lower variance, more practical for high-dimensional latents.

Neural Variational Inference and Learning (NVIL)

Mnih & Gregor (2014) developed NVIL around the same time:

• Uses REINFORCE with variance reduction
• Input-dependent baselines
• Works with discrete latents (unlike reparameterization)

When to use which:

• Continuous latents → Reparameterization (AEVB)
• Discrete latents → REINFORCE/NVIL

Experimental Setup

Datasets

The paper evaluates on two datasets:

Preprocessing

For MNIST:

• Original images have continuous values in [0, 1]
• Treated as Bernoulli parameters (probability of pixel being “on”)
• Binary cross-entropy loss natural for this interpretation

For Frey Face:

• Continuous grayscale values
• Used Gaussian decoder with learned variance

Architecture

The paper uses simple MLPs:

MNIST (Bernoulli decoder):
├── Encoder: 784 → 500 → (μ, log σ²) for J-dimensional latent
├── Latent: J ∈ {2, 5, 10, 20, 50}
└── Decoder: J → 500 → 784 (sigmoid output)

Frey Face (Gaussian decoder):
├── Encoder: 560 → 200 → (μ, log σ²) for J-dimensional latent
├── Latent: J ∈ {2, 5, 10, 20}
└── Decoder: J → 200 → (μ_out, log σ²_out) for 560 dimensions

Training Details

Key finding: With $M = 100$ minibatch size, a single sample ($L = 1$) per datapoint is sufficient!

Results

Marginal Likelihood Estimation

The paper reports lower bounds on the marginal log-likelihood $\log p(\mathbf{x})$.

For evaluation, they use importance sampling with $k$ samples to get a tighter bound:

$$\log p(\mathbf{x}) \geq \mathbb{E}\left[\log \frac{1}{k} \sum_{l=1}^{k} \frac{p(\mathbf{x}, \mathbf{z}^{(l)})}{q(\mathbf{z}^{(l)}|\mathbf{x})}\right]$$

Where $\mathbf{z}^{(l)} \sim q(\mathbf{z}|\mathbf{x})$.

MNIST Results

AEVB outperforms Wake-Sleep across all latent dimensions tested.

Comparison: AEVB vs Wake-Sleep

The paper provides detailed comparisons:

Observations:

1. AEVB consistently outperforms Wake-Sleep
2. Performance improves with more latent dimensions
3. Diminishing returns after ~20 dimensions for MNIST

Frey Face Results

For continuous data with Gaussian decoder:

(Values are log-likelihood, higher is better)

Visualization: 2D Latent Space

With $J = 2$ latent dimensions, we can visualize the learned manifold.

The paper shows that sampling uniformly from the 2D latent space and decoding produces:

• Smooth transitions between digit classes (MNIST)
• Continuous variations in facial expression/pose (Frey Face)

This demonstrates the VAE learns a meaningful, organized latent space.

Key Experimental Findings

Finding 1: Single Sample Suffices

With minibatch size $M = 100$:

• $L = 1$ (one sample per datapoint) works well
• No need for multiple samples like traditional Monte Carlo methods

Why? The minibatch provides variance reduction through averaging across datapoints.

Finding 2: Recognition Model Works

The learned encoder $q_\phi(\mathbf{z}|\mathbf{x})$ successfully approximates the true posterior:

• Low KL divergence to prior (regularized latent space)
• Good reconstructions (accurate posterior location)

Finding 3: Scalability

Training time scales linearly with:

• Dataset size $N$ (minibatch training)
• Latent dimension $J$
• Network size

No per-datapoint optimization — unlike traditional variational inference.

Finding 4: Smooth Latent Space

The learned latent space exhibits:

• Interpolation: Points between encoded samples decode to plausible outputs
• Clustering: Similar inputs map to nearby latent codes
• Disentanglement: Different latent dimensions capture different factors of variation

Computational Efficiency

The paper emphasizes that AEVB trains as fast as a regular neural network:

Where $D$ = input dim, $H$ = hidden dim, $J$ = latent dim.

Limitations Noted

The paper acknowledges several limitations:

1. Gaussian assumption: The encoder assumes diagonal Gaussian posterior

   • May not capture complex multimodal posteriors
   • Later work (normalizing flows) addresses this

2. Reconstruction vs Generation tradeoff: High KL weight → blurry samples

   • The β-VAE literature explores this tradeoff

3. Simple architectures: The paper uses small MLPs

   • Modern VAEs use deep CNNs, attention, etc.

Summary