Paper Breakdown
Variational Auto-Encoders (VAE) Explained
A layman's guide to the Kingma & Welling paper that revolutionized generative AI and became the foundation for models like Stable Diffusion.
7 sections
~45 min read
Based on arXiv:1312.6114
1 Section 1 of 7
Introduction & Problem Scenario
The central question, generative model setup, and the three intractability problems.
The generative model Intractable marginal likelihood Recognition model
2 Section 2 of 7
The Variational Bound (ELBO)
The mathematical derivation of the variational lower bound — Equations (1), (2), (3).
KL divergence term Reconstruction term Two forms of the ELBO
3 Section 3 of 7
SGVB Estimator & AEVB Algorithm
The practical estimator that makes VAE training possible — Equations (5), (6), (7), (8).
Monte Carlo estimation Minibatch training Algorithm 1
4 Section 4 of 7
The Reparameterization Trick
The key innovation that enables backpropagation through sampling — Equation (4).
The gradient problem Location-scale families Gradient flow
5 Section 5 of 7
VAE Example: Neural Network Implementation
The concrete implementation using MLPs — Equations (9), (10) and PyTorch code.
Encoder MLP Decoder (Bernoulli/Gaussian) Final estimator
6 Section 6 of 7
Related Work & Experiments
Connections to Wake-Sleep, experimental results on MNIST and Frey Face.
Wake-Sleep comparison MNIST results Latent space visualization
7 Section 7 of 7
Appendix: Mathematical Details
KL divergence derivation for Gaussians (Appendix B) and MLP specifications (Appendix C).
Closed-form KL MLP architecture Full implementation