Variational Autoencoders (VAEs) are a core idea in modern generative modelling: they learn a compact “latent space” that can be sampled to generate new data resembling what the model has seen. What makes a VAE different from a standard autoencoder is not the neural network architecture alone, but the probabilistic framing. VAEs explicitly model uncertainty and learn a distribution over latent variables rather than a single deterministic code. If you are exploring a generative AI course, understanding the mathematics behind VAEs helps you move beyond “it works” to “why it works,” especially when models are used for data synthesis, anomaly detection, or representation learning.
1) The Generative Story: Latent Variables and Likelihood
At a high level, a VAE assumes data points xxx are generated from hidden causes zzz (latent variables). The model defines:
- A prior over latents: p(z)p(z)p(z), commonly a standard normal N(0,I)\mathcal{N}(0, I)N(0,I)
- A decoder (likelihood): pθ(x∣z)p_\theta(x \mid z)pθ(x∣z), parameterised by a neural network with parameters θ\thetaθ
The goal is to learn θ\thetaθ so that the marginal likelihood pθ(x)p_\theta(x)pθ(x) is large for training data:
pθ(x)=∫pθ(x∣z) p(z) dzp_\theta(x) = \int p_\theta(x \mid z)\, p(z)\, dzpθ(x)=∫pθ(x∣z)p(z)dz
This integral is the problem: it is usually intractable because the decoder is a neural network and the integral is high-dimensional. VAEs solve this by introducing an approximation to the posterior distribution of zzz given xxx.
2) Approximate Inference: The Encoder as a Variational Posterior
The true posterior is:
pθ(z∣x)=pθ(x∣z)p(z)pθ(x)p_\theta(z \mid x) = \frac{p_\theta(x \mid z)p(z)}{p_\theta(x)}pθ(z∣x)=pθ(x)pθ(x∣z)p(z)
Since pθ(x)p_\theta(x)pθ(x) requires that hard integral, we cannot compute pθ(z∣x)p_\theta(z \mid x)pθ(z∣x) directly. VAEs introduce a tractable approximation:
qϕ(z∣x)q_\phi(z \mid x)qϕ(z∣x)
This is the encoder, parameterised by ϕ\phiϕ, typically outputting the parameters of a Gaussian:
qϕ(z∣x)=N(μϕ(x),diag(σϕ2(x)))q_\phi(z \mid x) = \mathcal{N}(\mu_\phi(x), \mathrm{diag}(\sigma^2_\phi(x)))qϕ(z∣x)=N(μϕ(x),diag(σϕ2(x)))
So instead of mapping xxx to one latent vector, the encoder maps xxx to a distribution over latent vectors. This distributional view is the “variational” part: we choose qϕq_\phiqϕ from a family of distributions and optimise it to be close to the true posterior.
If a generative AI course mentions “inference networks,” this is exactly what they mean: a neural network that performs fast approximate Bayesian inference.
3) The ELBO: Turning Intractable Likelihood into an Optimisable Objective
VAEs optimise a lower bound on log?pθ(x)\log p_\theta(x)logpθ(x), called the Evidence Lower Bound (ELBO):
log?pθ(x)≥Eqϕ(z∣x)[log?pθ(x∣z)]−KL(qϕ(z∣x) ∥ p(z))\log p_\theta(x) \ge \mathbb{E}_{q_\phi(z\mid x)}[\log p_\theta(x\mid z)] – \mathrm{KL}\big(q_\phi(z\mid x)\,\|\,p(z)\big)logpθ(x)≥Eqϕ(z∣x)[logpθ(x∣z)]−KL(qϕ(z∣x)∥p(z))
This objective has two intuitive terms:
- Reconstruction term:
- Eqϕ(z∣x)[log?pθ(x∣z)]\mathbb{E}_{q_\phi(z\mid x)}[\log p_\theta(x\mid z)]Eqϕ(z∣x)[logpθ(x∣z)]
- Encourages the decoder to reconstruct xxx well from sampled zzz. Depending on data type, pθ(x∣z)p_\theta(x\mid z)pθ(x∣z) might be Gaussian (continuous data) or Bernoulli/Categorical (binary or discrete-like representations).
- Regularisation term (KL divergence):
- KL(qϕ(z∣x) ∥ p(z))\mathrm{KL}\big(q_\phi(z\mid x)\,\|\,p(z)\big)KL(qϕ(z∣x)∥p(z))
- Encourages the encoder’s latent distribution to stay close to the prior. This is the key to a smooth, sampleable latent space without it, the model could memorise data with scattered encodings that do not support meaningful sampling.
So the VAE learns a latent space that balances two forces: represent enough information to reconstruct data, but remain organised and “prior-shaped” so random samples from p(z)p(z)p(z) decode into plausible outputs.
4) The Reparameterisation Trick: Making Sampling Differentiable
A practical challenge appears in the reconstruction term: we sample z∼qϕ(z∣x)z \sim q_\phi(z\mid x)z∼qϕ(z∣x), but sampling is not differentiable in the usual sense, and we need gradients to train neural networks.
The VAE uses the reparameterisation trick. For Gaussian latents:
z=μϕ(x)+σϕ(x)⊙ϵ,ϵ∼N(0,I)z = \mu_\phi(x) + \sigma_\phi(x) \odot \epsilon,\quad \epsilon \sim \mathcal{N}(0, I)z=μϕ(x)+σϕ(x)⊙ϵ,ϵ∼N(0,I)
Now randomness is isolated in ϵ\epsilonϵ, which is independent of ϕ\phiϕ. The network outputs μϕ(x)\mu_\phi(x)μϕ(x) and σϕ(x)\sigma_\phi(x)σϕ(x), and gradients can flow through them because the transformation is differentiable. This one idea makes end-to-end training of probabilistic latent variable models feasible with standard backprop.
In many generative AI course modules, you will see this as the bridge between probability and deep learning optimization.
Conclusion
VAEs are best understood as probabilistic models trained with approximate inference. The encoder learns a variational posterior qϕ(z∣x)q_\phi(z\mid x)qϕ(z∣x), the decoder models pθ(x∣z)p_\theta(x\mid z)pθ(x∣z), and training maximises the ELBO combining reconstruction quality with KL-based latent regularisation. This is why VAEs learn latent spaces that are not just compressions, but structured, continuous regions you can sample from to generate data. If your goal is to build reliable generative systems, mastering these foundations is one of the most practical payoffs of a generative AI course.
Business Name: ExcelR – Data Analyst, Data Science & Generative AI Course in Noida
Address: Myworx, A-5, 2nd Floor, near Noida Sector 16 Metro Station, Gautam Budh Nagar, Block A, Noida Sector 3, Noida, Uttar Pradesh 201301
Phone Number: 09187195453
Email ID: [email protected]
