3 Foundations of Generative Modelling

What does it mean for a machine to generate data? Consider the task of producing a new face photograph that looks realistic but does not depict any real person, or synthesising a plausible completion of a half-written paragraph. In each case, we want to draw a fresh sample from some complex, high-dimensional distribution (the distribution of natural face images, or of coherent English text) even though we only have access to a finite collection of examples from that distribution.

This chapter formalises the generative modelling problem and develops the core ideas that every later chapter builds on. We start from the simplest possible setting, sampling from a known one-dimensional distribution, and progressively introduce the key abstractions: push-forward maps, latent variables, and the reparameterisation trick. Along the way, we connect each idea to the concrete generative model families studied in Parts II–V of this book.

What Does “Generation” Mean?

Suppose we observe data $x_1,x_2,\dots ,x_n$ drawn independently from an unknown distribution $p_{\mathrm{data}}$ over a space $𝒳$ (e.g. $𝒳={\mathrm{ℝ}}^d$ for images with $d$ pixels). The goal of generative modelling is to construct a mechanism that can produce new samples whose distribution closely approximates $p_{\mathrm{data}}$.

At first glance the problem sounds daunting: $p_{\mathrm{data}}$ lives in high dimensions, is unknown, and may have a very complicated shape (multiple modes, thin ridges, etc.). Yet every approach we will study reduces the problem to two ingredients:

1. a source of randomness $z\sim p_Z$ that is easy to sample (e.g. a standard Gaussian), and

2. a deterministic map $g_{\theta }:𝒵\to 𝒳$ that transforms simple noise into complex data.

The art lies in how we design $g_{\theta }$ and how we train it. Each family of generative models makes a different choice, as summarised in Table 1.

Approach	How $g_{\theta }$ is trained	Chapters
Autoregressive models	Maximise $\log p_{\theta }(x)$ via chain rule	3, 6
Normalising flows	Maximise $\log p_{\theta }(x)$ via change of variables	14
VAEs	Maximise ELBO (lower bound on $\log p_{\theta }(x)$)	14
GANs	Adversarial game (no explicit likelihood)	9–13
Diffusion models	Denoise score matching	16

Sampling from Known Distributions

Before tackling the full generative problem, we review how to sample from a known distribution. Every method rests on the same idea: transform uniform or Gaussian noise into the target distribution.

The Inverse-CDF Method

Let $X$ be a continuous random variable with cumulative distribution function $F_X(x)=\mathrm{Pr}(X\le x)$. Because $F_X$ is monotonically non-decreasing and maps $\mathrm{ℝ}\to [\mathrm{0,1}]$, we can (under mild regularity conditions) invert it.

The Box–Muller Transform

The Gaussian distribution $𝒩(\mathrm{0,1})$ does not have a closed-form inverse CDF. A classical workaround is the Box–Muller transform: given two independent uniform samples $U_1,U_2\sim \mathrm{Uniform}(\mathrm{0,1})$, define (BOX Muller)$$\begin{array}{ccc}& Z_1=\sqrt{-2\ln U_1}\cos (2\pi U_2),Z_2=\sqrt{-2\ln U_1}\sin (2\pi U_2).& \text{<span class="tml-eqn"></span>}\end{array}$$ Then $Z_1$ and $Z_2$ are independent $𝒩(\mathrm{0,1})$ random variables.

Rejection Sampling

When neither the inverse CDF nor a clever transform is available, we can fall back on rejection sampling. Suppose we want to sample from a target density $p(x)$ but can only evaluate it (up to a normalising constant). Choose a proposal density $q(x)$ that is easy to sample and satisfies $p(x)\le Mq(x)$ for some constant $M$.

Importance Sampling

Sometimes we do not need fresh samples but rather need to estimate an expectation ${𝔼}_{x\sim p}[f(x)]$ when $p$ is hard to sample from. Importance sampling re-writes the expectation using a tractable proposal $q$: (Importance Sampling)$$\begin{array}{ccc}& {𝔼}_{x\sim p}[f(x)]={𝔼}_{x\sim q}[\frac{p(x)}{q(x)}f(x)]={𝔼}_{x\sim q}[w(x)f(x)],& \text{<span class="tml-eqn"></span>}\end{array}$$ where $w(x)=p(x)/q(x)$ are the importance weights.

Maximum Likelihood Estimation for Generation

The simplest route to a generative model is: choose a parametric family $p_{\theta }$, fit it to data by MLE, then sample from the fitted distribution.

Once we have $\hat{\theta }$, generation is straightforward: sample $x_{\mathrm{new}}\sim p_{\hat{\theta }}$.

Autoregressive Factorisation

For a $d$-dimensional vector $𝒙=(x_1,\dots ,x_d)$, the chain rule of probability gives an exact factorisation: (Chain RULE GEN)$$\begin{array}{ccc}& p(𝒙)=p(x_1)p(x_2|x_1)p(x_3|x_1,x_2)\cdots p(x_d|x_1,\dots ,x_{d-1})=\prod _{i=1}^{d}p(x_i|x_{<i}).& \text{<span class="tml-eqn"></span>}\end{array}$$ If each conditional $p(x_i|x_{<i})$ is modelled by a parametric distribution (say a Gaussian or categorical) whose parameters are output by a neural network, we obtain an autoregressive model. Generation proceeds sequentially: sample $x_1$, then $x_2$ given $x_1$, and so on.

The Push-Forward Distribution

We now formalise the idea of “transforming noise into data.”

The push-forward viewpoint unifies all generative models:

Connection to Change of Variables

When $g$ is a diffeomorphism (smooth, invertible, with smooth inverse), the push-forward density can be computed explicitly via the change-of-variables formula from Chapter 2: (PUSH Forward Density)$$\begin{array}{ccc}& p_X(x)=p_Z(g^{-1}(x))\left|\det \frac{\partial g^{-1}}{\partial x}\right|.& \text{<span class="tml-eqn"></span>}\end{array}$$ This equation is the backbone of normalising flows: the generator $g_{\theta }$ is an invertible neural network, and the log-likelihood (NF Loglik)$$\begin{array}{ccc}& \log p_X(x)=\log p_Z(g_{\theta }^{-1}(x))+\log \left|\det \frac{\partial g_{\theta }^{-1}}{\partial x}\right|& \text{<span class="tml-eqn"></span>}\end{array}$$ can be maximised exactly, with no approximations needed.

The Generative Modelling Problem

We now state the central problem of this book precisely.

Latent Variable Models

In practice, the data we observe (images, text, speech) lives in a very high-dimensional space, yet its intrinsic structure is often much lower-dimensional. Latent variable models make this explicit by introducing a hidden (latent) variable $z$ that “explains” the observations.

The Intractability Problem

The marginal likelihood involves an integral over the entire latent space, which is typically intractable: it cannot be computed in closed form when $p_{\theta }(x|z)$ is parameterised by a neural network.

This intractability has two consequences:

1. We cannot evaluate $p_{\theta }(x)$ exactly, so standard MLE is not directly applicable.

2. The posterior $p_{\theta }(z|x)=p_{\theta }(x|z)p(z)/p_{\theta }(x)$ is also intractable, since it requires knowing $p_{\theta }(x)$.

Different model families deal with this in different ways:

Model	Strategy for intractability
Normalising flows	Make $g_{\theta }$ invertible so the integral is not needed
VAEs	Introduce an approximate posterior $q_{\varphi }(z|x)$ and optimise a lower bound (ELBO)
GANs	Avoid computing $p_{\theta }(x)$ entirely; use an adversarial discriminator instead
Diffusion models	Decompose generation into many small denoising steps, each tractable

The Reparameterisation Trick

Suppose we want to optimise an objective that involves an expectation over a parameterised distribution. For instance, in a VAE the ELBO contains $${𝔼}_{z\sim q_{\varphi }(z|x)}[\log p_{\theta }(x|z)].$$ To perform gradient descent on $\varphi$, we need ${\nabla }_{\varphi }{𝔼}_{z\sim q_{\varphi }}[\cdot ]$. But the sampling operation $z\sim q_{\varphi }$ is not differentiable: how can we back-propagate through a random draw?

Now $z$ is a deterministic, differentiable function of $\varphi$ (and $ϵ$), so the gradient $${\nabla }_{\varphi }{𝔼}_{ϵ\sim 𝒩(0,I)}\left[\log p_{\theta }\right(x|{\mu }_{\varphi }(x)+{\sigma }_{\varphi }(x)\odot ϵ)]$$ can be estimated by Monte Carlo and back-propagated through the network as usual.

Evaluating Generative Models

How do we know whether a generative model produces “good” samples? This is surprisingly subtle; we need to assess both quality (do individual samples look realistic?) and diversity (does the model cover the full data distribution, or does it suffer from mode collapse?).

Tractability and the Road Ahead

Let us take stock of the fundamental tension in generative modelling. We want a model that is simultaneously:

1. Expressive: capable of modelling complex, multimodal distributions in high dimensions.

2. Tractable: allowing efficient training and sampling.

3. Likelihood-evaluable: permitting computation of $p_{\theta }(x)$ for evaluation and comparison.

No single model achieves all three perfectly. fig:triangle illustrates the trade-off.

The remainder of this book explores each major model family in depth:

• Part II: Probabilistic Generative Models. Graphical models and Gaussian mixtures provide the classical foundation: exact inference is possible in small models, but approximate inference (EM, variational methods) is needed for larger ones.

• Part III: Generative Adversarial Networks. GANs side-step likelihood computation entirely, using an adversarial game to train the generator. They excel at sample quality but can suffer from mode collapse and training instability.

• Part IV: Variational and Transport Methods. VAEs and normalising flows provide principled likelihood-based training. VAEs use amortised variational inference; flows use invertible architectures for exact likelihood.

• Part V: Diffusion Models. Diffusion models decompose the generation process into many small denoising steps, achieving state-of-the-art quality with stable training.

Each approach represents a different resolution of the expressiveness–tractability–likelihood triangle. The mathematical tools needed to understand these trade-offs (divergence measures, optimisation theory, and deep learning) are developed in the next two chapters.

Exercises