20 Video Diffusion Models

Why Video Diffusion?

Consider a single photograph of a dog catching a frisbee in mid-air. The image captures fur texture, the shape of the frisbee, the blur of motion, the colour of the sky. A modern image diffusion model can generate such a photograph with startling fidelity, sampling it from a learned distribution over pixel grids $𝒙\in {\mathrm{ℝ}}^{H\times W\times C}$. Now consider generating a video of that same dog leaping, catching the frisbee, and landing: 120 frames at 24 frames per second, each frame a $1280\times 720$ RGB image. The sample space has exploded from roughly one million pixel values to over one hundred million, and worse, those hundred million values are not independent. They must obey the laws of physics, the biomechanics of a running dog, the aerodynamics of a spinning frisbee, and the continuity of a coherent narrative.

This is the fundamental challenge of video generation: the data is vastly higher-dimensional, and the constraints binding those dimensions together are far richer than anything encountered in still images. A video is not merely a stack of independent pictures; it is a structured spatiotemporal object whose frames are coupled by motion, lighting consistency, object permanence, and causal relationships. Getting any one of these wrong, a hand that gains a sixth finger between frames, a shadow that reverses direction, a ball that passes through a wall, immediately breaks the illusion.

The dimensionality explosion

Let us make the scale concrete. An image diffusion model operates on tensors of shape $H\times W\times C$. For a $512\times 512$ RGB image, the sample space has dimensionality (DIM Image)$$\begin{array}{ccc}& d_{\text{image}}=512\times 512\times 3=\mathrm{786,432.}& \text{<span class="tml-eqn"></span>}\end{array}$$ A video adds a temporal axis of $F$ frames. The resulting video tensor lives in ${\mathrm{ℝ}}^{F\times H\times W\times C}$. For a modest 5-second clip at 24 fps and $1280\times 720$ resolution, (DIM Video)$$\begin{array}{ccc}& d_{\text{video}}=\underset{F}{\underbrace{120}}\times \underset{H}{\underbrace{720}}\times \underset{W}{\underbrace{1280}}\times \underset{C}{\underbrace{3}}=\mathrm{331,776,000}\approx 3.3\times {10}^8.& \text{<span class="tml-eqn"></span>}\end{array}$$ This is roughly 422 times the dimensionality of a single $512\times 512$ image. At 4K resolution ($3840\times 2160$), the ratio climbs to over $\mathrm{3,700}\times$. The situation is even more daunting for longer videos: a two-minute clip at 720p contains nearly $8\times {10}^9$ values.

Beyond dimensionality: structural requirements

Raw dimensionality is only part of the story. What makes video generation qualitatively harder than image generation is the set of structural constraints that a plausible video must satisfy. We organise these into four categories.

1. Temporal coherence. Adjacent frames must depict the same scene with smooth, physically plausible transitions. Objects cannot teleport, disappear, or change identity between frames. Formally, the distribution $p({𝒙}^{f+1}|{𝒙}^f)$ has extremely concentrated support compared to the marginal $p({𝒙}^{f+1})$.

2. Physical plausibility. Objects must obey (approximate) physical laws: gravity pulls things downward, rigid bodies do not deform, fluids flow continuously, and light reflects consistently. Violations are immediately noticeable to human viewers, even when they cannot articulate the specific physical law being broken.

3. Narrative consistency. A video tells a story, however simple. A person who picks up a cup must later put it down or continue holding it; a car driving left-to-right should not suddenly reverse without cause. This requires the model to maintain a form of causal reasoning over extended temporal horizons.

4. Perceptual quality. Each individual frame must be a high-quality image, free of artifacts such as blurring, colour banding, or unnatural textures. Furthermore, the temporal texture must be correct: natural videos have characteristic motion blur, lighting flicker, and depth-of-field effects that synthetic videos often lack.

The video generation desiderata

Before diving into specific methods, let us enumerate the properties that an ideal video generation system should possess. These desiderata will serve as evaluation criteria throughout the chapter.

1. Temporal coherence: Generated frames should form a smooth, consistent sequence with no flickering, identity drift, or discontinuous motion.

2. Visual quality: Each frame should be indistinguishable from a real photograph, with sharp details, correct colours, and natural textures.

3. Physical plausibility: Objects should obey approximate physical laws: gravity, rigid-body mechanics, fluid dynamics, and conservation of mass.

4. Controllability: The user should be able to control the generated content through natural interfaces: text descriptions, reference images, camera trajectories, or combinations thereof.

5. Scalability: The system should support variable resolutions (from 256p to 4K), variable durations (from 1 second to minutes), and variable aspect ratios without retraining.

6. Efficiency: Generation should be fast enough for interactive applications, ideally producing a 5-second clip in under 30 seconds on a single GPU.

7. Diversity: Different random seeds should produce meaningfully different videos for the same prompt, reflecting the true variability of the data distribution.

No current system satisfies all seven desiderata simultaneously. The history of the field can be read as a sequence of innovations, each improving one or two desiderata, sometimes at the cost of others.

A brief history of video generation

The pursuit of learned video generation has evolved through several paradigm shifts. We summarise the key milestones below.

Chapter roadmap

This chapter unfolds in seven stages covering the complete landscape of video diffusion models. fig:vdiff:roadmap provides a visual roadmap. We begin with the mathematical foundations, establishing the spatiotemporal data manifold, the three-pillar framework, the extension of image diffusion to video, and the computational challenges that motivate everything that follows. The remaining six stages address the architectural and algorithmic innovations that make large-scale video generation practical.

Foundations (you are here)

Why video is hard, the spatiotemporal data manifold, the three-pillar framework, extending image diffusion to video, and the computational challenge.

Compression

sec:vdiff:video-ae,sec:vdiff:causal-3dvae, sec:vdiff:discrete-tokens,sec:vdiff:latent-temporal: video autoencoders and the metrics that judge them; causal 3D VAEs, from 3D convolutions and the left-padding trick to the full training objective; discrete tokenisation by vector quantisation and lookup-free quantisation; and the temporal structure of the resulting latent space.

Architectures

sec:vdiff:dit,sec:vdiff:factored-attn, sec:vdiff:3d-pe,sec:vdiff:conditioning,sec:vdiff:moe: spatiotemporal patchification and the video Diffusion Transformer with adaLN-Zero; the central efficiency decision of factoring attention; 3D RoPE and length generalisation; the three conditioning pathways; and mixture-of-experts layers.

Objectives, sampling and training

sec:vdiff:noise-schedules,sec:vdiff:flow-matching, sec:vdiff:sampling-acceleration,sec:vdiff:training-scale: why image noise schedules must be shifted for video; flow matching on optimal-transport paths and rectified flow; sampling acceleration by distillation and feature caching; and multi-stage training at scale, including data curation and resolution curricula.

Control and personalisation

sec:vdiff:cfg-video,sec:vdiff:i2v, sec:vdiff:motion-camera,sec:vdiff:t2v-pipeline, sec:vdiff:personalisation: multi-condition classifier-free guidance; image-to-video; camera control via Plücker rays and video ControlNets; LLM prompt enhancement and the full text-to-video pipeline; and LoRA and DreamBooth personalisation.

Long video and world models

sec:vdiff:long-video,sec:vdiff:temporal-consistency, sec:vdiff:editing,sec:vdiff:physics,sec:vdiff:ar-diffusion: autoregressive chunking and hierarchical keyframes for long-horizon generation; measuring and enforcing temporal consistency; video editing; physics-aware generation and the “is this a world model?” question; and autoregressive-diffusion hybrids.

Evaluation, applications and synthesis

sec:vdiff:evaluation,sec:vdiff:applications, sec:vdiff:connections,sec:vdiff:open-problems: FVD and decomposed benchmarks such as VBench; specialised applications from human motion to talking heads; connections to the rest of the book; and five open problems.

The Spatiotemporal Data Manifold

Before we can build models that generate video, we must understand the mathematical structure of the data they aim to produce. In this section, we formalise video as a mathematical object, introduce the key structural properties that distinguish video from collections of independent images, and develop the notation that will serve us throughout the chapter.

Video as a continuous function

At the most abstract level, a video is a function that maps spacetime coordinates to colour values. Let $T_{\text{dur}}>0$ denote the video duration in seconds. We define:

In practice, we never observe $v$ in its continuous form. Cameras sample this function on a discrete grid determined by the frame rate $r$ (temporal sampling) and the sensor resolution $H\times W$ (spatial sampling). This gives rise to the video tensor.

The Nyquist perspective

The continuous-to-discrete transition is governed by the Nyquist-Shannon sampling theorem. Temporally, a video sampled at $r$ fps can faithfully represent motion frequencies up to $r/2$ Hz. A spinning wheel at 13 Hz sampled at 24 fps will alias, appearing to rotate backwards (the wagon-wheel effect). Spatially, the sensor resolution limits the spatial frequencies that can be captured.

This proposition has a practical implication for video generation: the frame rate of the generated video determines the highest temporal frequency that can be represented. A model generating at 8 fps cannot faithfully represent fast motions that require 24 fps to resolve; it can only produce temporally smooth, low-frequency motion.

Optical flow and the brightness constancy equation

One of the most important structural properties of natural video is the relationship between adjacent frames induced by object motion. If a scene point moves with velocity $𝒗={(v_u,v_w)}^{𝖳}$ in the image plane, and its brightness does not change during the motion, then the brightness constancy equation holds:

(Brightness Constancy)$$\begin{array}{ccc}& \frac{\partial I}{\partial t}+\nabla I\cdot 𝒗=0,& \text{<span class="tml-eqn"></span>}\end{array}$$ where $I(t,u,w)$ is the image intensity at time $t$ and spatial location $(u,w)$, and $\nabla I={(\partial I/\partial u,\partial I/\partial w)}^{𝖳}$ is the spatial gradient.

Temporal autocorrelation

The statistical relationship between frames at different time offsets is captured by the temporal autocorrelation function.

For natural video, the autocorrelation decays smoothly with lag, reflecting the fact that nearby frames are highly similar while distant frames may depict entirely different content. This decay rate varies dramatically across video types: a static surveillance camera produces near-constant autocorrelation, while an action movie has rapid decay due to frequent cuts and fast motion.

The decorrelation timescale ${\tau }_0$ has direct implications for model design. A model with a temporal receptive field of $K$ frames can capture correlations up to lag $K$; if $K\ll {\tau }_0$, the model will miss long-range temporal dependencies and produce temporally incoherent output.

Formalising temporal coherence

We now give a formal definition of temporal coherence, the most important structural property that distinguishes video from a random sequence of images.

This lemma reveals an important property: temporal coherence constrains how much a video can change over any time interval. A tightly coherent video (small $\delta$) can only depict slow changes; depicting fast motion requires either larger $\delta$ or a higher frame rate (so that each individual frame-to-frame change is small even though the overall motion is fast).

The video manifold hypothesis

The manifold hypothesis, well-established in the image domain, posits that high-dimensional data concentrates near a much lower-dimensional manifold. For video, this hypothesis is even more compelling: the constraints of physics, scene composition, and visual coherence restrict the set of plausible videos to a tiny fraction of the ambient space ${\mathrm{ℝ}}^{FHWC}$.

Visualising the spatiotemporal structure

Three Pillars of Video Generation

Every successful video generation system, regardless of its specific architecture, must address three fundamental challenges. We call these the three pillars of video generation: compression, dynamics, and control. This taxonomy provides a unifying lens through which to understand the diverse landscape of methods we will encounter throughout the chapter.

Pillar 1: Compression

The first pillar addresses the dimensionality challenge: how to reduce the video tensor to a manageable size without losing information critical for reconstruction. The key insight is that natural videos have enormous redundancy, both spatial (neighbouring pixels are correlated) and temporal (adjacent frames share most of their content).

Spatial compression.

A standard image VAE [12] compresses each frame independently, reducing spatial resolution by a factor of $s_H\times s_W$ (typically $8\times 8$). This transforms $𝖷\in {\mathrm{ℝ}}^{F\times H\times W\times C}$ to $𝖹\in {\mathrm{ℝ}}^{F\times H/s_H\times W/s_W\times C^{\prime }}$, achieving a compression ratio of $s_H\cdot s_W\cdot C/C^{\prime }$ per frame.

Temporal compression.

Adjacent frames share so much content that we can also compress along the time axis. A temporal compression factor of $s_F$ reduces $F$ frames to $F/s_F$ latent frames. Combined with spatial compression, the total compression ratio becomes (Compression Ratio VAE)$$\begin{array}{ccc}& r_{\text{total}}=\frac{F\cdot H\cdot W\cdot C}{(F/s_F)\cdot (H/s_H)\cdot (W/s_W)\cdot C^{\prime }}=\frac{s_F\cdot s_H\cdot s_W\cdot C}{C^{\prime }}.& \text{<span class="tml-eqn"></span>}\end{array}$$

The compression-quality tradeoff.

Higher compression ratios reduce the computational burden on the sampler but inevitably lose fine-grained details. The autoencoder's reconstruction quality sets an upper bound on the generated video quality: no matter how good the sampler is, it cannot produce details that the decoder cannot reconstruct. Finding the right compression ratio is a critical design decision.

Pillar 2: Dynamics

The second pillar is the most distinctive aspect of video generation: modelling the temporal evolution of visual content. This is where video generation fundamentally departs from image generation. The sampler $𝒮$ must produce latent tensors whose frames, when decoded, form a temporally coherent sequence.

Factored dynamics.

A common approach factorises the denoiser into interleaved spatial and temporal blocks. Writing the network as a composition of $L$ block pairs, (Factored Denoiser)$$\begin{array}{ccc}& {ϵ}_{\theta }({𝖹}_t,t)=({ℬ}_L^{\text{temp}}\circ {ℬ}_L^{\text{spat}}\circ \cdots \circ {ℬ}_1^{\text{temp}}\circ {ℬ}_1^{\text{spat}})({𝖹}_t,t),& \text{<span class="tml-eqn"></span>}\end{array}$$ where each ${ℬ}_{\ell }^{\text{spat}}$ processes every frame independently (inheriting its weights from an image diffusion model) and each ${ℬ}_{\ell }^{\text{temp}}$ mixes information across frames at a fixed spatial location. Note that the blocks are composed, not added: the two branches are not two competing noise predictions but two stages of one network. The temporal blocks are typically zero-initialised, so that at the start of training the video model reproduces the pretrained image model applied frame by frame. This factorisation enables transfer from pretrained image models and reduces the computational cost of temporal modelling.

Attention-based dynamics.

The dominant mechanism for modelling dynamics in modern video diffusion models is temporal attention: given a sequence of per-frame features ${\{{𝒉}^f\}}_{f=0}^{F-1}$, temporal attention computes (Temporal Attention)$$\begin{array}{ccc}& \text{TemporalAttn}({𝒉}^f)=\sum _{f^{\prime }=0}^{F-1}\frac{\exp \left({({𝑾}_Q{𝒉}^f)}^{𝖳}({𝑾}_K{𝒉}^{f^{\prime }})/\sqrt{d}\right)}{{\sum }_{f^{\prime \prime }}\exp \left({({𝑾}_Q{𝒉}^f)}^{𝖳}({𝑾}_K{𝒉}^{f^{\prime \prime }})/\sqrt{d}\right)}{𝑾}_V{𝒉}^{f^{\prime }},& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${𝑾}_Q,{𝑾}_K,{𝑾}_V$ are the query, key, and value projection matrices and $d$ is the head dimension.

Convolutional dynamics.

An alternative is to use 1D convolutions along the time axis (or 3D convolutions jointly over space and time). These capture local temporal patterns efficiently but have limited receptive fields. In practice, many architectures combine temporal convolutions for local patterns with temporal attention for global coherence.

Pillar 3: Control

The third pillar addresses how to steer the generation process toward desired outputs. Without control, a video generation model is a random video sampler, interesting theoretically but of limited practical utility. Control comes in many forms.

Text conditioning.

The most common control signal is a text prompt, injected via cross-attention between the text embedding and the diffusion model's internal representations. Let ${𝒄}_{\text{text}}\in {\mathrm{ℝ}}^{N_c\times d_{\text{text}}}$ be the sequence of $N_c$ text token embeddings from a frozen language model. (We reserve $L$ for the number of network layers throughout the chapter, so the prompt length gets its own symbol.) Cross-attention computes: (Cross Attention)$$\begin{array}{ccc}& \text{CrossAttn}(𝒉,{𝒄}_{\text{text}})=\mathrm{softmax}\left(\frac{({𝑾}_Q𝒉){({𝑾}_K{𝒄}_{\text{text}})}^{𝖳}}{\sqrt{d}}\right){𝑾}_V{𝒄}_{\text{text}},& \text{<span class="tml-eqn"></span>}\end{array}$$ where $𝒉$ is the spatial-temporal feature at a given position and the softmax operates over the text sequence length $N_c$.

Image conditioning.

Image-to-video generation conditions on one or more reference frames. The conditioning image can be injected by concatenation (appending it as an extra channel to the noisy input), by cross-attention, or by replacing the first frame of the noisy sequence with the (possibly encoded) conditioning image.

Structural control.

Fine-grained control over motion and layout can be achieved through auxiliary conditioning signals: depth maps, pose sequences, edge maps, segmentation masks, or motion trajectories. These are typically injected through lightweight adapter networks (ControlNets [61]) that leave the base model unchanged.

Classifier-free guidance for video.

As in image diffusion, classifier-free guidance (CFG) [62] amplifies the influence of the conditioning signal by extrapolating away from the unconditional prediction: (CFG)$$\begin{array}{ccc}& {\widetilde{ϵ}}_{\theta }({𝖹}_t,t,𝒄)={ϵ}_{\theta }({𝖹}_t,t,⌀)+w({ϵ}_{\theta }({𝖹}_t,t,𝒄)-{ϵ}_{\theta }({𝖹}_t,t,⌀)),& \text{<span class="tml-eqn"></span>}\end{array}$$ where $w\ge 1$ is the guidance scale and $⌀$ denotes the null conditioning. We use this convention throughout the chapter: $w=1$ recovers the ordinary conditional prediction and $w>1$ extrapolates beyond it. (The literature also writes CFG as $(1+w^{\prime }){ϵ}_{𝒄}-w^{\prime }{ϵ}_{⌀}$; the two scales differ by one, $w=1+w^{\prime }$, so quoted numerical ranges must always be read together with their convention.) For video, CFG can be applied jointly to all frames (preserving temporal coherence in the guidance direction) or independently per frame (risking temporal artifacts).

The three-pillar framework: a unified view

fig:vdiff:three-pillars shows how different techniques map onto the three-pillar framework. Every video generation system makes specific design choices for each pillar, and the interplay between these choices determines the system's capabilities and limitations. For instance, Sora [6] uses a spacetime patchifier for compression (Pillar 1), a Diffusion Transformer for dynamics (Pillar 2), and text cross-attention with classifier-free guidance for control (Pillar 3).

From Image Diffusion to Video Diffusion

In this section, we make the conceptual leap from image diffusion to video diffusion precise. We show that the mathematical framework of diffusion models extends naturally to video tensors, but that the resulting model must satisfy structural requirements that go far beyond anything needed for images. We assume familiarity with the image diffusion framework from 19, particularly the DDPM training objective and the reparameterisation trick.

The joint forward process

Recall that in image diffusion, the forward process gradually corrupts a clean image ${𝒙}_0\in {\mathrm{ℝ}}^{H\times W\times C}$ by adding Gaussian noise according to a variance schedule ${\{{\beta }_t\}}_{t=1}^T$: (Image Forward)$$\begin{array}{ccc}& q({𝒙}_t|{𝒙}_0)=\mathrm{Normal}({𝒙}_t;\sqrt{{\bar{\alpha }}_t}{𝒙}_0,(1-{\bar{\alpha }}_t)𝑰),& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${\alpha }_t=1-{\beta }_t$ and ${\bar{\alpha }}_t={\prod }_{s=1}^t{\alpha }_s$. For video, we simply replace the image ${𝒙}_0$ with the video tensor ${𝖹}_0$, treating the entire video as a single high-dimensional object.

Why independent frame diffusion fails

The simplest approach to video diffusion would be to apply an image diffusion model independently to each frame. This is tempting because it allows direct reuse of powerful pretrained image models. However, it fails catastrophically.

The video DDPM training algorithm

The training procedure for video diffusion is a direct extension of image DDPM. The key change is that the denoiser ${ϵ}_{\theta }$ now takes a video tensor as input and must predict noise for all frames jointly.

The denoiser must learn spatiotemporal structure

We now state a key proposition that motivates the architectural innovations of sec:vdiff:dit,sec:vdiff:factored-attn.

The reverse process and sampling

The reverse process mirrors the image case. Starting from ${𝖹}_T\sim \mathrm{Normal}(0,𝑰)$, we iteratively denoise: (Reverse)$$\begin{array}{ccc}& p_{\theta }({𝖹}_{t-1}|{𝖹}_t)=\mathrm{Normal}({𝖹}_{t-1};{𝝁}_{\theta }({𝖹}_t,t),{\sigma }_t^2𝑰),& \text{<span class="tml-eqn"></span>}\end{array}$$ where the predicted mean is (Predicted MEAN)$$\begin{array}{ccc}& {𝝁}_{\theta }({𝖹}_t,t)=\frac{1}{\sqrt{{\alpha }_t}}({𝖹}_t-\frac{{\beta }_t}{\sqrt{1-{\bar{\alpha }}_t}}{ϵ}_{\theta }({𝖹}_t,t,𝒄)).& \text{<span class="tml-eqn"></span>}\end{array}$$

The Computational Challenge

We have established that video diffusion is mathematically straightforward: simply apply the image diffusion framework to a higher-dimensional tensor. The devil, however, is in the computational details. In this section, we quantify the computational cost of naive video diffusion and show that it is prohibitively expensive, motivating the compression and architectural innovations discussed in the subsequent files.

The attention bottleneck

The dominant computational cost in modern diffusion models comes from self-attention layers. In an image diffusion model operating on a spatial grid of $H^{\prime }\times W^{\prime }$ latent pixels, each self-attention layer has complexity: (Image Attention Flops)$$\begin{array}{ccc}& {\text{FLOPs}}_{\text{image}}=O({(H^{\prime }{W}^{\prime })}^2\cdot d),& \text{<span class="tml-eqn"></span>}\end{array}$$ where $d$ is the attention head dimension. For video, if we apply full spatiotemporal attention (every position attends to every other position across all frames), the cost becomes: (Video Attention Flops)$$\begin{array}{ccc}& {\text{FLOPs}}_{\text{video}}=O({(F^{\prime }{H}^{\prime }{W}^{\prime })}^2\cdot d)=O(F^{\prime 2}{H}^{\prime 2}{W}^{\prime 2}\cdot d).& \text{<span class="tml-eqn"></span>}\end{array}$$

Memory constraints

Beyond FLOPs, memory is often the binding constraint. Attention requires storing the key-value (KV) matrices for all tokens: (KV Memory)$$\begin{array}{ccc}& {\text{Memory}}_{\text{KV}}=2\cdot N\cdot d\cdot n_{\text{heads}}\cdot L\cdot b_{\text{dtype}},& \text{<span class="tml-eqn"></span>}\end{array}$$ where $N=F^{\prime }{H}^{\prime }{W}^{\prime }$ is the total number of tokens, $n_{\text{heads}}$ is the number of attention heads, $L$ is the number of layers, and $b_{\text{dtype}}$ is the bytes per element (2 for FP16/BF16, 1 for FP8).

Scaling analysis

fig:vdiff:compute-scaling illustrates how different attention strategies scale with the number of latent frames. The quadratic cost of full spatiotemporal attention becomes prohibitive beyond $F^{\prime }\approx 32$ latent frames: at the $4\times$ temporal compression and 24 fps of Example 8 that is roughly five seconds of video, and a single attention layer already costs over 400 TFLOPs. This motivates two complementary strategies, taken up in turn in the two stages that follow:

1. Compress more aggressively (Video Autoencoders): reduce $F^{\prime }$, $H^{\prime }$, and $W^{\prime }$ through better video autoencoders, directly reducing the token count $N=F^{\prime }{H}^{\prime }{W}^{\prime }$.

2. Factorise attention (Factored Attention for Video): replace full spatiotemporal attention with separate spatial and temporal attention passes, reducing the cost from $O(F^{\prime 2}{H}^{\prime 2}{W}^{\prime 2})$ to $O(F^{\prime }{(H^{\prime }{W}^{\prime })}^2+H^{\prime }{W}^{\prime }\cdot F^{\prime 2})$.

Scaling with resolution and duration

The computational cost of video generation scales polynomially with all three axes: spatial resolution, temporal duration, and model size. Let us write the total cost of sampling a video as: (Total COST)$$\begin{array}{ccc}& C_{\text{total}}=T_{\text{steps}}\cdot L\cdot C_{\text{attn}}(F^{\prime },H^{\prime },W^{\prime },d)+C_{\text{other}},& \text{<span class="tml-eqn"></span>}\end{array}$$ where $T_{\text{steps}}$ is the number of diffusion sampling steps, $L$ is the number of transformer layers, $C_{\text{attn}}$ is the per-layer attention cost, and $C_{\text{other}}$ accounts for feedforward layers, normalisations, and the decoder.

Parameter doubled	Cost multiplier (full attn)	Cost multiplier (factored)
Side length ($H^{\prime }$ and $W^{\prime }$)	$16\times$	$16\times$
Duration ($F^{\prime }$)	$4\times$	$\sim 2\times$
Model depth ($L$)	$2\times$	$2\times$
Head dimension ($d$)	$2\times$	$2\times$
Sampling steps ($T_{\text{steps}}$)	$2\times$	$2\times$

The memory wall

Even with FlashAttention eliminating the $O(N^2)$ memory cost of the attention matrix, video diffusion models face a “memory wall” from three sources:

1. Model parameters: A DiT-XL model has 675M parameters ($\sim$1.35 GB in FP16). Scaled-up video models can exceed 10B parameters ($\sim$20 GB in FP16).

2. Activations: During training, intermediate activations must be stored for backpropagation. With $L=28$ layers, $N=\mathrm{432,000}$ tokens, $d_{\text{model}}=1024$ and BF16 storage, a single $N\times d_{\text{model}}$ tensor costs 0.88 GB per layer, so the two tensors per layer counted by $2\cdot N\cdot d_{\text{model}}\cdot L\cdot b_{\text{dtype}}$ already come to 50 GB. A real DiT block retains many more than two - the QKV projections, the attention output, and above all the $4d_{\text{model}}$-wide MLP hidden state, which alone costs four times a single $d_{\text{model}}$-wide tensor - so the true figure comfortably exceeds 200 GB.

3. The video tensor itself: The full-resolution input/output video must fit in memory during encoding and decoding. A single 720p, 120-frame video in FP32 requires $120\times 720\times 1280\times 3\times 4\approx 1.3$ GB.

Comparison to other generative modalities

It is instructive to compare the computational demands of video generation to other generative modalities, as this comparison highlights why video remains the most challenging frontier.

Summary and preview of solutions

The computational analysis in this section paints a clear picture: naive video diffusion, applying full spatiotemporal attention to all frames jointly, is prohibitively expensive for any video longer than a few seconds at modest resolution. Table 2 summarises the key challenges and the solutions we will develop in subsequent files.

Challenge	Where addressed	Key technique
High-dimensional input	sec:vdiff:video-ae,sec:vdiff:causal-3dvae	Causal 3D video autoencoders with spatial and temporal compression
Quadratic attention cost	Factored Attention for Video	Factored spatial/temporal attention, 3D window attention
Conditioning complexity	sec:vdiff:conditioning,sec:vdiff:motion-camera	Cross-attention and adaLN pathways, ControlNet adapters
Slow sampling	Sampling Acceleration	Consistency distillation, feature caching, few-step solvers
Data and training scale	Training at Scale	Multi-stage resolution curricula, mixed image-video training, data curation
Drift over long horizons	Long Video Generation	Autoregressive chunking, hierarchical keyframes

We now turn to the first and most impactful lever for reducing computational cost: compressing the video tensor into a compact latent representation. Video Autoencoders sets up video autoencoders and the reconstruction metrics that judge them; Causal 3D Variational Autoencoders builds the causal 3D VAE that has become the field's default, from 3D convolutions and the left-padding trick up to the full training objective; Discrete Tokenisation for Video covers the discrete alternative, vector quantisation and lookup-free quantisation; and Latent Space Temporal Structure closes the loop by showing that a temporally smooth video maps to a temporally smooth latent trajectory, which is precisely what makes latent video diffusion work at all.

Video Autoencoders

Diffusion models are powerful, but they are also expensive. The iterative denoising process that makes them work requires evaluating a neural network dozens or hundreds of times per sample, and each evaluation operates on a tensor whose size equals the output resolution. For images, this cost is manageable: a $256\times 256\times 3$ image contains roughly $\mathrm{200,000}$ values. For video, the situation is dramatically worse. A modest $16$-frame clip at $256\times 256$ resolution already contains $3.1$ million values; a $2$-second clip at $720\text{p}$ ($1280\times 720$) and $24$ frames per second exceeds $130$ million values. Running a diffusion model directly in this pixel space is computationally prohibitive.

The solution, first demonstrated for images by Rombach et al. [14] in Latent Diffusion Models (LDMs), is to separate the problem into two stages. First, train an autoencoder that compresses the data into a compact latent space. Second, run the diffusion process in that latent space rather than in pixel space. The autoencoder absorbs the perceptual details (textures, edges, colour gradients) that do not require iterative refinement, while the diffusion model focuses on the semantic structure (object identity, motion, scene composition) that benefits from the iterative process.

For video, this two-stage strategy is not merely convenient; it is essential. The compression ratio determines the practical feasibility of the entire pipeline. In this section, we formalise the two principal approaches to video compression, spatial-only and spatiotemporal, and analyse the trade-offs that govern the design of video autoencoders.

Why Pixel-Space Video Diffusion Is Impractical

To appreciate the necessity of latent compression, let us quantify the computational burden of pixel-space video diffusion. Consider a video tensor $𝖷\in {\mathrm{ℝ}}^{F\times H\times W\times 3}$. The diffusion model must process this tensor at every denoising step, and a typical generation requires $T$ steps (often $T=50$ to $1000$).

This example illustrates a broader principle: the autoencoder is not an optional optimisation but a structural prerequisite for video diffusion at practical resolutions.

Spatial-Only Compression

The simplest approach to video compression is to ignore the temporal dimension entirely and encode each frame independently using a pre-trained image autoencoder.

The spatial-only approach has clear advantages. It reuses image autoencoders (such as the one from Stable Diffusion) without any modification, inheriting their mature training and high reconstruction quality. The frame count $F$ is preserved in the latent space, so temporal modelling is left entirely to the diffusion model.

However, this design has a fundamental limitation: it completely ignores the temporal redundancy present in natural video. Consecutive frames in a video are highly correlated; objects move smoothly, lighting changes gradually, and backgrounds persist across many frames. A spatial-only encoder encodes each frame as if it were an independent image, wasting capacity on information that is redundant across time.

Spatiotemporal Compression

To exploit temporal redundancy, we need an autoencoder that processes the video as a 3D volume rather than a stack of independent images.

The spatiotemporal approach compresses along all three dimensions simultaneously. Because consecutive frames share most of their content, the encoder can represent temporal redundancy far more efficiently than the spatial-only approach.

Comparing the Two Approaches

Table 3 summarises the key trade-offs between spatial-only and spatiotemporal video autoencoders.

Property	Spatial-Only	Spatiotemporal
Temporal compression	None ($F^{\prime }=F$)	Yes ($F^{\prime }<F$)
Temporal redundancy	Not exploited	Exploited
Reuses image AE	Yes	No (new architecture)
Latent size for $F=48$	$48\times h\times w$	$12\times h\times w$
Causal design needed	No	Yes (for streaming)
Training data	Images suffice	Videos required
Reconstruction quality	Per-frame optimal	Jointly optimised

Architecture Overview

fig:vdiff:2d-vs-3d-ae illustrates the two approaches side by side. The spatial-only encoder processes each frame through a 2D convolutional network, producing independent latent maps that are stacked along the time axis. The spatiotemporal encoder processes the entire video through a 3D convolutional network, producing a compressed latent volume with reduced temporal extent.

The connection between video autoencoders and the VAE theory developed in 15 is direct: both spatial-only and spatiotemporal video autoencoders are trained as variational autoencoders, optimising an ELBO that balances reconstruction fidelity against latent regularisation. The key differences lie in the architecture of the encoder and decoder (2D vs. 3D convolutions) and the structure of the latent space (independent per-frame vs. jointly compressed). The vector quantisation techniques from 16 also play a role, as we will see in Discrete Tokenisation for Video.

Reconstruction Quality Metrics

How do we measure the quality of a video autoencoder? Unlike text compression, where lossless reconstruction is the standard, video autoencoders are lossy by design. The quality of the lossy reconstruction must be measured along multiple axes.

Causal 3D Variational Autoencoders

The previous section motivated the need for spatiotemporal video compression and contrasted it with the simpler spatial-only approach. We now turn to the mathematical core of spatiotemporal video compression: the causal 3D variational autoencoder (Causal 3D-VAE). This architecture combines three ingredients: (1) 3D convolutions that process spatial and temporal dimensions jointly, (2) a causal masking scheme that prevents future frames from influencing past latents, and (3) a variational training objective that regularises the latent space for downstream diffusion.

The causality constraint is not merely a technical nicety. Without it, the latent code ${𝒛}^f$ for frame $f$ would depend on all frames in the video, including frames $f+1,f+2,\dots ,F$. This creates two problems. First, it prevents autoregressive generation, where frames are produced sequentially and future frames are not yet available when encoding past frames. Second, it couples the latent representations in a way that makes it difficult for the diffusion model to generate temporally coherent video, because changing one frame's latent code would implicitly alter all other latent codes.

3D Convolutions for Video

Standard 2D convolutions operate on spatial dimensions $(H,W)$ with a kernel of shape $(k_h,k_w)$. To process video, we extend convolutions to operate on the temporal dimension as well, producing a 3D convolution with a kernel of shape $(k_t,k_h,k_w)$.

The 3D convolution extends naturally from its 2D counterpart. Each output location $(t,h,w)$ aggregates information from a spatiotemporal neighbourhood of size $k_t\times k_h\times k_w$ centred around the corresponding input location. The parameter $k_t$ controls the temporal receptive field: a kernel with $k_t=3$ looks at the current frame plus one frame in each temporal direction.

Causal 3D Convolutions

A standard 3D convolution with temporal kernel size $k_t>1$ looks both forward and backward in time. For frame $t$, the convolution accesses frames $t-\lfloor k_t/2\rfloor$ through $t+\lfloor k_t/2\rfloor$. This bidirectional temporal access is acceptable during training (all frames are available), but it creates a fundamental problem for autoregressive inference and streaming applications where future frames have not yet been observed.

The solution is to restrict the convolution to use only past and current frames, producing a causal 3D convolution.

In practice, causal 3D convolutions are implemented by applying asymmetric temporal padding: $k_t-1$ frames of zero-padding are prepended to the temporal dimension, and no padding is appended. This ensures that the output at time $t$ depends only on inputs at times $\le t$, matching the standard causal convention used in autoregressive language models (17).

fig:vdiff:causal-vs-noncausal illustrates the difference between causal and non-causal receptive fields for a stack of two layers with $k_t=3$.

The Causal 3D-VAE Training Objective

With causal 3D convolutions as the building block, we can now define the full training objective for a causal 3D variational autoencoder. The objective follows the standard VAE framework (see 15) but is applied to video tensors with architectures that respect the causal constraint.

In practice, the reconstruction term is evaluated using a Gaussian decoder, which reduces to a mean squared error: (Recon MSE)$$\begin{array}{ccc}& {𝔼}_{q_{\varphi }(𝖹|𝖷)}[\log p_{\theta }(𝖷|𝖹)]\approx -\frac{1}{2{\sigma }^2}\Vert 𝖷-{𝒟}_{\theta }(𝖹){\Vert }_F^2+\text{const},& \text{<span class="tml-eqn"></span>}\end{array}$$ where $\Vert \cdot {\Vert }_F$ denotes the Frobenius norm over all spatiotemporal dimensions and ${\sigma }^2$ is a fixed noise variance (often set to $1$).

Temporal Compression Bound

How much temporal compression is achievable without destroying the video content? Intuitively, if consecutive frames are very similar (low motion), then aggressive temporal compression is possible. If frames change rapidly (high motion), compression must be more conservative. The following proposition makes this intuition precise.

The Complete Training Loss

The ELBO alone is insufficient for training a high-quality video autoencoder. Modern video autoencoders augment the ELBO with perceptual and adversarial losses that improve visual quality beyond what pixel-level reconstruction can achieve.

Encoder-Decoder Architecture

The encoder and decoder of a causal 3D-VAE follow the familiar hierarchical structure of image autoencoders, extended to three dimensions. fig:vdiff:3dvae-architecture shows a typical design.

Factorised Spatiotemporal Processing

Full 3D convolutions are parameter-heavy and computationally expensive. A common optimisation is to factorise each 3D convolution into separate spatial and temporal components.

This factorisation was popularised by S3D [63] and R(2+1)D [64] in the video understanding literature and has been widely adopted in video generation. The Open-Sora [18] encoder uses this factorised design throughout, applying 2D spatial convolutions inherited from a pre-trained image autoencoder and adding temporal convolution layers that are trained from scratch.

Training Procedure

Training a causal 3D-VAE involves alternating between the autoencoder (encoder + decoder) and the discriminator, following the same adversarial training protocol used for image autoencoders.

Discrete Tokenisation for Video

The previous two sections developed continuous latent video autoencoders, where the latent space is a real-valued tensor $𝖹\in {\mathrm{ℝ}}^{F^{\prime }\times H^{\prime }\times W^{\prime }\times C^{\prime }}$. An alternative approach, rooted in the VQ-VAE framework (see 16), replaces the continuous latent space with a discrete one. Each spatial (or spatiotemporal) location in the latent tensor is mapped to the nearest entry in a learned codebook, producing a sequence of discrete tokens that represent the video.

Discrete tokenisation has a compelling appeal: it unifies video generation with the autoregressive language modelling paradigm that has proven so successful for text. Once a video is encoded as a sequence of discrete tokens, it can be modelled by a standard autoregressive transformer (GPT-style) or a masked transformer (BERT-style), inheriting the powerful sequence modelling capabilities developed for natural language processing.

Vector Quantisation for Video

The foundation of discrete video tokenisation is vector quantisation (VQ), which maps continuous encoder outputs to the nearest entry in a finite codebook.

For a video tensor, quantisation is applied independently to each spatiotemporal location in the latent tensor. The encoder produces $𝖹=ℰ(𝖷)\in {\mathrm{ℝ}}^{F^{\prime }\times H^{\prime }\times W^{\prime }\times d}$, and each vector ${𝒛}_{f,h,w}\in {\mathrm{ℝ}}^d$ is quantised independently: (VQ Video)$$\begin{array}{ccc}& {\widehat{𝒛}}_{f,h,w}=q({𝒛}_{f,h,w})={𝒆}_{k_{f,h,w}^{\ast }},k_{f,h,w}^{\ast }={\mathrm{arg\hspace{0.17em}min}}_k\Vert {𝒛}_{f,h,w}-{𝒆}_k{\Vert }_2.& \text{<span class="tml-eqn"></span>}\end{array}$$ The resulting discrete token sequence has length $N=F^{\prime }\times H^{\prime }\times W^{\prime }$, with each token taking values in $\{1,\dots ,K\}$.

Training with the Straight-Through Estimator

The quantisation operator $q(𝒛)={\mathrm{arg\hspace{0.17em}min}}_k\Vert 𝒛-{𝒆}_k\Vert$ has zero gradients almost everywhere (and undefined gradients at the Voronoi boundaries). To train the encoder through the quantisation bottleneck, VQ-VAE [13] uses the straight-through estimator (STE): during the forward pass, the quantised values $\widehat{𝒛}$ are used; during the backward pass, gradients are copied directly from the decoder input to the encoder output, bypassing the quantisation step: (STE)$$\begin{array}{ccc}& \widehat{𝒛}=𝒛+\mathrm{sg}[\widehat{𝒛}-𝒛],& \text{<span class="tml-eqn"></span>}\end{array}$$ where $\mathrm{sg}[\cdot ]$ denotes the stop-gradient operator. This expression evaluates to $\widehat{𝒛}$ in the forward pass (since $𝒛+(\widehat{𝒛}-𝒛)=\widehat{𝒛}$) but has gradient ${\nabla }_{𝒛}\widehat{𝒛}=𝑰$ in the backward pass (since the stop-gradient term contributes zero gradient).

The VQ-VAE training loss combines reconstruction with two commitment terms that keep encoder outputs and codebook entries close:

The three terms serve distinct purposes. The reconstruction loss trains the decoder. The codebook loss moves codebook entries toward encoder outputs (training only the codebook, since the encoder output is stopped). The commitment loss moves encoder outputs toward codebook entries (training only the encoder, since the codebook entry is stopped). Together, they ensure that encoder outputs and codebook entries remain close, preventing codebook collapse (where large portions of the codebook go unused).

Lookup-Free Quantisation

A persistent challenge with standard VQ is that increasing the codebook size $K$ beyond a few thousand entries leads to codebook underutilisation: many codebook entries are never selected, wasting representational capacity. MAGVIT-v2 [16] introduced lookup-free quantisation (LFQ) to address this problem.

LFQ eliminates the codebook collapse problem entirely: every “codebook entry” (vertex of the hypercube) is equally accessible, and utilisation is determined solely by the distribution of encoder outputs. The sign function is non-differentiable, but gradients can be estimated using a smooth approximation such as $\tanh (\alpha 𝒛)$ with $\alpha \gg 1$, or by using the straight-through estimator.

The VQ-Video Pipeline

fig:vdiff:vq-pipeline illustrates the complete VQ-based video tokenisation pipeline, from input frames through 3D encoding, quantisation, and decoding.

Latent Space Temporal Structure

The preceding sections developed the machinery for compressing video into a latent space, whether continuous (causal 3D-VAE) or discrete (VQ-based tokenisation). We now turn to a fundamental question: what properties does the latent space inherit from the temporal structure of the input video? Understanding this question is essential for designing diffusion models that generate temporally coherent video in latent space.

The central insight of this section is both simple and powerful: temporal smoothness in pixel space translates to temporal smoothness in latent space, provided the encoder satisfies mild regularity conditions. This means that if consecutive video frames are similar (as they almost always are in natural video), then their latent codes are also similar. The diffusion model can therefore generate smooth video by generating smooth trajectories in latent space.

Latent Temporal Lipschitz Continuity

We begin with the fundamental theorem connecting pixel-space smoothness to latent-space smoothness.

First-Frame Conditioning

A common design pattern in video generation is to condition the diffusion model on the first frame of the video. The user provides an image, and the model generates a video that begins with that image and continues with plausible motion. This requires the autoencoder to treat the first frame specially.

This decomposition has several advantages. First, it allows the first frame to be encoded at full temporal resolution (no temporal compression), preserving the fine details of the conditioning image. Second, it enables the use of a pre-trained image encoder (such as the Stable Diffusion VAE) for the first frame, while using a video-specific encoder for the remaining frames. Third, during inference, the first frame's latent code can be provided as a fixed conditioning signal, and the diffusion model only needs to generate the remaining latent frames.

Latent Interpolation and Smooth Generation

The Lipschitz continuity theorem (Theorem 3) tells us that smooth videos produce smooth latent trajectories. The converse is equally important for generation: smooth latent trajectories should produce smooth videos. This motivates the study of latent interpolation.

Temporal Structure in the Latent Diffusion Process

The latent temporal structure has direct implications for the design of the diffusion model that operates in the latent space. Because the latent trajectory of a natural video is smooth (Theorem 3), the noise schedule and denoising architecture can be designed to exploit this smoothness.

To formalise this, consider the temporal frequency content of the latent tensor. For a fixed spatial location $(h^{\prime },w^{\prime })$ and channel $c^{\prime }$, the temporal signal ${𝒛}_{h^{\prime },w^{\prime },c^{\prime }}\in {\mathrm{ℝ}}^{F^{\prime }}$ is a one-dimensional sequence. Its discrete Fourier transform reveals the frequency content: (Latent FFT)$$\begin{array}{ccc}& {\hat{Z}}_{h^{\prime },w^{\prime },c^{\prime }}(\omega )=\sum _{f^{\prime }=0}^{F^{\prime }-1}{z}_{f^{\prime },h^{\prime },w^{\prime },c^{\prime }}{e}^{-2\pi i\omega f^{\prime }/F^{\prime }},\omega =0,1,\dots ,F^{\prime }-1.& \text{<span class="tml-eqn"></span>}\end{array}$$

For a temporally smooth latent (small $\Vert {𝒛}^{f+1}-{𝒛}^f\Vert$), the energy $|\hat{Z}(\omega ){|}^2$ is concentrated at low frequencies ($\omega \approx 0$, and equivalently $\omega \approx F^{\prime }$, since the two ends of the DFT index range carry the same low frequency), while the noise $𝝐$ has uniform spectral energy across all frequencies. The denoising model can exploit this asymmetry by attending to temporal patterns at different frequency scales.

Latent Trajectories as Curves

A useful geometric perspective is to view the sequence of latent codes $({𝒛}^1,{𝒛}^2,\dots ,{𝒛}^{F^{\prime }})$ as a discrete curve in the latent space ${\mathrm{ℝ}}^{H^{\prime }\times W^{\prime }\times C^{\prime }}$. This curve-based view provides intuitions for how the diffusion model generates video.

fig:vdiff:latent-trajectory visualises this concept. For a natural video containing smooth motion (e.g., a camera pan or a walking person), the latent trajectory forms a smooth curve. For a video with abrupt scene changes (e.g., a cut between shots), the trajectory contains discontinuities.

Implications for Diffusion Model Design

The temporal structure of the latent space has several important implications for the design of the diffusion model that operates within it.

Temporal attention is essential.

Because adjacent latent frames are correlated (as guaranteed by Theorem 3), the denoising model must attend across the temporal dimension to exploit these correlations. A model that processes each latent frame independently would ignore the most predictable structure in the data. This motivates the temporal attention mechanisms and 3D architectures developed in sec:vdiff:dit,sec:vdiff:factored-attn.

Noise scheduling can be temporal-aware.

The spectral concentration result (Lemma 2) suggests that temporal low-frequency components of the latent video should be denoised first (they contain the large-scale motion and scene structure), while high-frequency temporal details can be added later. This matches the coarse-to-fine generation observed in practice.

The diffusion model generates curves, not frames.

From the geometric perspective of Latent Trajectories as Curves, the diffusion model's task is not to generate $F^{\prime }$ independent latent frames but to generate a smooth curve in the latent space. This is a lower-dimensional problem than generating $F^{\prime }$ independent points, because the curve is constrained by temporal continuity.

Latent Space Quality and Downstream Diffusion

The quality of the latent space directly affects the quality of the downstream diffusion model. A well-structured latent space has several desirable properties that simplify the diffusion task.

Connecting Latent Compression to Video Diffusion

This section has established the mathematical foundations of latent space temporal structure. The key results can be summarised as follows:

1. Temporal smoothness transfers (Theorem 3): Lipschitz encoders map smooth pixel-space videos to smooth latent trajectories.

2. Spectral concentration (Lemma 2): smooth latent trajectories have energy concentrated at low temporal frequencies, creating a spectral separation between signal and noise that aids denoising.

3. First-frame conditioning (Definition 19): treating the first frame specially provides an anchor point for generation and enables image-to-video tasks.

4. Dimensionality reduction (Proposition 11): temporal smoothness constrains the latent trajectory to a small region of the full latent space, reducing the effective dimensionality of the generation problem.

Together, these results explain why latent video diffusion works as well as it does. The autoencoder compresses the video into a manageable latent space; the temporal structure of natural video ensures that the latent trajectories are smooth and low-dimensional; and the diffusion model exploits this structure to generate coherent video by tracing plausible curves through the latent space.

The next five sections build on this foundation to develop the architecture of the diffusion model that runs inside the latent space. The Diffusion Transformer (Video Diffusion Transformer (DiT)) is designed to process spatiotemporal latent tensors, cutting them into spacetime patches and modulating each block by the diffusion timestep. Because full spatiotemporal attention over those patches is unaffordable, Factored Attention for Video factorises it into spatial and temporal passes-the mechanism that exploits, at the level of the architecture, exactly the temporal smoothness established here. 3D Positional Encoding supplies the 3D positional encodings that tell those attention layers where in space and time each token sits, and Conditioning Mechanisms develops the conditioning pathways, including the concatenation route that carries the first-frame anchor of Definition 19 into image-to-video generation. Mixture-of-Experts for Video closes the architectural pillar by buying back capacity with mixture-of-experts layers. Only then, with an architecture in hand, do we return to the training objective and the noise schedule.

Video Diffusion Transformer (DiT)

The latent video diffusion framework developed in the preceding sections requires a neural network backbone that can process spatiotemporal latent tensors $𝖹\in {\mathrm{ℝ}}^{F^{\prime }\times H^{\prime }\times W^{\prime }\times C^{\prime }}$ and predict either the noise $𝝐$, the clean data ${𝖹}_0$, or the velocity field ${𝒗}_t$ at each denoising step. Early video diffusion models inherited the U-Net architecture from image diffusion, inflating 2D convolutions to 3D and inserting temporal attention layers between spatial blocks. While functional, this approach suffers from several limitations: the U-Net's hierarchical structure complicates scaling, the interleaving of convolution and attention creates architectural heterogeneity, and the inductive biases of convolution (locality, translation equivariance) may be unnecessarily restrictive for a model that already operates in a compressed latent space.

The Diffusion Transformer (DiT), introduced by Peebles and Xie [9] for images and subsequently extended to video by multiple groups, replaces the U-Net entirely with a plain transformer. The key insight is that, after latent compression, the effective resolution is small enough that a transformer operating on patch tokens can match or exceed U-Net performance while offering superior scaling behaviour.

In this section, we develop the DiT architecture for video from first principles. We begin with the spatiotemporal patchification that converts a latent tensor into a token sequence, then describe the adaptive layer normalisation mechanism that conditions the transformer on the diffusion timestep, and finally compare the scaling properties of DiT and U-Net architectures.

Spatiotemporal Patchification

The first step in processing a latent video tensor with a transformer is to convert the 3D grid of latent values into a 1D sequence of tokens. This is accomplished by dividing the tensor into non-overlapping 3D patches and projecting each patch to a token vector.

The patchification process is illustrated in fig:vdiff:patchify. Note the analogy with the Vision Transformer (ViT): just as ViT converts a 2D image into a sequence of 2D patch tokens, the video DiT converts a 3D latent tensor into a sequence of 3D patch tokens. The critical difference is that each patch spans multiple frames, so the initial token embedding already captures local spatiotemporal structure.

The DiT Block with Adaptive Layer Normalisation

After patchification, the token sequence $({𝒉}_1^{(0)},\dots ,{𝒉}_N^{(0)})$ is processed by a stack of $L$ transformer blocks. Each block follows the standard pre-normalisation transformer design (LayerNorm, Attention, LayerNorm, MLP), but with a critical modification: the layer normalisation is adaptive, conditioned on the diffusion timestep $t$ and optionally on other global conditioning signals.

The full DiT block combines adaLN with multi-head self-attention (MHSA) and a position-wise feed-forward network (FFN), using the adaLN-Zero initialisation strategy.

Mathematical Analysis of adaLN

To understand why adaLN is an effective conditioning mechanism, we analyse its effect on the feature distribution at each layer.

Comparison with U-Net Architectures

The shift from U-Net to DiT for video diffusion is not merely an architectural preference; it reflects fundamental differences in how the two architectures scale with model size and data.

Factored Attention for Video

The self-attention mechanism is both the greatest strength and the primary computational bottleneck of the video Diffusion Transformer. Given a token sequence of length $N$, standard multi-head self-attention has time and memory complexity $O(N^{2}d)$ and $O(N^{2}h)$, respectively, where $d$ is the hidden dimension and $h$ is the number of heads. For video, the token count $N=N_t\cdot N_h\cdot N_w$ is the product of temporal, height, and width dimensions, which can easily reach tens of thousands. Full spatiotemporal attention over all tokens is often computationally prohibitive, motivating the development of factored attention strategies that decompose the joint spatiotemporal attention into smaller, more manageable operations.

In this section, we analyse the computational cost of full spatiotemporal attention, develop the mathematical framework for factored attention, prove approximation bounds, and describe practical variants including joint attention and windowed attention.

Full Spatiotemporal Attention

We begin by quantifying the cost of the naive approach: computing attention over all video tokens simultaneously.

Spatial-Then-Temporal Factorisation

The key observation that enables efficient video attention is that the spatiotemporal token grid has inherent structure: tokens at the same spatial location across different frames are temporally related, while tokens within the same frame are spatially related. Factored attention exploits this structure by decomposing the joint attention into separate spatial and temporal operations.

The factored attention achieves dramatic computational savings, but at a cost: information can only flow between spatially distant positions in different frames indirectly, through the composition of spatial and temporal attention. It is tempting to try to bound the resulting error against full attention, and it is worth being precise about why no useful bound of that kind exists.

What can be stated cleanly is the effect of one specific operation: masking the cross terms inside a single attention layer and renormalising. This is the “axial” attention pattern in which each query attends to its own frame and its own spatial column, and it is the closest single-layer object to the factorisation.

Alternating Block Design

In practice, factored attention is implemented by alternating between spatial attention blocks and temporal attention blocks. Each block is a full DiT block (Definition 26) with the attention restricted to a subset of tokens.

Joint Attention with Text Tokens

Not every design in this section restricts the attention graph; this one enlarges it. Joint attention, used in models such as CogVideoX [17], concatenates the text tokens with the video tokens and runs one attention over the combined sequence, rather than relegating text to a separate cross-attention mechanism. It is important to be clear that this is not a saving: the cost is quadratic in the combined length $N_c+N_v$, strictly more than the $O(N_v^2)$ of video self-attention alone. Joint attention is adopted for the quality of the text-video interaction it enables, and it is affordable only because $N_c\ll N_v$.

3D Window Attention

A complementary approach to factored attention is to restrict attention to local 3D windows, analogous to the Swin Transformer for images.

3D Positional Encoding

Transformers are permutation-equivariant by design: without positional information, a transformer cannot distinguish between a token representing the top-left corner of frame 1 and a token representing the bottom-right corner of frame 50. For video generation, positional encoding is not merely a technical detail; it is the mechanism by which the model understands the fundamental distinction between spatial and temporal relationships.

In this section, we develop three approaches to 3D positional encoding: the direct extension of sinusoidal encodings to three axes, Rotary Position Embeddings (RoPE) adapted to 3D grids, and learnable positional embeddings. We analyse the extrapolation properties of each approach and discuss their implications for generating videos at resolutions and durations not seen during training.

Why Position Matters for Video

To appreciate the importance of positional encoding, consider what a video DiT would produce without it.

Formally, each token in the video DiT corresponds to a 3D grid position $(n_t,n_h,n_w)$ where $n_t\in \{0,\dots ,N_t-1\}$, $n_h\in \{0,\dots ,N_h-1\}$, and $n_w\in \{0,\dots ,N_w-1\}$. The positional encoding must map this triple to a $d$-dimensional vector (or a modification of the attention computation) that enables the transformer to reason about spatiotemporal relationships.

3D Sinusoidal Positional Encoding

The simplest approach extends the classical 1D sinusoidal encoding of Vaswani et al. to three independent axes.

3D Rotary Position Embeddings (RoPE)

Rotary Position Embeddings (RoPE), introduced by Su et al. [65], encode positional information by rotating query and key vectors in the attention computation. The 3D extension applies independent rotations for each spatial and temporal axis.

Resolution and Length Generalisation

A critical practical question is whether a model trained at one spatiotemporal resolution can generate video at a different resolution or duration without retraining. The positional encoding scheme determines the answer.

To mitigate extrapolation failures, several techniques have been proposed.

Conditioning Mechanisms

A video diffusion model rarely generates video unconditionally. In practice, generation is guided by text descriptions, reference images, camera parameters, motion maps, or combinations thereof. The conditioning mechanism is the architectural component that injects this external information into the denoising process. Different types of conditioning signals have different characteristics (sequential text, spatial images, scalar timesteps), and each requires a tailored injection pathway.

In this section, we formalise three principal conditioning pathways used in modern video DiT architectures: cross-attention for sequential conditioning (text), adaptive layer normalisation for global conditioning (timestep, noise level), and concatenation for spatial conditioning (reference images, masks). We then analyse how these mechanisms interact and prove an information-theoretic result relating conditioning to entropy reduction.

Three Conditioning Pathways

Cross-Attention for Text Conditioning

Text conditioning is the most common form of control in video generation. Given a text prompt (e.g., “A cat playing piano in a jazz club”), a pre-trained text encoder produces a sequence of contextual embeddings that guide the denoising process.

Dual Text Encoders

A number of generation systems employ two text encoders with complementary strengths: a language model (such as T5 or LLaMA) that provides rich semantic embeddings, and a contrastive model (such as CLIP) that provides embeddings aligned with visual features. This is a design choice rather than a universal one - CogVideoX, for instance, conditions on T5-XXL alone - but it is common enough, and instructive enough, to be worth setting out carefully.

Concatenation Conditioning

For conditioning signals that have the same spatiotemporal structure as the video (reference images, masks, depth maps), the simplest injection pathway is channel-wise concatenation before patchification.

Conditioning as Entropy Reduction

We now provide an information-theoretic perspective on conditioning, showing that each conditioning signal reduces the entropy of the generation distribution in a quantifiable way.

Mixture-of-Experts for Video

Scaling a video diffusion model to billions of parameters improves quality, but the computational cost of each denoising step grows proportionally with the parameter count. For video, where each sample requires dozens of sequential denoising steps on a high-dimensional latent tensor, this cost quickly becomes prohibitive. Mixture-of-Experts (MoE) architectures address this tension by decoupling model capacity (total parameters) from per-token computation (FLOPs per forward pass). The idea is simple: replace the dense feed-forward network in each transformer block with a collection of expert sub-networks, and for each token, route it to only a small subset of experts. The total parameter count can be enormous (providing capacity to represent complex functions), while the per-token FLOPs remain manageable (because only a few experts are activated per token).

In this section, we formalise the MoE layer, analyse the routing mechanism, derive the load balancing loss, and discuss the intriguing phenomenon of expert specialisation by noise level in video diffusion models.

The MoE Layer

Top-$k$ Routing

The most common routing strategy selects the top-$k$ experts for each token based on the gating scores.

Load Balancing

A well-known failure mode of MoE is expert collapse: the router learns to always send tokens to the same small subset of experts, leaving most experts unused. This wastes parameters and reduces the effective capacity of the model.

Expert Specialisation by Noise Level

A remarkable empirical observation in MoE-based video diffusion models is that different experts tend to specialise for different noise levels (timesteps). At high noise ($t\approx 1$), certain experts dominate, handling coarse, global structure prediction. At low noise ($t\approx 0$), different experts activate, handling fine detail refinement.

Capacity Factor and Auxiliary Losses

In distributed training, MoE layers introduce a unique challenge: tokens must be dispatched to the devices hosting each expert, and imbalanced routing can cause memory overflow on some devices while leaving others idle. The capacity factor controls how many tokens each expert can process.

From Architecture to Objective

sec:vdiff:dit,sec:vdiff:factored-attn,sec:vdiff:3d-pe,sec:vdiff:conditioning,sec:vdiff:moe have specified a network. Its design reduces to five decisions, each of which is a purchase made against a fixed compute budget:

1. Tokenisation (Spatiotemporal Patchification): spatiotemporal patches of size $p_t\times p_h\times p_w$ turn the latent tensor into $N=N_t{N}_h{N}_w$ tokens. The patch size is the cheapest available lever on cost, and the bluntest one on quality.

2. Conditioning on the noise level (The DiT Block with Adaptive Layer Normalisation): adaLN supplies a global, $t$-dependent per-channel gain and offset, and the adaLN-Zero gates make every block the identity at initialisation. As Proposition 13 makes precise, this buys trainability, not representational power.

3. The attention pattern (Factored Attention for Video): full spatiotemporal attention is $O(N^2)$ and, at realistic token counts, simply unaffordable. Factorisation, alternating blocks and 3D windows each restrict the attention graph in a different way, with provable savings and empirical - not provable - quality costs.

4. Position (3D Positional Encoding): 3D RoPE makes the attention logit a function of the displacement $(\mathrm{\Delta }{n}_t,\mathrm{\Delta }{n}_h,\mathrm{\Delta }{n}_w)$ alone, which is what allows a model trained at one resolution and duration to be run at another.

5. Capacity (Mixture-of-Experts for Video): mixture-of-experts decouples parameter count from per-token FLOPs, at the price of a routing problem that must be regularised into balance.

What this network is for has so far been left implicit. We have assumed only that it consumes a noised latent and a timestep and emits a prediction, without committing to what that prediction is, how the noise levels are distributed, or how the trained model is run at inference. Those are the subjects of the sections that follow, and each of them turns out to require modification for video rather than inheritance from image diffusion.

Noise Schedules for Video takes up the noise schedule. A schedule tuned for a ${256}^2$ image is systematically too easy for a video latent that is an order of magnitude larger, because at equal per-element signal-to-noise ratio a higher-dimensional observation carries more evidence about the clean sample; the remedy is a shift of the log-SNR curve by the ratio of dimensionalities, tempered by the fact that temporal correlation makes the effective dimensionality of a video smaller than its nominal one. Flow Matching for Video replaces the diffusion SDE with flow matching along straight optimal-transport paths, and shows why straightness is what makes few-step sampling viable; it also covers rectified flow, the v-prediction parameterisation, and the stochastic-interpolant framework that contains both flow matching and diffusion as special cases. Sampling Acceleration attacks inference cost along three orthogonal axes - fewer steps via deterministic solvers and consistency distillation, cheaper steps via temporal feature caching, and better solvers - and shows how the savings compound. Finally, Training at Scale describes how such models are actually trained: multi-stage curricula that begin with images and end at high resolution, mixed image-video batches, data curation pipelines that discard the great majority of the raw corpus, and the compute budgets these recipes imply.

Only after those pieces are in place does the question of control - guidance, image conditioning, camera trajectories, personalisation - become well posed, and that is where the chapter turns next.

Noise Schedules for Video

The noise schedule is the heartbeat of every diffusion model. It determines how quickly the data signal is destroyed during the forward process and, consequently, how the denoiser must allocate its capacity across timesteps. For images, the design of noise schedules is well-studied: the cosine schedule of Nichol and Dhariwal, the linear schedule of Ho et al., and the learned schedules of Kingma et al. all provide effective ways to spread the denoising task across a manageable number of steps.

For video, however, the situation is fundamentally different. A video tensor $𝖹\in {\mathrm{ℝ}}^{F\times H\times W\times C}$ contains far more elements than an image tensor $𝒛\in {\mathrm{ℝ}}^{H\times W\times C}$, and those elements are highly correlated along the temporal axis. Both of these facts have profound consequences for the signal-to-noise ratio (SNR) at each timestep, and ignoring them leads to degenerate training behaviour. In this section, we develop the theory of noise schedules for video, starting from the observation that standard image schedules fail when applied naively to video data.

Why Image Schedules Fail for Video

Recall the forward process of a continuous-time diffusion model. Given a clean data sample ${𝖹}_0$, the noisy version at time $t\in [0,1]$ is (Forward ZT)$$\begin{array}{ccc}& {𝖹}_t={\alpha }_t{𝖹}_0+{\sigma }_t𝝐,𝝐\sim \mathrm{Normal}(0,𝑰),& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${\alpha }_t$ and ${\sigma }_t$ define the schedule. The signal-to-noise ratio at time $t$ is (SNR DEF)$$\begin{array}{ccc}& \text{SNR}(t)=\frac{{\alpha }_t^2}{{\sigma }_t^2}.& \text{<span class="tml-eqn"></span>}\end{array}$$ For images, a typical cosine schedule produces $\text{SNR}(0)\gg 1$ (nearly clean) and $\text{SNR}(1)\ll 1$ (nearly pure noise), with a smooth transition between the two extremes.

The problem arises when we consider what the denoiser “sees” at each timestep. At intermediate $t$, the denoiser receives a mixture of signal and noise. For the denoiser to learn effectively, it must encounter timesteps where the task is neither trivially easy (pure signal) nor impossibly hard (pure noise). The range of “useful” timesteps depends on the dimensionality of the data.

To make this precise, consider the log-SNR, defined as ${\lambda }_t=\log ({\alpha }_t^2/{\sigma }_t^2)$. For an image diffusion model at resolution $H\times W$ with $C$ latent channels, the effective dimensionality is $d_{\text{image}}=H\times W\times C$. For a video with $F$ latent frames, it is $d_{\text{video}}=F\times H\times W\times C$. The ratio $d_{\text{video}}/d_{\text{image}}=F$ can be large (e.g., $F=16$ to $33$ for typical video models).

The SNR Shift Principle

The solution, first proposed in Imagen Video [5] and analysed in detail by Chen [23], is to shift the noise schedule so that the SNR at each timestep is adjusted for the data dimensionality.

The shift factor $s<1$ when $d_{\text{video}}>d_{\text{ref}}$, which decreases the SNR at every timestep. Equivalently, the log-SNR curve is shifted downward, moving the “informative” range of timesteps earlier. This is the rule used in practice, and for spatial upscaling it is well supported empirically: natural images are redundant at every scale, so quadrupling the pixel count does buy the denoiser close to a fourfold gain in effective SNR.

Temporal Redundancy and the Shift It Justifies

The shift principle of Definition 48 counts every dimension of the video tensor as if it were a fresh copy of the signal. As Remark 44 observed, that is the perfectly-redundant limit. We now make the redundancy of a video measurable and derive the shift it actually justifies.

The right notion is a spectral one. Write $n$ for the ambient dimension of the clean latent and $𝑪$ for its covariance. If the $n$ coordinates were uncorrelated with equal variances, nothing could be averaged; if they were all copies of a single scalar, everything could be. The participation ratio $\mathrm{trace}{(𝑪)}^2/\mathrm{trace}({𝑪}^2)$ interpolates between these two extremes, and its reciprocal (normalised by $n$) is exactly the factor by which the SNR must be lowered.

Note that (a) and (b) bracket the truth from the two sides: real video sits strictly between them, so the correct shift is smaller in magnitude than the dimension-counting shift, by $\log (F/r_F)=\log F_{\mathrm{eff}}$ nats.

Resolution-Adapted SNR

Modern video generation systems produce outputs at multiple resolutions, either through cascaded super-resolution or through direct generation at variable resolution with packing strategies. The noise schedule should adapt to the target resolution.

Visualising the SNR Shift

The following figure illustrates how the log-SNR curve shifts downward as the data dimensionality increases from a single image to increasingly long video sequences.

Implementation: Shifted Cosine Schedule

In practice, the shifted schedule is implemented by modifying the standard cosine schedule. The reference cosine schedule defines (Cosine REF)$$\begin{array}{ccc}& {\alpha }_t^{\text{ref}}=\cos \left(\frac{\pi t}{2}\right),{\sigma }_t^{\text{ref}}=\sin \left(\frac{\pi t}{2}\right),& \text{<span class="tml-eqn"></span>}\end{array}$$ so that ${\text{SNR}}_{\text{ref}}(t)={\cot }^2(\pi t/2)$. The shifted video schedule is obtained by applying the log-SNR offset: (Shifted Cosine)$$\begin{array}{ccc}& {\text{SNR}}_{\text{video}}(t)={\cot }^2\left(\frac{\pi t}{2}\right)\cdot \frac{d_{\text{ref}}}{d_{\text{video}}}.& \text{<span class="tml-eqn"></span>}\end{array}$$ To express this in terms of $({\alpha }_t,{\sigma }_t)$ satisfying ${\alpha }_t^2+{\sigma }_t^2=1$ (variance-preserving), we define (Shifted Alpha Sigma)$$\begin{array}{ccc}& {\alpha }_t=\frac{1}{\sqrt{1+1/{\text{SNR}}_{\text{video}}(t)}},{\sigma }_t=\frac{1}{\sqrt{1+{\text{SNR}}_{\text{video}}(t)}}.& \text{<span class="tml-eqn"></span>}\end{array}$$

Flow Matching for Video

The diffusion models we have discussed so far are formulated as stochastic processes: the forward process adds noise through a stochastic differential equation (SDE), and the reverse process removes it by running the SDE backwards in time. An alternative formulation, flow matching, replaces the stochastic process with a deterministic ordinary differential equation (ODE) that transports samples from a noise distribution to the data distribution. This ODE-based viewpoint offers several advantages: simpler training objectives, straighter sampling trajectories (enabling fewer integration steps), and a natural connection to optimal transport theory.

Flow matching has become the dominant training paradigm for recent video diffusion models: Movie Gen [32] and Stable Diffusion 3 [24] are trained with an explicit flow matching objective on linear paths, and several video systems have followed. The picture is not uniform, however, and it is worth being accurate about it. CogVideoX [17] uses the $v$-prediction parameterisation of a variance-preserving diffusion, which is closely related to but not identical with flow matching (Proposition 30); Stable Video Diffusion [8] is trained in the EDM framework with a preconditioned denoiser and a log-normal noise distribution, not with flow matching at all. In this section we develop the mathematical foundations of flow matching for video, building on the framework of Lipman et al. [25] and Liu et al. [26].

From Diffusion to Flow

Recall the forward process of a diffusion model: ${𝖹}_t={\alpha }_t{𝖹}_0+{\sigma }_t𝝐$, where $𝝐\sim \mathrm{Normal}(0,𝑰)$. This can be viewed as defining a probability path $p_t(𝖹)$ that interpolates between the data distribution $p_0=p_{\text{data}}$ and the noise distribution $p_1=\mathrm{Normal}(0,𝑰)$.

There exists a time-dependent velocity field ${𝒖}_t(𝖹)$ such that the probability path $p_t$ is the pushforward of $p_0$ along the flow induced by ${𝒖}_t$. Formally, $p_t$ satisfies the continuity equation: (Continuity)$$\begin{array}{ccc}& \frac{\partial p_t}{\partial t}+\nabla \cdot (p_t{𝒖}_t)=0.& \text{<span class="tml-eqn"></span>}\end{array}$$ The key insight of flow matching is that we can directly learn ${𝒖}_t$ without ever working with the SDE formulation. Instead of training a score function ${\nabla }_{𝖹}\log p_t(𝖹)$ and converting it to a velocity field, we train a neural network $v_{\theta }({𝖹}_t,t)$ to predict the velocity field directly.

The Video Flow Matching Objective

We now formalise the flow matching framework for video generation. Let ${𝖹}_0\sim p_{\text{data}}$ be a clean latent video and ${𝖹}_1\sim \mathrm{Normal}(0,𝑰)$ be pure noise.

The definition is general; it applies to any choice of interpolation between ${𝖹}_0$ and ${𝖹}_1$. The most common choice, and the one that connects to optimal transport theory, is the linear (or optimal transport) path.

Under the OT path convention of Definition 50, the VFM loss becomes particularly simple: (VFM OT LOSS)$$\begin{array}{ccc}& {ℒ}_{\text{VFM}}(\theta )={𝔼}_{t,{𝖹}_0,{𝖹}_1}\Vert v_{\theta }((1-t){𝖹}_0+t{𝖹}_1,t)-({𝖹}_1-{𝖹}_0){\Vert }^2.& \text{<span class="tml-eqn"></span>}\end{array}$$ The training target ${𝖹}_1-{𝖹}_0$ is simply the displacement from data to noise, a fixed vector for each training pair $({𝖹}_0,{𝖹}_1)$; sampling integrates the same field in the opposite direction.

Why OT Paths Are Preferred

The OT path is not the only possible interpolation. One could also use the variance-preserving (VP) path defined by ${𝖹}_t=\cos (\pi t/2){𝖹}_0+\sin (\pi t/2){𝖹}_1$, which corresponds to the DDPM forward process with the cosine schedule of (Cosine REF). However, the OT path has a crucial geometric advantage.

Rectified Flow and Iterative Straightening

While the OT path is straight for individual sample pairs, the marginal velocity field $v_{\theta }({𝖹}_t,t)$ may still generate curved trajectories, because different data points ${𝖹}_0$ paired with different noise points ${𝖹}_1$ create crossing paths in the shared space. Rectified flow [26] addresses this by iteratively straightening the learned flow.

v-Prediction Parameterisation

The flow matching velocity $v_{\theta }({𝖹}_t,t)$ is closely related to the classical parameterisations used in diffusion models. We now make this connection precise.

Under the linear interpolation ${𝖹}_t=(1-t){𝖹}_0+t{𝖹}_1$ (where ${𝖹}_0$ is data at $t=0$ and ${𝖹}_1\sim \mathrm{Normal}(0,𝑰)$ is noise at $t=1$), the velocity field can be decomposed in terms of either the noise prediction ${𝝐}_{\theta }$ or the data prediction ${\widehat{𝖹}}_{0,\theta }$.

Training Algorithm

We now present the complete training procedure for video flow matching.

Sampling Algorithm

Given a trained velocity network $v_{\theta }$, generating a video requires solving the ODE (FLOW ODE)$$\begin{array}{ccc}& \frac{\mathrm{d}{𝖹}_t}{\mathrm{d}t}=v_{\theta }({𝖹}_t,t,𝒄),{𝖹}_1\sim \mathrm{Normal}(0,𝑰),& \text{<span class="tml-eqn"></span>}\end{array}$$ backwards in time, from the noise endpoint $t=1$ down to the data endpoint $t=0$ (Remark 49). The simplest approach is the Euler method, which for a backward integration subtracts $\mathrm{\Delta }t$ times the velocity at each step.

Stochastic Interpolants Perspective

The flow matching framework can be generalised through the stochastic interpolants framework of Albergo et al. [66], which provides a unified view encompassing both deterministic flows and stochastic diffusion processes.

Setting $c(t)=0$ recovers the deterministic flow matching framework. Setting $c(t)>0$ adds stochasticity to the interpolation, which can improve mode coverage at the cost of requiring more sampling steps. The OT flow matching framework corresponds to $a(t)=1-t$, $b(t)=t$, $c(t)=0$.

Sampling Acceleration

Generating a high-quality video with a flow matching or diffusion model typically requires 25 to 100 ODE or SDE steps, each of which involves a full forward pass through a large transformer. For a model with billions of parameters operating on tens of thousands of spatiotemporal tokens, a single forward pass may take 1 to 5 seconds on a modern GPU. Multiplying by the number of steps, generating a single short video clip can take minutes, and producing long, high-resolution content can take hours.

This computational burden is the single largest barrier to the practical deployment of video diffusion models. In this section, we survey three complementary strategies for accelerating sampling: (1) deterministic sampling with DDIM-type methods, (2) consistency distillation to reduce the number of required steps, and (3) temporal feature caching to reduce the cost of each step.

Deterministic Sampling with DDIM

The Denoising Diffusion Implicit Models (DDIM) framework [10] provides a family of non-Markovian reverse processes that share the same marginal distributions as the DDPM reverse process but allow deterministic sampling.

For video diffusion, the DDIM update rule at timestep $t$ with noise schedule $({\alpha }_t,{\sigma }_t)$ is: (DDIM Update)$$\begin{array}{ccc}& {𝖹}_{t-1}={\alpha }_{t-1}\underset{\text{predicted }{𝖹}_0}{\underbrace{\left(\frac{{𝖹}_t-{\sigma }_t{𝝐}_{\theta }({𝖹}_t,t)}{{\alpha }_t}\right)}}+\underset{\text{direction pointing to }{𝖹}_t}{\underbrace{\sqrt{{\sigma }_{t-1}^2-{\eta }^2{\tilde{\sigma }}_t^2}\cdot {𝝐}_{\theta }({𝖹}_t,t)}}+\underset{\text{random noise}}{\underbrace{\eta {\tilde{\sigma }}_t{𝝐}^{\prime }}},& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${𝝐}^{\prime }\sim \mathrm{Normal}(0,𝑰)$, ${\tilde{\sigma }}_t={\sigma }_{t-1}{\sigma }_t^{-1}\sqrt{1-{\alpha }_t^2/{\alpha }_{t-1}^2}$, and $\eta \in [0,1]$ controls the stochasticity.

Consistency Distillation for Video

Consistency models [60] offer a principled approach to few-step generation. The core idea is to train a model that maps any point on the ODE trajectory directly to the trajectory's endpoint (the clean data), enabling one-step generation.

The consistency loss enforces a simple property: any two points on the same ODE trajectory should map to the same output, namely the clean endpoint ${𝖹}_0$ of that trajectory. If $f_{\varphi }$ satisfies this property exactly, then evaluating $f_{\varphi }({𝖹}_1,1)$ at the initial noise point directly produces the clean video, enabling one-step generation.

Temporal Feature Caching

The strategies above reduce the number of denoising steps. A complementary approach is to reduce the cost per step by reusing computations across consecutive timesteps. This is motivated by a key observation: the features computed by the denoiser change slowly across adjacent timesteps.

The caching strategy is particularly effective for video diffusion because the denoising trajectory is smooth: consecutive timesteps produce similar intermediate features, especially in the early (high-noise) and late (low-noise) stages of the trajectory where the model's behaviour changes slowly.

Caching Strategies in Practice

Several recent works have developed practical caching strategies for video diffusion.

Distillation Pipeline

Consistency distillation (Definition 53) is one instance of a broader family of distillation approaches. The general idea is to train a “student” model that mimics the output of a “teacher” model (the full multi-step sampler) in fewer steps.

Combining Acceleration Strategies

The three acceleration strategies described above are complementary and can be combined for multiplicative speedup:

1. Step reduction (consistency distillation or progressive distillation): reduce from 50 steps to 4 to 8 steps, yielding $6\times$ to $12\times$ speedup.

2. Feature caching (PAB + T-GATE): reduce the cost per step by $1.5\times$ to $2.5\times$ through selective reuse of attention outputs.

3. Higher-order solvers (DPM-Solver, Heun): achieve the same quality with $1.5\times$ to $2\times$ fewer steps than Euler.

Training at Scale

Training a state-of-the-art video diffusion model is one of the most resource-intensive tasks in machine learning. The models contain billions of parameters, the training data consists of millions of video clips, and the training runs consume thousands of GPU-days. Naive approaches (randomly initialising a large model and training it end-to-end on full-resolution video) are both wasteful and unstable. Instead, every successful video generation system uses a carefully staged training procedure that progressively increases the resolution, duration, and complexity of the training data.

In this section, we formalise the multi-stage training paradigm, derive a transfer learning bound that explains why image pretraining helps, and describe the progressive resolution curriculum used by leading systems.

Multi-Stage Training

The multi-stage training paradigm decomposes the video generation training process into a sequence of stages, each building on the previous one.

The typical progression for a modern video generation system is:

1. Stage 1: Image pretraining. Train (or fine-tune) the model on a large dataset of images, treating each image as a single-frame “video.” This stage teaches the model spatial generation (objects, textures, composition) using the abundant supply of image data.

2. Stage 2: Low-resolution video. Extend the model to short video clips at low resolution (e.g., 16 frames at $256\times 256$). The temporal layers (temporal attention, temporal convolutions) are initialised randomly or from a motion module. This stage teaches basic motion and temporal coherence.

3. Stage 3: Medium-resolution video. Increase the resolution and duration (e.g., 33 frames at $480\times 480$). The model learns finer motion details and higher-quality spatial features.

4. Stage 4: High-resolution video. Train at the target resolution (e.g., 65 frames at $720\text{p}$ or $1080\text{p}$) on a curated, high-quality dataset. This final stage refines the model's output quality.

Transfer Learning Bound

Why does image pretraining help with video generation, and what exactly is left for the video stages to learn? The following theorem answers both questions by splitting the video score into a per-frame part - which an image model already supplies - and a temporal correction, and then computing the size of the correction in a tractable model.

Progressive Resolution Curriculum

The resolution progression from low to high is not merely a convenience; it provides a coarse-to-fine learning signal that stabilises training and improves final quality.

Several design choices govern the curriculum:

1. Image stages precede video stages. Stages 1 and 2 train only on images, building spatial generation capability. The temporal layers are either absent or frozen during these stages.

2. Resolution increases gradually. Jumping directly from ${256}^2$ to $720\text{p}$ often leads to training instability (loss spikes, mode collapse). Intermediate steps allow the model to adapt its spatial features progressively.

3. Frame count increases with resolution. At low resolution, training with many frames is computationally feasible and teaches long-range temporal coherence. At high resolution, memory constraints limit the frame count, but the temporal knowledge from earlier stages transfers.

4. Batch size decreases as resolution increases. GPU memory is the binding constraint, so the number of samples per step falls as each sample grows. It does not fall fast enough to hold the work per step fixed: for tab:vdiff:curriculum the total pixels processed per optimisation step are $1.3\times {10}^8$, $2.7\times {10}^8$, $2.7\times {10}^8$, $5.4\times {10}^8$ and $9.6\times {10}^8$ for stages 1 to 5 - a monotone sevenfold increase. Later stages are therefore more expensive per step as well as per sample, which is why they are also the shortest.

5. High-quality data in later stages. Early stages can use large, noisy datasets (web-scraped videos). Later stages use smaller, carefully curated datasets with high aesthetic quality, stable motion, and accurate captions.

Mixed Image-Video Training

A key practical technique is to mix image and video data within the same training batch, even during video-focused stages. This prevents catastrophic forgetting of spatial generation quality during video fine-tuning.

The mixed training objective is: (Mixed LOSS)$$\begin{array}{ccc}& {ℒ}_{\text{mixed}}(\theta )=(1-{\lambda }_{\text{vid}})\cdot {𝔼}_{{𝒛}_0\sim p_{\text{image}}}[{ℒ}_{\text{FM}}(\theta ;{𝒛}_0)]+{\lambda }_{\text{vid}}\cdot {𝔼}_{{𝖹}_0\sim p_{\text{video}}}[{ℒ}_{\text{FM}}(\theta ;{𝖹}_0)],& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${\lambda }_{\text{vid}}\in (0,1)$ controls the mixing ratio and ${ℒ}_{\text{FM}}$ is the flow matching loss. Images are treated as single-frame videos ($F=1$) and processed by the same model architecture, with the temporal attention layers receiving a single token in the temporal dimension.

Data Curation

The quality of the training data is at least as important as the model architecture and training recipe. Video data from the web is noisy, poorly captioned, and highly variable in quality. Careful curation is essential.

Scaling Laws and Compute Budgets

The relationship between model size, data size, and generation quality for video diffusion models is an active area of research. While precise scaling laws analogous to the Chinchilla scaling laws for language models have not yet been established, several empirical observations guide practice.

Practical Training Considerations

Beyond the high-level training pipeline, several practical considerations are critical for training video diffusion models at scale.

1. Learning rate scheduling. Each stage typically uses a warmup period (1K to 5K steps) followed by cosine decay. The peak learning rate often decreases across stages (e.g., $3\times {10}^{-4}$ for image pretraining, $1\times {10}^{-4}$ for low-resolution video, $5\times {10}^{-5}$ for high-resolution video) to prevent overwriting the knowledge from earlier stages.

2. Gradient accumulation. At high resolution, the batch size per GPU is severely limited (often 1 to 2 videos). Gradient accumulation across multiple micro-batches is used to achieve an effective batch size of 16 to 64, which is necessary for stable optimisation.

3. Model parallelism. For models exceeding 5B parameters, single-GPU training is impossible. Tensor parallelism (splitting layers across GPUs), sequence parallelism (splitting the token sequence), and pipeline parallelism are all used. The Open-Sora project [31] provides implementations of these parallelism strategies specifically optimised for video DiT models.

4. Noise schedule adaptation. The noise schedule (Noise Schedules for Video) must be adjusted at each stage to account for the changing resolution. As the resolution increases, the log-SNR shift $\mathrm{\Delta }\lambda$ from Proposition 28 increases in magnitude, requiring the schedule to be shifted toward higher noise levels.

5. EMA for sampling. An exponential moving average of the model weights is maintained throughout training and used for evaluation and sampling. The EMA decay rate is typically 0.9999 to 0.99999, with lower decay (faster averaging) in early stages and higher decay (slower averaging) in later stages where the model is closer to convergence.

Summary and Looking Ahead

This section has covered the core training methodology for large-scale video diffusion models. The key insights are:

1. Noise schedules must adapt to the redundancy of video data (Noise Schedules for Video). The log-SNR curve is shifted downward; the dimension-counting rule $-\log (d_{\text{video}}/d_{\text{ref}})$ is the perfectly-redundant limit and is a good approximation on the spatial axes, while on the temporal axis the factor $F$ should be replaced by the measured redundancy $r_F\le F$ of Theorem 6.

2. Flow matching is the dominant training paradigm (Flow Matching for Video). The OT path provides straight trajectories that enable efficient sampling, and the v-prediction parameterisation is numerically stable across all timesteps.

3. Sampling acceleration is essential for practical deployment (Sampling Acceleration). Consistency distillation, temporal feature caching, and higher-order solvers combine to yield order-of-magnitude speedups.

4. Multi-stage training with image pretraining is the established recipe (Training at Scale). Progressive resolution curricula, mixed image-video training, and careful data curation are all critical for achieving state-of-the-art quality.

We turn next to the third pillar, control. The following sections develop classifier-free guidance in the multi-condition setting that video demands, image-to-video synthesis, camera control through Plücker ray conditioning, video ControlNets for structural conditioning, LLM-based prompt enhancement in the text-to-video pipeline, and parameter-efficient personalisation with LoRA, DreamBooth and their quantised variants. Evaluation - the Fréchet Video Distance, VBench and human protocols - is taken up later, once there is something to evaluate.

Classifier-Free Guidance for Video

Classifier-free guidance (CFG) has become the default mechanism for improving sample quality in diffusion models. In the image domain, the idea is straightforward: during training, randomly drop the conditioning signal and learn both a conditional and an unconditional denoiser; at inference, extrapolate away from the unconditional prediction toward the conditional one. For video, the situation is considerably richer because the conditioning signal is no longer a single text prompt. A video diffusion model may be conditioned on text descriptions, reference images, camera trajectories, motion maps, audio tracks, or any combination of these. Each conditioning modality carries different information and may warrant a different guidance strength.

In this section, we formalise multi-condition CFG for video, establish its Bayesian interpretation, describe the dropout schedules used during training, and illustrate the geometric intuition behind guided denoising.

Review: Single-Condition CFG

We begin with a brief review of the single-condition case, following [62]. Let ${ϵ}_{\theta }({𝖹}_t,t,c)$ denote a noise-prediction network conditioned on signal $c$, and let ${ϵ}_{\theta }({𝖹}_t,t,⌀)$ denote the same network evaluated with the conditioning signal replaced by a null token $⌀$. The guided noise prediction is (CFG Single)$$\begin{array}{ccc}& {\widehat{ϵ}}_{\theta }({𝖹}_t,t,c)={ϵ}_{\theta }({𝖹}_t,t,⌀)+w({ϵ}_{\theta }({𝖹}_t,t,c)-{ϵ}_{\theta }({𝖹}_t,t,⌀)),& \text{<span class="tml-eqn"></span>}\end{array}$$ where $w\ge 1$ is the guidance scale. When $w=1$, we recover the standard conditional prediction. When $w>1$, we amplify the “direction” from the unconditional prediction toward the conditional one, producing samples that are more strongly aligned with $c$ at the cost of reduced diversity.

Multi-Condition CFG for Video

In video generation, the model typically receives multiple conditioning signals simultaneously. A text-to-video model conditioned on text $c_{\text{text}}$ and a reference image $c_{\text{img}}$ must balance fidelity to the textual description against visual consistency with the reference frame. Adding camera parameters $c_{\text{cam}}$ or motion trajectories $c_{\text{motion}}$ introduces further dimensions of control.

For the common case of text and image conditioning, this becomes (CFG TEXT IMG)$$\begin{array}{ccc}& {\widehat{ϵ}}_{\theta }={ϵ}_{\theta }({𝖹}_t,⌀)+w_{\text{text}}({ϵ}_{\theta }({𝖹}_t,c_{\text{text}})-{ϵ}_{\theta }({𝖹}_t,⌀))+w_{\text{img}}({ϵ}_{\theta }({𝖹}_t,c_{\text{img}})-{ϵ}_{\theta }({𝖹}_t,⌀)),& \text{<span class="tml-eqn"></span>}\end{array}$$ where we suppress the time argument for brevity. This formulation allows independent control over how strongly the generated video aligns with the text description versus the reference image. Systems that accept both a prompt and a reference frame - image-animation and video-continuation models - expose exactly this pair of dials, and the two weights are almost never equal: the reference frame pins down appearance, layout and lighting exactly, whereas the prompt constrains only what is described in words. A weaker weight on the more informative condition is therefore the usual setting; Example 41 collects the ranges that are typical in practice.

Guidance need not be a single scalar shared by the whole clip. The noise tensor has a frame axis, and nothing forces the guidance weight to be constant along it.

Compositional Guidance

The multi-condition formulation in Definition 57 treats each conditioning signal independently: the guidance directions are summed without accounting for potential interactions. A more nuanced approach considers joint conditioning, where some combinations of conditions are dropped together.

For two conditions (text and image), the compositional formulation includes three guidance terms: text only, image only, and joint text-plus-image. With appropriate weights, this captures synergistic effects that the additive formulation in (Multi CFG) misses.

CFG as Approximate Bayesian Inference

The score-function viewpoint in Remark 62 hints at a deeper probabilistic interpretation. We now formalise this.

Multi-Condition Extension

For multiple conditions, the Bayesian interpretation extends naturally under a conditional independence assumption.

This result provides principled guidance for choosing the weights $w_k$: each weight controls the “temperature” of the corresponding likelihood term. Setting $w_k=1$ recovers the standard posterior contribution from condition $c_k$; $w_k>1$ amplifies it; $w_k=0$ ignores it entirely.

Training with Conditional Dropout

For CFG to work, the model must be trained to handle missing conditions. This is achieved by randomly dropping conditions during training with specified probabilities.

Dynamic Guidance Schedules

A constant guidance scale throughout the denoising process is suboptimal. Early denoising steps (high noise) determine large-scale structure; late steps (low noise) refine fine details. Applying the same guidance weight at both stages can lead to over-saturation in the details or under-specification of the structure.

Both directions are in use, and which one is right is a property of the modality rather than a universal truth. Image samplers often front-load guidance, on the reasoning that the high-noise steps choose the semantic content and therefore need the strongest push towards the prompt. For video the argument frequently runs the other way: temporal coherence is settled during the early, high-noise steps, when the model commits to an overall motion trajectory and scene layout, and strong guidance at that point distorts the motion field by over-weighting per-frame conditional alignment at the expense of the inter-frame correlations that only the unconditional branch carries. On that reasoning the guidance is kept modest while the motion is being decided and raised once the layout is fixed and only per-frame detail remains.

Geometric Illustration of CFG

The following diagram illustrates the geometry of multi-condition CFG. The unconditional denoising direction, the text-conditional direction, and the image-conditional direction are vectors in the latent space. CFG forms a weighted combination of the differences between the conditional and unconditional directions.

Negative Prompts and Guidance Rejection

An extension of CFG replaces the unconditional prediction with a negative prompt prediction. Instead of extrapolating away from ${ϵ}_{\theta }({𝖹}_t,t,⌀)$, we extrapolate away from ${ϵ}_{\theta }({𝖹}_t,t,c_{\text{neg}})$, where $c_{\text{neg}}$ is a text description of undesired attributes (e.g., “blurry, low quality, jittery motion, deformed hands”).

(Negative Prompt)$$\begin{array}{ccc}& {\widehat{ϵ}}_{\theta }={ϵ}_{\theta }({𝖹}_t,t,c_{\text{neg}})+w({ϵ}_{\theta }({𝖹}_t,t,c)-{ϵ}_{\theta }({𝖹}_t,t,c_{\text{neg}})).& \text{<span class="tml-eqn"></span>}\end{array}$$

For video, negative prompts are especially useful for suppressing common artefacts such as temporal flickering, morphing faces, and physically implausible motion. The negative prompt effectively steers the denoising trajectory away from the manifold of low-quality videos, complementing the positive guidance that steers toward the desired content.

Image-to-Video Synthesis

One of the most practically useful modes of video generation is image-to-video (I2V) synthesis: given a single still image, generate a plausible video that “brings the image to life.” The input image provides a strong visual prior (appearance, layout, lighting, identity of objects), and the model must hallucinate coherent motion, camera dynamics, and temporal evolution consistent with both the image and an optional text description.

Image-to-video synthesis is central to creative workflows (animating photographs, concept art, and product images), visual effects (creating establishing shots from matte paintings), and scientific visualisation (simulating dynamics from initial conditions). It is also a stepping stone toward longer video generation, where autoregressive approaches generate each clip conditioned on the last frame of the previous clip.

In this section, we formalise the I2V problem, describe the conditioning mechanisms used by state-of-the-art systems, and analyse how information from the first frame propagates through the network.

Problem Formulation

The quality of an I2V model is measured along three axes: (i) visual fidelity of each generated frame, (ii) temporal coherence of the motion across frames, and (iii) identity preservation of objects, textures, and lighting established by the conditioning image. The third axis distinguishes I2V from text-to-video: the generated video must look like a natural continuation of the specific input image, not merely a plausible video matching the text description.

Conditioning Mechanisms

There are three principal mechanisms for injecting the first-frame information into the diffusion model: concatenation conditioning, cross-attention conditioning, and hybrid approaches that combine both.

Concatenation conditioning

The most direct approach concatenates the clean first-frame latent with the noised latents of the remaining frames along the channel dimension.

Either way, the extra $C^{\prime }$ channels widen the first convolutional layer of the denoiser, whose additional input weights are initialised to zero so that the pre-trained model is exactly recovered at the start of fine-tuning.

Stable Video Diffusion [8] uses the broadcast form of (I2V Concat), with one refinement: the conditioning latent is noise-augmented before concatenation, that is, a small amount of noise is added to ${𝒛}_0^1$ and the corresponding noise level is supplied to the network. The purpose is to close the train/test gap - at inference the conditioning frame is a real photograph rather than a frame drawn from the training distribution - and to give the user a dial that trades fidelity to the input image against the amount of motion the model is willing to invent. (SVD Concat)$$\begin{array}{ccc}& {\widetilde{𝖹}}_t=[{𝖹}_t;1_F\otimes {\widetilde{𝒛}}_0^1],{\widetilde{𝒛}}_0^1={𝒛}_0^1+{\sigma }_{\text{aug}}{𝝐}^{\prime }.& \text{<span class="tml-eqn"></span>}\end{array}$$

Cross-attention conditioning

An alternative to concatenation injects the first-frame information through cross-attention layers, analogous to how text embeddings are injected in text-to-image models. The first frame is encoded by a frozen image encoder (e.g., CLIP or DINOv2) to produce a sequence of image tokens ${𝑬}_{\text{img}}\in {\mathrm{ℝ}}^{N_{\text{img}}\times d}$, where $N_{\text{img}}$ is the number of spatial tokens. Each attention layer in the denoiser computes (I2V Crossattn)$$\begin{array}{ccc}& \text{CrossAttn}(𝑸,{𝑬}_{\text{img}})=\text{softmax}\left(\frac{𝑸{𝑾}_Q{({𝑬}_{\text{img}}{𝑾}_K)}^{𝖳}}{\sqrt{d}}\right){𝑬}_{\text{img}}{𝑾}_V,& \text{<span class="tml-eqn"></span>}\end{array}$$ where $𝑸$ are queries derived from the noised video latents.

Cross-attention conditioning is more parameter-efficient than concatenation (no additional input channels needed) and allows the model to selectively attend to different parts of the reference image at different frames and layers. However, it provides a more “semantic” representation of the first frame, which may lose fine-grained pixel-level details that concatenation preserves.

Hybrid conditioning

State-of-the-art I2V systems typically combine both mechanisms: concatenation for pixel-level detail and cross-attention for semantic guidance. This hybrid approach, used in systems such as Stable Video Diffusion [8] and DynamiCrafter [67], provides the denoiser with both low-level (exact pixel values of the reference frame) and high-level (semantic features extracted by a vision encoder) information about the conditioning image.

Information Propagation from the First Frame

A fundamental question in I2V synthesis is: how does information from the first frame reach distant frames? If the model processes only local temporal neighbourhoods, frame $F$ may receive little influence from frame $1$. The answer depends on the architectural choices of the denoiser.

The non-uniformity has a characteristic shape: frames closer to the conditioning image attend to it more strongly, and the attention mass falls away with temporal distance. A convenient toy model is $a_f\propto 1/f$, under which frame $2$ attends about $8\times$ more strongly to the anchor than frame $16$ does. We use this profile below purely as an illustrative functional form - it is not a measurement of any particular released model, and the decay exponent varies with architecture, clip length and training data. The qualitative point it captures is the one that matters practically: the anchor's influence is strongest at the start of the clip and weakest at the end, which is exactly where identity drift is observed.

State-of-the-Art I2V Systems

We briefly survey three influential I2V systems that illustrate different design choices.

Stable Video Diffusion (SVD).

Blattmann et al. [8] extend the Stable Diffusion image model to video by inserting temporal attention layers between the existing spatial attention layers. The model is trained in three stages: (1) image pre-training, (2) video pre-training on a large but noisy dataset, and (3) video fine-tuning on a smaller, high-quality dataset - a curation argument as much as an architectural one, and the paper's main empirical claim is that the third stage is what separates a usable model from an unusable one. SVD generates 14–25 frames at $576\times 1024$ resolution and serves as the backbone for many downstream I2V applications.

Its conditioning design is worth stating precisely, because it is often misdescribed. The released image-to-video models are purely image-conditioned. There are two pathways and neither of them carries text: the conditioning frame is concatenated channel-wise after noise augmentation ((SVD Concat)), and the CLIP text embedding that the underlying image model consumed through cross-attention is replaced by the CLIP image embedding of that same frame. A prompt has nowhere to enter, so SVD offers a single guidance weight rather than the text/image pair of (CFG TEXT IMG) - which is why the frame-axis ramp of Example 40 is where its guidance sophistication lives.

I2VGen-XL.

Zhang et al. [68] propose a cascaded I2V pipeline with two stages: a base model that generates low-resolution video from the input image, and a refinement model that upsamples to high resolution. The key insight is that the base model focuses on motion generation (getting the dynamics right), while the refinement model focuses on visual quality (sharpening textures and preserving identity). This decomposition allows each stage to be optimised independently.

DynamiCrafter.

Xing et al. [67] take a hybrid conditioning approach, combining concatenation with dual cross-attention to both CLIP image features and text features. A distinguishing feature is the use of a “visual context” module that extracts multi-scale features from the conditioning image and injects them at multiple layers of the denoiser. This provides richer structural information than single-scale cross-attention and helps preserve fine details across long temporal horizons.

I2V Pipeline Diagram

Motion and Camera Control

Text prompts and reference images specify what should appear in a video, but they provide limited control over how the scene moves. A filmmaker does not merely describe “a dog running in a park”; they specify the camera angle, the tracking shot direction, the speed of the zoom, and the trajectory of the subject. This level of control requires conditioning signals that go beyond language and into the domain of geometry and physics.

In this section, we introduce two families of spatial control for video diffusion: camera control, which specifies the viewpoint trajectory, and motion control, which specifies the movement of objects within the scene. We formalise the mathematical representations used for each (Plücker coordinates for cameras, dense flow fields and sparse trajectories for motion) and describe the architectural mechanisms that inject these signals into the diffusion model.

Camera Representations

A camera in 3D space is characterised by its pose: a rotation matrix $𝑹\in \text{SO}(3)$ and a translation vector $𝒕\in {\mathrm{ℝ}}^3$. Together, these define the rigid transformation from world coordinates to camera coordinates, (World TO CAM)$$\begin{array}{ccc}& {𝒙}_{\text{cam}}=𝑹{𝒙}_{\text{world}}+𝒕.& \text{<span class="tml-eqn"></span>}\end{array}$$ For video, we have a sequence of camera poses ${\{({𝑹}_f,{𝒕}_f)\}}_{f=1}^F$, one per frame, defining the camera trajectory.

The intrinsic parameters are captured by the camera matrix $𝑲\in {\mathrm{ℝ}}^{3\times 3}$: (Intrinsic Matrix)$$\begin{array}{ccc}& 𝑲=\left(\begin{array}{ccc}{f}_x& 0& c_x\\ 0& f_y& c_y\\ 0& 0& 1\end{array}\right),& \text{<span class="tml-eqn"></span>}\end{array}$$ where $(f_x,f_y)$ are the focal lengths in pixel units and $(c_x,c_y)$ is the principal point. For a pixel at coordinates $(u,v)$ in frame $f$, the corresponding 3D ray direction in world coordinates is (RAY Direction)$$\begin{array}{ccc}& {𝒅}_f(u,v)={𝑹}_f^{𝖳}{𝑲}^{-1}\left(\begin{array}{c}u\\ v\\ 1\end{array}\right).& \text{<span class="tml-eqn"></span>}\end{array}$$

While rotations and translations are sufficient to describe camera pose, they are not ideal as conditioning signals for neural networks. Rotations live on the non-Euclidean manifold $\text{SO}(3)$, and naively encoding them as $3\times 3$ matrices introduces redundant parameters and discontinuities. Plücker coordinates provide a more natural parameterisation for camera rays.

The Plücker ray map for an entire video is a tensor $𝖯\in {\mathrm{ℝ}}^{F\times H\times W\times 6}$, where the entry at $(f,h,w)$ contains the 6D Plücker coordinates of the ray from the $f$-th camera through pixel $(h,w)$. This tensor can be precomputed from the camera trajectory and intrinsics before generation begins.

Camera-Conditioned Diffusion

Camera Ray Diagram

Motion Control via Trajectories and Flow

Camera control specifies how the viewpoint moves; motion control specifies how objects within the scene move. There are two principal representations for motion conditioning: sparse trajectories (a small number of control points with specified 2D paths) and dense optical flow (a per-pixel displacement field).

Sparse trajectory conditioning

A sparse trajectory specifies the motion of $N$ control points across $F$ frames: (Sparse TRAJ)$$\begin{array}{ccc}& 𝒯={\{(u_n^f,v_n^f)\}}_{n=1,\dots ,N}^{f=1,\dots ,F},& \text{<span class="tml-eqn"></span>}\end{array}$$ where $(u_n^f,v_n^f)$ is the 2D position of control point $n$ in frame $f$. The user specifies these trajectories (e.g., by drawing arrows on the first frame), and the model generates video where the corresponding image regions follow the specified paths.

DragNUWA [35] encodes sparse trajectories as a sequence of Gaussian heatmaps centred at each control point, producing a conditioning tensor of shape $F\times H\times W\times N$. Each channel corresponds to one control point, with the heatmap peak indicating the desired position at each frame.

Dense flow conditioning

Dense flow conditioning provides a per-pixel motion specification. The conditioning signal is an optical flow field ${𝑭}^{f\to f+1}\in {\mathrm{ℝ}}^{H\times W\times 2}$ for each consecutive pair of frames, specifying the 2D displacement $(\mathrm{\Delta }u,\mathrm{\Delta }v)$ of each pixel from frame $f$ to frame $f+1$. This is the richest possible motion specification but requires the user (or an upstream motion planning module) to provide detailed per-pixel flow, which is rarely available in practice.

Video ControlNet

ControlNet [69] is the dominant architecture for injecting structured spatial control into diffusion models. Originally designed for images (conditioning on edges, depth maps, pose skeletons), it extends naturally to video by processing spatiotemporal control signals.

The zero initialisation is the key design principle of ControlNet. At the start of training, the zero-convolution layers output exactly $0$, so the combined features ${𝒉}_i^{\text{out}}={𝒉}_i^{\text{base}}$ and the model behaves identically to the pre-trained base model. As training progresses, the zero-convolution weights grow from zero, gradually introducing the control signal without disrupting the learned representations.

ControlNet Architecture Diagram

Types of Motion and Camera Control

We summarise the principal forms of spatial control for video diffusion and the systems that implement them.

Control Type	Representation	Injection	System
Camera trajectory	Plücker rays	Concatenation / ControlNet	CameraCtrl [33]
Camera + object motion	Pose + flow	Dual ControlNet	MotionCtrl [34]
Drag-based motion	Sparse trajectories	Heatmap encoding	DragNUWA [35]
Character animation	Pose skeleton	Reference attention	AnimateAnyone [36]
Motion patterns	Learned motion LoRA	Adapter injection	AnimateDiff [37]

MotionCtrl.

Wang et al. [34] propose a unified controller that handles both camera motion and object motion. Camera motion is encoded as per-frame pose embeddings added to the temporal attention layers. Object motion is encoded as sparse trajectory maps processed by a lightweight convolutional encoder. The two motion types are disentangled during training by alternating between camera-motion and object-motion supervision, allowing independent control at inference time.

AnimateAnyone.

Hu et al. [36] address the specific problem of character animation: given a single image of a person and a sequence of target body poses (represented as OpenPose skeletons), generate a video of that person performing the corresponding motion. The architecture uses a reference attention mechanism, where features from the reference image are injected into the temporal attention layers of the denoiser to preserve the identity and appearance of the subject.

Mathematical Properties of Plücker Conditioning

We conclude the discussion of camera control with a proposition relating Plücker conditioning to the underlying 3D geometry.

This proposition explains why Plücker coordinates are effective for camera conditioning: the network can learn to enforce multi-view consistency by learning functions of the reciprocal product, which directly encodes the geometric relationship between rays across frames.

Text-to-Video with Prompt Enhancement

The preceding sections have examined individual components of video generation systems: autoencoders, diffusion architectures, conditioning mechanisms, and control signals. In this section, we assemble these components into a complete text-to-video (T2V) pipeline and address a critical but often overlooked step: prompt enhancement, the process of transforming a user's terse text input into a detailed description that maximises the quality and specificity of the generated video.

The Full T2V Pipeline

A modern text-to-video system comprises five stages, illustrated in fig:vdiff:t2v-pipeline:

1. Prompt enhancement: An LLM rewrites the user's terse prompt into a detailed description specifying visual elements, motion dynamics, camera behaviour, and scene composition.

2. Text encoding: The enhanced prompt is encoded by one or more text encoders (T5 for semantic features, CLIP for vision-aligned features) to produce conditioning embeddings.

3. Latent diffusion: A video DiT generates latent video frames by iteratively denoising in the latent space of the video autoencoder, conditioned on the text embeddings.

4. Latent decoding: The video VAE decoder maps the denoised latent tensor to pixel-space video frames.

5. Post-processing: Optional upsampling, frame interpolation, and quality enhancement produce the final output.

Prompt Enhancement with LLMs

Users rarely provide the level of detail that a video diffusion model needs to produce high-quality output. A typical user prompt might be “a cat playing with yarn,” while the model performs best with detailed descriptions such as “A fluffy orange tabby cat sits on a sunlit hardwood floor, batting at a ball of red yarn with its right paw. The yarn unravels slowly as the cat pulls it. Warm afternoon light streams through a window on the left side of the frame. The camera is stationary, framing the scene at eye level with a shallow depth of field.”

Dual Text Encoding

Many state-of-the-art T2V models use more than one text encoder, each capturing a different aspect of the text. The number is a design choice rather than a settled question: production systems ship with one, two and three encoders, and we look at an example of each below.

The rationale for dual encoding is that the two encoders provide complementary information:

• T5 [38] is a large encoder–decoder language model trained on text-only data; conditioning uses the encoder tower only. That encoder is bidirectional: every token attends to every other, in both directions, so each output embedding is contextualised by the whole prompt rather than only by what precedes it. This is precisely why it is a good conditioning encoder - it excels at fine-grained semantic meaning, compositional structure, and long-range dependencies. T5 embeddings are rich in linguistic information: word order, syntactic relationships, and semantic nuances.

• CLIP [39] is a vision-language model trained on image-text pairs. Its text encoder produces embeddings that are aligned with visual features, making them particularly useful for specifying visual appearance. CLIP embeddings are rich in visual information: colour, texture, spatial layout, and stylistic attributes.

T2V Pipeline Diagram

System-Level Considerations

We briefly discuss design choices made by three influential T2V systems.

Sora.

OpenAI's Sora [6] frames video generation as “world simulation,” emphasising that the model should learn physical laws, object permanence, and 3D consistency from data alone, without explicit physics engines. Sora uses a spacetime patches approach, processing the video as a sequence of 3D patches that are fed to a DiT. The prompt enhancement step (using GPT-4) is a critical component, transforming user queries into detailed scene descriptions that guide the model's internal world simulation.

CogVideoX.

CogVideoX [17] introduces an “expert transformer” architecture. The name suggests two networks, but the mechanism is subtler and cheaper to describe than that: text tokens and video tokens live in one sequence and are processed by one full 3D attention, and what is duplicated per modality is only the adaptive layer-normalisation and the associated projections - separate scale and shift parameters for text tokens and for video tokens, both modulated by the diffusion timestep. The motivation is a numerical one: text embeddings from a frozen T5 and video latents from a 3D VAE have very different scales and distributions, and forcing them through shared normalisation statistics degrades the fusion.

It is worth being explicit that this design does not reduce attention cost. Attention runs over the concatenated sequence, so it remains quadratic in the combined text-plus-video length; the expert parameters buy alignment quality, not efficiency. The efficiency argument in CogVideoX is made elsewhere - in the 3D VAE's temporal compression and in progressive training that starts with short, low-resolution clips and grows duration and resolution over the schedule.

Movie Gen.

Movie Gen [32] is designed as a “cast of foundation models,” where different components (video generation, audio generation, editing) are implemented as separate models that share a common architecture and training paradigm. The video generation component uses a 30B-parameter transformer trained on a mixture of image and video data, with the data mixing ratio carefully tuned to balance per-frame quality against temporal coherence.

Personalisation and Adaptation

A pre-trained video diffusion model captures the distribution of generic videos seen during training. For many applications, however, we want to generate videos featuring specific subjects (a particular person, pet, product, or scene), specific visual styles (anime, watercolour, cinematic noir), or specific motion patterns (a characteristic dance move, a recurring camera technique). Personalisation adapts a pre-trained model to these specific requirements using a small amount of reference data, typically a few images or short video clips.

The central challenge is efficiency: a state-of-the-art video diffusion model may have 5–30 billion parameters. Full fine-tuning on a few reference examples is (i) computationally expensive, (ii) prone to catastrophic forgetting of the model's general capabilities, and (iii) prone to overfitting to the small reference set. Low-rank adaptation (LoRA) addresses all three issues by restricting the parameter update to a low-dimensional subspace.

Low-Rank Adaptation (LoRA)

The key insight is that the weight update $\mathrm{\Delta }𝑾=𝑩𝑨$ is constrained to the set of rank-$r$ matrices. If the task-specific adaptation requires changes that are approximately low-rank (which empirical evidence strongly suggests), LoRA can capture these changes with far fewer parameters than full fine-tuning.

Approximation Quality of LoRA

How well can a rank-$r$ update approximate the optimal full-rank adaptation? The following proposition provides a bound.

LoRA for Video DiT

Applying LoRA to a video diffusion transformer requires choosing which layers to adapt and with what rank. The design space includes:

1. Spatial attention: Adapting the spatial self-attention projections ($𝑸$, $𝑲$, $𝑽$, output) changes the model's visual features: how it represents textures, shapes, and object appearances. This is the most important target for subject personalisation (learning a specific face or object).

2. Temporal attention: Adapting the temporal self-attention projections changes the model's motion dynamics: how objects move, how scenes evolve, and what temporal patterns are generated. This is the primary target for motion personalisation (learning a specific dance move or camera technique).

3. Cross-attention: Adapting the cross-attention projections (where text embeddings interact with visual features) changes the model's text-visual alignment: how it maps words to visual concepts. This is useful for learning new vocabulary (associating a trigger word with a specific subject or style).

4. Feed-forward layers: Adapting the MLP layers changes the model's non-linear feature transformations. This is less commonly targeted by LoRA in video diffusion but can be useful for style adaptation.

DreamBooth for Video

DreamBooth [70] fine-tunes a diffusion model on a small set of images (3–5) of a specific subject, associating it with a unique identifier token (e.g., “a photo of [V] dog”). The key insight is the prior preservation loss, which ensures that the model retains its general capabilities while learning the new subject.

The prior preservation term prevents the model from “forgetting” the general concept of “dog” while it learns the specific appearance of the subject. Without it, the model would collapse: all dogs would look like the specific subject, and the trigger token “[V]” would become synonymous with the class label.

Composing Adapters

A powerful feature of LoRA-based personalisation is the ability to compose multiple adapters. If adapter 1 contributes $\mathrm{\Delta }{𝑾}_1={𝑩}_1{𝑨}_1$ and adapter 2 contributes $\mathrm{\Delta }{𝑾}_2={𝑩}_2{𝑨}_2$, the composed model uses (LORA Compose)$$\begin{array}{ccc}& {𝑾}^{\prime }=𝑾+{\alpha }_1{𝑩}_1{𝑨}_1+{\alpha }_2{𝑩}_2{𝑨}_2,& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${\alpha }_1,{\alpha }_2\in [0,1]$ are blending weights. This allows, for example, combining a subject adapter (to generate a specific person) with a style adapter (to render in a specific artistic style) and a motion adapter (to specify a particular motion pattern).

Quantised LoRA (QLoRA)

For very large video diffusion models, even loading the pre-trained weights into GPU memory can be challenging. QLoRA [71] addresses this by quantising the frozen base weights to 4-bit precision while keeping the LoRA adapter weights in full precision (16-bit or 32-bit).

Practical Considerations

We conclude with practical guidelines for personalising video diffusion models.

1. Rank selection: The rank $r$ controls the trade-off between expressiveness and efficiency. For subject personalisation, $r=8$–$32$ is typical; for style transfer, $r=4$–$16$ often suffices; for complex motion patterns, $r=16$–$64$ may be needed. A useful diagnostic is to plot the singular value spectrum of the learned $\mathrm{\Delta }𝑾$ after training: if the smallest singular values are near zero, the rank is too high (wasting parameters); if they are large, the rank may be too low.

2. Learning rate: LoRA fine-tuning typically uses learning rates $10\times$ to $100\times$ higher than full fine-tuning because the gradient signal must be “concentrated” into a much smaller parameter subspace. Common values are $1\times {10}^{-4}$ to $5\times {10}^{-4}$ for LoRA versus $1\times {10}^{-6}$ to $5\times {10}^{-6}$ for full fine-tuning.

3. Training duration: Overfitting is the primary risk when personalising on a small dataset. For subject personalisation with 3–5 reference images, 500–2000 training steps are typical. For motion personalisation from short video clips, 1000–5000 steps are common. Early stopping based on a held-out validation metric (e.g., CLIP similarity to the reference) is recommended.

4. Regularisation: Beyond the DreamBooth prior preservation loss, additional regularisation strategies include: itemize

5. Weight decay on the LoRA parameters.

6. Gradient clipping to prevent sudden large updates.

7. Augmenting the reference data with random crops, colour jitter, and temporal shifts. itemize

8. Which layers to adapt: Not all layers contribute equally to personalisation. Attention projections in the middle layers of the transformer are most important for both subject and style adaptation; early and late layers contribute less. Adapting only the middle $50\%$ of layers can reduce the adapter size by half with minimal quality loss.

Summary and Forward Pointers

This section has covered the core techniques for adapting pre-trained video diffusion models to specific subjects, styles, and motion patterns. LoRA provides a parameter-efficient mechanism for adaptation, with theoretical guarantees on approximation quality (Proposition 37) and practical recipes for rank selection, learning rate tuning, and regularisation. DreamBooth extends the paradigm to subject-driven generation with prior preservation. Adapter composition enables combining multiple adaptations without retraining.

Everything in the last five sections has taken the model's native generation window for granted: a clip of a few seconds, generated in one shot, and then controlled. The next chapter of the story drops that assumption. Long Video Generation confronts the fact that these models generate two to ten seconds while users want minutes, and works through the two ways out - autoregressive chunking, which accumulates drift, and hierarchical keyframe generation, which trades that drift for a planning problem. Temporal Consistency and Coherence then asks what temporal consistency even means quantitatively, and what an attention window of finite size can and cannot guarantee about it. Video Editing turns from generation to editing an existing video, where the control problem reappears in a sharper form: the output must change in one respect and stay identical in every other. Physics-Aware Video Generation and World Models closes with the question the whole field is circling - whether a model that predicts video frames has thereby learned anything about the world that produced them - and Autoregressive-Diffusion Hybrids with the autoregressive and diffusion paradigms converging on each other.

The connection to the present section is direct. Personalisation and control are how you steer a single short clip; long-horizon generation is where those same steering signals must be kept consistent across many clips, and consistency is precisely what degrades first.

Long Video Generation

The video diffusion models we have studied so far generate clips of fixed, modest duration: typically 2 to 10 seconds, corresponding to 16 to 240 frames depending on the frame rate. This window is dictated by the model's temporal attention span and by the memory required to process the full latent tensor during training. Yet the videos people actually want to create, a product demonstration, a short film, a gameplay walkthrough, span minutes or hours. Bridging the gap between a model's native generation window and the durations demanded by real applications is one of the central open problems in video generation.

The difficulty is not merely computational. Even if we had unlimited memory, training a single model on very long sequences would require prohibitively large datasets of consistently annotated long videos, and the quadratic cost of attention over hundreds or thousands of frames would dominate the compute budget. Instead, the field has developed a family of strategies that compose short-window models to produce long outputs while maintaining temporal coherence across chunk boundaries. We examine the two principal paradigms: autoregressive chunk generation and hierarchical keyframe generation.

The fundamental tension

Let us formalise the problem. Suppose our diffusion model can generate video chunks of $F$ frames each. We wish to produce a video of $N\gg F$ frames. The model's learned distribution $p_{\theta }({𝖷}^{1:F})$ captures statistics of $F$-frame sequences, but says nothing directly about the joint distribution $p({𝖷}^{1:N})$ over the full video. Any long-generation strategy must therefore make assumptions about how to factorise or approximate $p({𝖷}^{1:N})$ using the model's native capacity.

Two broad failure modes characterise long video generation. The first is drift: the video's content gradually shifts away from its initial state, so that characters change appearance, scenes morph imperceptibly, and the visual style wanders. The second is repetition: the model falls into a low-energy attractor and produces the same motion pattern in a loop, typically a subtle rocking or breathing motion with no narrative progression. Good long-generation methods must combat both simultaneously.

Autoregressive chunk generation

The most natural approach to long video generation is to produce the video one chunk at a time, conditioning each new chunk on the tail of the previous one. This mirrors the autoregressive structure of language models, where each token is generated conditioned on all preceding tokens.

In practice, the conditioning is implemented by one of several mechanisms.

1. Latent initialisation. Encode the conditioning frames ${𝖷}_{F-\delta +1:F}^{k-1}$ into the latent space and concatenate or replace the corresponding positions in the noisy latent tensor before denoising begins. The denoising process then “paints in” only the new frames while respecting the boundary.

2. Cross-attention conditioning. Feed the conditioning frames (or their features) as additional keys and values to the temporal cross-attention layers. This gives the model soft access to the context without hard-copying any frames.

3. Noise blending. During the reverse diffusion process, maintain a “clean” copy of the overlap region and blend it with the denoised prediction at each step, using a schedule that transitions smoothly from noisy to clean. This avoids hard boundaries.

The critical question is: how do errors behave as we chain more chunks? The question has to be posed carefully, because the obvious formulation is vacuous. The deviation of the generated video from a reference video is a concatenation of the per-chunk deviations along the time axis, not a sum of vectors: if every chunk satisfies $\Vert {𝖷}^k-{𝖷}^{\ast ,k}\Vert \le ϵ$ then (Error Concatenation)$$\begin{array}{ccc}& {\Vert {𝖷}^{1:K}-{𝖷}^{\ast ,1:K}\Vert }^2=\sum _{k=1}^K{\Vert {𝖷}^k-{𝖷}^{\ast ,k}\Vert }^2\le K{ϵ}^2& \text{<span class="tml-eqn"></span>}\end{array}$$ holds deterministically, with no independence assumption, no zero-mean assumption and no cross terms to cancel. Worse, this bound is insensitive to the very thing we care about: it is the same whether the chunks were generated independently or chained, because it never models the fact that chunk $k$ is conditioned on chunk $k-1$'s already-corrupted output.

What actually degrades over a long autoregressive rollout is not the per-frame reconstruction error but the cumulative displacement of the video's slowly varying content: a character's face, the colour palette, the lighting direction, the camera's implicit position. Each chunk inherits the previous chunk's version of that content and adds its own small increment, and the increments never get undone. The object to bound is therefore the running sum of those increments, and the following statement makes that model explicit.

Hierarchical keyframe generation

Autoregressive methods process the video strictly left-to-right, which means that early decisions irreversibly constrain later content. An alternative paradigm first generates a sparse set of keyframes that establish the global structure, then fills in the gaps through interpolation. This “coarse-to-fine” strategy mirrors how human animators work: first sketch the key poses, then draw the in-between frames.

The hierarchical approach offers several advantages over purely autoregressive generation.

1. Global coherence. Because the keyframes are generated jointly (or with global conditioning), they establish a consistent narrative arc that the interpolation stages must respect. This reduces drift.

2. Parallelism. The interpolation between two keyframes is independent of the interpolation between two other keyframes. This enables embarrassingly parallel generation at each level, dramatically reducing wall-clock time.

3. Error isolation. An error in one interpolation segment does not propagate to distant segments, because the keyframes anchor both endpoints. This limits error accumulation to within-segment drift.

The hierarchical approach can be analysed as a tree-structured computation, where the root generates global structure and the leaves produce individual frames. This tree structure enables a clean complexity analysis.

Temporal noise schedules for long generation

A subtle but important consideration in autoregressive chunk generation is the design of the noise schedule at chunk boundaries. When the model begins denoising a new chunk, the overlap region contains “clean” frames from the previous chunk, while the remaining positions are pure noise. This mismatch between clean and noisy regions can cause artifacts.

The noise ramp ensures a smooth transition from the clean context to the fully noisy region, avoiding the hard boundary that would otherwise cause visible seams.

Combining strategies: the hybrid approach

The autoregressive and hierarchical approaches are not mutually exclusive. In fact, the most effective long video generation systems combine both.

This hybrid approach inherits the global coherence of hierarchical generation (through planned keyframes) and the local quality of autoregressive generation (through segment-level diffusion).

The long-range consistency challenges of video generation connect deeply to the memory architectures studied in 24. The fundamental question of how to maintain coherent state over extended sequences arises in both language modelling and video generation, and solutions from one domain often inspire progress in the other. Similarly, the problem of generating consistent content over long durations relates to the continual learning challenges discussed in 26, where models must maintain stable representations while adapting to new inputs.

Temporal Consistency and Coherence

A beautiful video that flickers, wobbles, or allows characters to subtly morph between frames is not a good video. Temporal consistency, the property that corresponding regions across frames depict the same content in a physically plausible manner, is arguably the single most important quality criterion for generated video. Human viewers are extraordinarily sensitive to temporal inconsistencies: a shadow that shifts by a few pixels, a texture that shimmers, or an expression that momentarily changes can render an otherwise photorealistic video immediately unconvincing.

In this section, we formalise what temporal consistency means, present quantitative metrics for measuring it, and study how architectural choices in diffusion models affect the degree of consistency achievable.

Measuring temporal consistency

Intuitively, temporal consistency means that if we warp frame $f+1$ back to the coordinate system of frame $f$ using the true optical flow, the result should match frame $f$ closely. This leads to the following formal definition.

While TC measures frame-to-frame smoothness, many applications require a stronger form of consistency: that specific entities maintain their identity throughout the video.

Attention window and consistency guarantees

A video diffusion model's temporal attention mechanism is the primary tool by which it enforces consistency across frames. If the model uses full temporal attention (every frame attends to every other frame), then the model can, in principle, detect and correct inconsistencies between any pair of frames. In practice, memory constraints often force the use of windowed or sparse attention, where each frame attends only to a local neighbourhood. Beyond the window, consistency between two frames is not enforced directly: it has to be relayed through a chain of intermediate frames, and the relay is lossy. The following result quantifies how lossy, under assumptions that must be stated explicitly because they, and not the attention mechanism, are what does the work.

Methods for improving temporal consistency

Given the fundamental limitations imposed by finite attention windows, researchers have developed several techniques to improve temporal consistency beyond what the base model provides.

Structural consistency via shared noise

An elegant family of consistency-improving techniques operates not on the model architecture but on the noise used to initialise the denoising process. The key insight is that if neighbouring frames start from correlated noise, the denoised outputs will naturally exhibit temporal correlation, even if the model processes each frame independently.

The parameter $\rho$ controls a trade-off between temporal consistency and per-frame diversity. At $\rho =0$, frames are fully independent (maximum diversity, minimum consistency). At $\rho =1$, all frames start from identical noise (maximum consistency, but identical content). In practice, $\rho \in [0.3,0.7]$ provides a good balance - corresponding to a shared variance fraction ${\rho }^2$ of only $0.09$ to $0.49$, which is worth keeping in mind when comparing $\rho$ values across papers that parameterise the mixture differently.

Video Editing

Generating video from scratch is impressive, but many practical applications require a different capability: editing an existing video while preserving its overall structure. Change the season from summer to winter, replace a character's outfit, alter the lighting from day to night, or modify the text on a sign, all while keeping the camera motion, character poses, and scene layout intact. This is the domain of video editing, which has emerged as one of the most practically valuable applications of video diffusion models.

The challenge is formidable. A good video edit must simultaneously: (i) faithfully execute the desired modification, (ii) preserve all unedited aspects of the source video, and (iii) maintain temporal consistency across the edit. Getting any one of these wrong produces artifacts: an edit that ignores the prompt, structural distortions in unedited regions, or flickering that reveals the frame-by-frame nature of the processing.

In this section, we study two foundational approaches to diffusion-based video editing: SDEdit (adding noise and denoising with a new prompt) and attention injection (sharing structural information between source and edited branches).

Video SDEdit

SDEdit [72] is one of the simplest and most elegant approaches to image editing with diffusion models, and its extension to video is natural. The core idea is to add noise to the source content up to some intermediate diffusion timestep $t^{\ast }$, then denoise from that point using the target prompt. The noise level controls the trade-off between faithfulness to the source and freedom to incorporate the edit.

The main limitation of Video SDEdit is that it processes the entire video uniformly: every frame receives the same noise level and the same editing prompt. This makes it difficult to perform localised edits (e.g., changing only one object) or edits that should affect different frames differently.

Attention injection for structure-preserving edits

A more sophisticated approach to video editing exploits the internal representations of the diffusion model, specifically the self-attention maps, to transfer structural information from the source video to the edited version. The key observation is that the self-attention keys and values in a diffusion U-Net or transformer encode the spatial layout and temporal structure of the content, while the queries drive the generation of new content.

Advanced video editing methods

The basic SDEdit and attention injection techniques have been refined and extended in numerous ways. We highlight several influential approaches.

Method	TF	Temporal	Local Edit	Quality
Video SDEdit	Yes	No	No	Medium
Attention Injection	Yes	Yes	No	High
TokenFlow	Yes	Yes	No	High
Pix2Video	Yes	Yes	No	Medium
FateZero	Yes	Yes	Yes	High
Rerender-A-Video	Yes	Yes	No	High

Physics-Aware Video Generation and World Models

The videos we have discussed generating so far are evaluated primarily on visual quality and temporal consistency. A video of a ball bouncing might look beautiful, but does the ball actually follow a parabolic trajectory? Does it accelerate downward at $9.8{\text{m/s}}^2$? Does it conserve momentum when it bounces? The frontier of video generation research asks a deeper question: can diffusion models learn not just the appearance of the physical world, but its dynamics?

This question connects video generation to the broader concept of world models: internal representations of how the world works that can be used for prediction, planning, and reasoning. If a video diffusion model can accurately predict what happens next in a physical scene, it is, in a meaningful sense, simulating physics. This section explores the theoretical foundations of this connection and surveys recent efforts to build physics-aware video generators.

World models: formalisation

The concept of a world model has a long history in cognitive science, control theory, and reinforcement learning. For our purposes, a world model is a conditional generative model that predicts future observations given past observations and (optionally) actions.

The world model framework is general enough to encompass several important special cases.

1. Video prediction models ($a_t=⌀$, $H=1$): predict the next frame given the past. These are the classical next-frame prediction models from the video prediction literature.

2. Interactive world models ($a_t\ne ⌀$): predict the visual consequence of taking action $a_t$ in the current state. These are directly relevant to robotics and game simulation.

3. Long-horizon world models ($H\gg 1$): predict extended future trajectories. These are the most ambitious and the most relevant to planning.

Physics-guided diffusion

One approach to incorporating physical knowledge into video generation is to use physics-guided diffusion: modifying the denoising process to steer generated videos toward physical plausibility. This is analogous to classifier guidance (Classifier-Free Guidance for Video), but instead of guiding with a classifier, we guide with a physics constraint.

The physics loss $𝒫$ can encode various physical constraints:

1. Conservation laws. Require that total energy, momentum, or mass is approximately conserved across frames: ${𝒫}_{\text{conserve}}={\sum }_f\Vert E(f+1)-E(f){\Vert }^2$, where $E(f)$ is an energy estimator applied to frame $f$.

2. Trajectory constraints. Require that tracked object positions follow smooth, physically plausible trajectories: ${𝒫}_{\text{traj}}={\sum }_f\Vert {\ddot{𝒙}}_{\text{obj}}^f-𝐠{\Vert }^2$, where ${\ddot{𝒙}}_{\text{obj}}^f$ is the estimated acceleration and $𝐠$ is gravitational acceleration.

3. Fluid dynamics. Require that observed flows satisfy the Navier-Stokes equations: ${𝒫}_{\text{fluid}}=\Vert {\partial }_t𝐯+(𝐯\cdot \nabla )𝐯-\nu {\nabla }^2𝐯+\nabla p{\Vert }^2$, where $𝐯$ is the estimated velocity field and $\nu ,p$ are viscosity and pressure.

4. Rigid body constraints. Require that rigid objects maintain their shape: ${𝒫}_{\text{rigid}}={\sum }_f\Vert d({𝒙}_i^f,{𝒙}_j^f)-d({𝒙}_i^1,{𝒙}_j^1){\Vert }^2$, where $d$ denotes pairwise distances between tracked points on the object.

Video diffusion as an implicit world model

A provocative hypothesis, articulated most prominently in the context of OpenAI's Sora [6], is that video diffusion models trained on sufficiently large and diverse video datasets may implicitly learn a world model. That is, without ever being explicitly trained on physics equations, they may learn to simulate the consequences of physical interactions from the statistical regularities present in the training data.

However, several important caveats temper this optimistic view.

Frontier systems: Sora and beyond

The connection between video generation and world modelling was thrust into the spotlight by OpenAI's Sora [6], which was described as a “world simulator.” Let us examine what this claim means and what evidence supports or qualifies it.

The connection between world models and agentic AI is deep and actively explored. We refer the reader to 27 for a thorough treatment of how world models can be used for planning and decision-making in agentic systems, and to 23 for the relationship between world modelling and recursive reasoning.

The gap between simulation and understanding

There is an important philosophical and practical distinction between a model that simulates physics and one that understands physics. A video diffusion model that produces realistic-looking bouncing balls is performing high-fidelity pattern matching against its training distribution. Whether this constitutes “understanding” of Newtonian mechanics is a matter of definition and debate.

We can formalise this distinction using the concept of generalisation depth: how far beyond the training distribution a model's physical predictions remain accurate.

A model with high generalisation depth has learned something closer to the underlying physical law rather than merely memorising surface statistics. Current video diffusion models have low generalisation depth: they produce plausible-looking physics for scenarios similar to their training data, but fail rapidly when physical parameters are perturbed beyond the training range.

Category	Example	Current	Difficulty
		Accuracy
Gravity	Ball falling	High	Easy
Inertia	Sliding object	High	Easy
Collision	Billiard balls	Medium	Medium
Fluid dynamics	Water pouring	Medium	Medium
Deformation	Cloth folding	Medium	Medium
Multi-body	Newton's cradle	Low	Hard
Conservation	Energy in bounce	Low	Hard
Counting	Fixed # of objects	Low	Hard
Novel physics	Zero gravity	Very low	Very hard

Autoregressive-Diffusion Hybrids

Throughout this chapter, we have treated diffusion as the primary generative mechanism for video. But diffusion is not the only game in town. Autoregressive models, which generate data one token at a time conditioned on all previous tokens, have been spectacularly successful in language modelling and have shown promise for visual generation through vector-quantised tokenisation. A natural question arises: can we combine the strengths of both paradigms?

The basic intuition behind autoregressive-diffusion hybrids is straightforward: autoregressive models excel at capturing long-range dependencies and sequential structure (they “know what to generate”), while diffusion models excel at producing high-quality continuous outputs (they “know how to generate it”). Combining them should yield models that can plan over long temporal horizons while producing visually strong results.

A caution before we proceed. This is a design space, not yet a settled family of systems, and the literature is looser with the word “hybrid” than it should be. Several of the systems routinely cited under this heading are not hybrids at all: they are pure autoregressive token models whose detokeniser happens to be a strong learned decoder. We will therefore describe the two architectural patterns first, and then place named systems on the resulting spectrum honestly - including those that turn out to sit at its endpoints rather than in the middle.

Frame-level AR with per-frame diffusion

The simplest hybrid architecture generates the video one frame at a time, using an autoregressive backbone to model the temporal sequence and a diffusion model to generate each frame's content.

The conditioning on ${𝒙}^{<f}$ can be implemented in several ways:

1. Direct concatenation. Concatenate the previous frames (or their latent encodings) to the noisy input of the diffusion model along the channel or temporal dimension. This is simple but scales poorly with context length.

2. Cross-attention conditioning. Encode the previous frames into a sequence of feature vectors and inject them via cross-attention in the denoising network. This scales better but requires careful attention to the memory cost.

3. Recurrent state conditioning. Process the previous frames through a recurrent network (LSTM, GRU, or state-space model) to produce a fixed-size hidden state, and condition the diffusion model on this state. This has constant memory cost regardless of context length but compresses the history lossy.

The frame-level hybrid inherits both the strengths and weaknesses of autoregressive generation. On the positive side, it can generate videos of arbitrary length (each frame is generated independently conditioned on the past), and the diffusion model ensures high per-frame quality. On the negative side, it suffers from the same error accumulation as autoregressive chunk generation (Proposition 39), and generation is inherently sequential (each frame must wait for all previous frames), eliminating the parallel sampling advantage of full-video diffusion.

Temporal AR, spatial diffusion

A more sophisticated hybrid decouples the temporal and spatial aspects of generation: an autoregressive model handles the temporal sequence of compact representations, and a diffusion model handles the spatial rendering of each representation into a full frame.

This factorisation has a clean interpretation: the autoregressive model serves as a “director,” planning what happens at each point in the video, while the diffusion model serves as a “renderer,” producing the high-quality visual output. The codes $c_f$ capture semantic content (object positions, actions, scene changes) while abstracting away pixel-level details (textures, lighting, exact colours) that the diffusion model handles.

Named systems on the AR–diffusion spectrum

The systems usually gathered under the “hybrid” heading do not all contain both mechanisms. It is worth being precise about which do, because the distinction is exactly the one this section is about.

Theoretical comparison of paradigms

To understand why hybrids are attractive, consider the complementary strengths and weaknesses of the pure paradigms.

Property	Pure AR	Pure Diffusion	Hybrid
Long-range planning	Strong	Weak	Strong
Per-frame quality	Medium	Strong	Strong
Generation speed	Slow	Medium	Medium
Variable length	Native	Requires tricks	Native
Multi-modal integration	Native	Requires adapters	Native
Training data needs	Tokenised video	Continuous video	Both
Drift over a rollout	Linear in steps	None within window	Intermediate

Architectural patterns for hybrids

Several architectural patterns have emerged for building hybrid systems.

Pattern 1: Token-then-render. Generate discrete tokens autoregressively, then render continuous outputs from those tokens. VideoPoet and MAGVIT-v2 + LM take this route with a feed-forward VQ decoder as the renderer, which is what makes them autoregressive rather than hybrid; the pattern becomes a hybrid only when the renderer is itself a conditional diffusion model. The advantage either way is a clean separation of concerns; the disadvantage is that quantisation introduces an information bottleneck (Proposition 44).

Pattern 2: Plan-then-diffuse. Generate a coarse plan (keyframes, layout, motion trajectory) autoregressively, then generate the full video conditioned on the plan using diffusion. This is related to the hierarchical keyframe generation of Hierarchical keyframe generation and to Emu Video's two-stage approach. The advantage is that the AR component operates in a much lower-dimensional space; the disadvantage is that the plan must capture enough information to guide the diffusion model.

Pattern 3: Interleaved AR-diffusion. Alternate between autoregressive steps (generating a new token or code) and diffusion steps (refining the current output). This is the most tightly coupled pattern and the hardest to train, but it allows the AR and diffusion components to inform each other throughout the generation process.

Everything so far has been about building video generators. The remainder of the chapter asks how we know whether any of it worked. We turn next to evaluation - the Fréchet Video Distance and its reading as a Wasserstein distance, the decomposed sixteen-dimensional VBench protocol, the temporal-consistency metrics of Temporal Consistency Metrics, and human evaluation protocols - and then to specialised applications that reuse the same backbone with different conditioning: human motion generation, audio-visual generation, talking-head synthesis and virtual try-on. We close by wiring the chapter into the rest of the book, from variational inference and optimal transport to memory architectures and agentic world models, and by stating five open problems that define the current frontier: real-time generation, minute-scale consistency, physical grounding, compositional generation, and unified multimodal generation.

Evaluation Metrics for Video Generation

Generating videos is only half the challenge; the other half is measuring whether the generated videos are any good. Unlike image generation, where the Fréchet Inception Distance (FID) [73] has become a near-universal yardstick, the evaluation of video generation models must contend with an additional temporal dimension that fundamentally complicates quality assessment. A video can have excellent per-frame quality yet exhibit temporal flickering; it can be temporally smooth yet lack diversity; it can match the data distribution in aggregate statistics yet fail to produce physically plausible motion.

In this section, we formalise the principal evaluation metrics used in the field, analyse their mathematical properties, and discuss their strengths and limitations. We begin with the Fréchet Video Distance (FVD), the most widely used automatic metric for video generation, and then turn to the more recent decomposed evaluation frameworks that aim to capture the multifaceted nature of video quality.

Fréchet Video Distance

The Fréchet Video Distance extends the FID from images to video by replacing the Inception-v3 feature extractor with a video-aware network, typically an Inflated 3D ConvNet (I3D) [74]. The I3D network processes short video clips and produces a feature vector that captures both spatial appearance and temporal dynamics.

The FVD inherits the mathematical structure of the FID and admits a clean interpretation in terms of optimal transport.

FVD Computation Pipeline

fig:vdiff:fvd-pipeline illustrates the end-to-end computation of FVD. The pipeline has four stages: (i) sample $N$ real and $M$ generated clips, (ii) extract I3D features from each clip, (iii) compute the sample mean and covariance of each feature set, and (iv) evaluate the closed-form FVD expression.

Limitations of FVD

Despite its widespread use, FVD has several well-known limitations.

Additional limitations include the following:

1. Gaussian assumption. FVD assumes that the feature distributions are well-approximated by Gaussians. If the true feature distribution is multimodal (as it typically is for diverse video datasets), the Gaussian fit discards important structural information.

2. Feature extractor bias. The I3D network was trained on the Kinetics dataset, which is dominated by human actions. FVD may therefore be more sensitive to motion quality in human-centric videos than in scenes involving natural phenomena, animals, or abstract content.

3. No per-sample quality. FVD is a distributional metric: it compares two sets of videos, not individual samples. It cannot tell you whether a specific generated video is good or bad.

4. Temporal resolution. The 16-frame input window of I3D limits FVD's ability to detect long-range temporal inconsistencies that manifest over seconds rather than fractions of a second.

Decomposed Evaluation: VBench

Recognising the limitations of single-number metrics, Huang et al. [57] introduced VBench, a comprehensive evaluation framework that decomposes video quality into 16 interpretable dimensions. The key insight is that “video quality” is not a monolithic concept but a composite of many independent axes, each of which can be measured separately and improved independently.

The decomposed nature of VBench allows researchers to diagnose specific weaknesses in their models. A model that scores poorly on “temporal flickering” but well on “imaging quality” clearly has a temporal consistency problem, not a per-frame quality problem. This diagnostic power is impossible with FVD alone.

Temporal Consistency Metrics

Beyond FVD and VBench, a family of lightweight metrics specifically targets temporal consistency, the quality that distinguishes a coherent video from a slideshow of pretty pictures.

In addition to CLIP-based metrics, several other temporal quality measures are commonly used:

1. Warping error: Given optical flow ${𝒗}^f$ between frames $f$ and $f+1$, the warping error measures how well frame $f+1$ can be reconstructed by warping frame $f$: (WARP Error)$$\begin{array}{ccc}& E_{\text{warp}}=\frac{1}{F-1}\sum _{f=1}^{F-1}\frac{1}{HW}\sum _{i,j}\Vert {𝒙}_{i,j}^{f+1}-\mathrm{warp}{({𝒙}^f,{𝒗}^f)}_{i,j}{\Vert }^2,& \text{<span class="tml-eqn"></span>}\end{array}$$ where $\mathrm{warp}({𝒙}^f,{𝒗}^f)$ warps frame ${𝒙}^f$ using flow ${𝒗}^f$ via bilinear interpolation (the same warping operator used in Definition 74; we avoid the symbol $𝒲$, which Definition 79 reserves for the world model).

2. Subject identity preservation : A face recognition model (e.g., ArcFace) extracts embeddings from detected faces across frames, and the cosine similarity between consecutive embeddings is averaged. This is particularly relevant for talking-head and character animation applications.

3. Motion magnitude score : The average optical flow magnitude $\bar{m}=\frac{1}{F-1}{\sum }_f\frac{1}{HW}{\sum }_{i,j}\Vert {𝒗}_{i,j}^f\Vert$ measures the overall dynamic degree of the video. A video with $\bar{m}\approx 0$ is essentially a static image, which may indicate a degenerate generation mode even if FVD is low.

Human Evaluation Protocols

Automatic metrics, no matter how sophisticated, remain imperfect proxies for human perception. For this reason, human evaluation remains the gold standard for assessing video generation quality.

The most common protocol is the two-alternative forced choice (2AFC): evaluators are shown pairs of videos (one real, one generated, or two generated by different models) and asked to choose which is more realistic or better matches a text prompt. The fraction of times a model's output is preferred yields a win rate that is directly interpretable.

A more nuanced protocol decomposes the evaluation into separate dimensions, mirroring the VBench structure. Evaluators are asked to rate each video on temporal consistency, visual quality, motion naturalness, and text alignment on a Likert scale (e.g., 1 to 5). This decomposed human evaluation is expensive (typically requiring hundreds of evaluators and thousands of judgements) but provides the most reliable signal for model comparison.

Evaluation Summary

Table 8 compares the key evaluation approaches discussed in this section.

Metric	Type	Strengths	Weaknesses	Best for
FVD	Distributional	Single number; widely used; cheap to compute; I3D features do see motion	16-frame window hides long-range drift; Gaussian fit; no per-sample score	Quick comparisons across models
VBench	Decomposed	Diagnostic; 16 dimensions; interpretable	Complex to set up; evaluator-dependent	Identifying specific failure modes
${\mathrm{TC}}_{\text{CLIP}}$	Per-video	Lightweight; semantic level	Local only: nearly blind to slow global drift	Temporal consistency screening
Warping error	Per-video	Pixel-level; motion-aware	Requires flow estimation; sensitive to occlusion	Motion quality assessment
Human eval	Perceptual	Gold standard; holistic	Expensive; slow; subjective	Final model comparison

Specialised Applications

The general-purpose text-to-video pipeline developed in the preceding sections can be specialised to a remarkable variety of application domains. In each case, the core diffusion backbone remains largely unchanged; what differs is the conditioning signal, the data domain, and the task-specific loss functions or architectural adaptations. In this section, we survey four particularly active application areas: human motion generation, audio-visual synthesis, talking head generation, and virtual try-on.

Human Motion Generation

Human motion generation aims to synthesise realistic sequences of body poses, typically represented as trajectories of joint positions or rotations over time. Unlike pixel-space video generation, the output is a structured skeletal sequence, making the problem both lower-dimensional and more constrained.

The Motion Diffusion Model (MDM) [75] pioneered the application of diffusion to text-conditioned motion generation. MDM trains a denoiser ${ϵ}_{\theta }({𝖰}_t,t,𝒄)$ that takes a noised motion sequence ${𝖰}_t$ at noise level $t$ and a text embedding $𝒄$ (e.g., “a person walks forward and then waves”), and predicts the clean motion ${𝖰}_0$. The loss is the standard denoising objective applied to the motion tensor: (MDM LOSS)$$\begin{array}{ccc}& {ℒ}_{\text{MDM}}={𝔼}_{t,{𝖰}_0,𝝐}[\Vert {𝖰}_0-{\widehat{𝖰}}_{\theta }({𝖰}_t,t,𝒄){\Vert }^2],& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${\widehat{𝖰}}_{\theta }$ is the ${𝒙}_0$-prediction variant of the denoiser.

The compact representation allows motion diffusion models to generate 4-second sequences in under one second on a single GPU, a stark contrast to the minutes required for pixel-space video generation. However, rendering the skeletal motion as a photorealistic video requires a separate rendering stage, often using a pretrained video diffusion model conditioned on the skeleton sequence [76][36].

Audio-Visual Generation

Videos in the real world are rarely silent: speech, music, ambient sounds, and sound effects are integral to the viewing experience. Audio-visual generation aims to produce video and audio jointly, or to condition one modality on the other.

The core architectural idea is to introduce cross-modal attention between audio and video representations within the diffusion backbone. Let ${𝖹}^{\text{vid}}\in {\mathrm{ℝ}}^{F^{\prime }\times H^{\prime }\times W^{\prime }\times C_v}$ be the video latent and ${𝖹}^{\text{aud}}\in {\mathrm{ℝ}}^{T_a\times C_a}$ be the audio spectrogram latent (where $T_a$ is the number of audio frames and $C_a$ is the frequency dimension). Cross-modal attention computes: (AV QKV)$$\begin{array}{cccc}& 𝑸& ={𝖹}^{\text{vid}}{𝐖}_Q,𝑲={𝖹}^{\text{aud}}{𝐖}_K,𝑽={𝖹}^{\text{aud}}{𝐖}_V,& \text{<span class="tml-eqn"></span>}\\ & \mathrm{CrossAttn}({𝖹}^{\text{vid}},{𝖹}^{\text{aud}})& =\mathrm{softmax}\left(\frac{𝑸{𝑲}^{𝖳}}{\sqrt{d_k}}\right)𝑽,& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${𝐖}_Q,{𝐖}_K,{𝐖}_V$ are learned projection matrices. This allows the video denoiser to attend to audio features at each denoising step, aligning visual events with their corresponding sounds.

Ruan et al. [77] demonstrated that a joint audio-video diffusion model (MM-Diffusion) can generate coherent audio-video pairs by running coupled diffusion processes in both modalities, with cross-attention providing the synchronisation signal. The key challenge is temporal alignment: a drum hit at time $t$ in the audio must correspond to the visual impact at the same time $t$ in the video. This is typically enforced through positional encoding that shares a common time axis across modalities.

Talking Head Synthesis

Talking head synthesis generates a video of a person speaking, driven by an audio clip, a text transcript, or both. The output must satisfy three simultaneous constraints: (i) lip movements must be synchronised with the audio, (ii) facial expressions must be natural and emotionally appropriate, and (iii) the identity of the person must remain consistent across all frames.

Facial Action Unit Conditioning

Many talking-head models condition on facial action units (AUs), a standardised encoding of facial muscle activations derived from the Facial Action Coding System (FACS). Each AU corresponds to a specific muscle group (e.g., AU1 = inner brow raiser, AU12 = lip corner puller). A vector of AU intensities $𝒂\in {[0,1]}^A$ (where $A$ is the number of tracked AUs, typically 17–46) provides a compact, disentangled representation of facial expression that can be extracted from audio using a learned mapping.

Lip Synchronisation Loss

A critical component of talking-head quality is lip-audio synchronisation. The standard approach uses a pretrained sync expert network $S$ that takes a short video clip (typically 5 frames centred on the mouth region) and the corresponding audio segment, and outputs a synchronisation score $S({𝒙}_{\text{mouth}}^{f-2:f+2},{𝒂}^{f-2:f+2})\in [0,1]$. The five-frame window is only defined for centre frames that have two neighbours on each side, so the loss is averaged over $f=3,\dots ,F-2$ (there are $F-4$ such frames, assuming $F\ge 5$): (SYNC LOSS)$$\begin{array}{ccc}& {ℒ}_{\text{sync}}=-\frac{1}{F-4}\sum _{f=3}^{F-2}\log S({𝒙}_{\text{mouth}}^{f-2:f+2},{𝒂}^{f-2:f+2}).& \text{<span class="tml-eqn"></span>}\end{array}$$ Implementations that prefer to score all $F$ frames replicate the first and last frame (and the corresponding audio) twice, so that the window $f-2:f+2$ is defined for every $f$; the sum then runs over $f=1,\dots ,F$ with the normaliser $1/F$. The two conventions differ only in how the four boundary frames are weighted.

EMO [78] and AniPortrait [79] represent the current state of the art in diffusion-based talking-head synthesis. EMO introduces a speed controller and a face region controller that allow fine-grained control over the dynamics and spatial extent of facial animations. AniPortrait uses a two-stage pipeline: first generating a sequence of facial landmarks from audio, then rendering those landmarks into a photorealistic video using a diffusion-based image animator.

Virtual Try-On

Virtual try-on generates images or videos of a person wearing a specified garment, given a reference image of the person and a flat-lay image of the garment. The diffusion-based approach conditions the generation on a warped garment obtained through appearance flow estimation.

For video virtual try-on, the warping must be temporally coherent: the garment must deform smoothly as the person moves. This is achieved by extending the appearance flow to a spatiotemporal field $𝒗\in {\mathrm{ℝ}}^{F\times H\times W\times 2}$ and adding a temporal smoothness regulariser: (FLOW Smooth)$$\begin{array}{ccc}& {ℒ}_{\text{smooth}}=\frac{1}{F-1}\sum _{f=1}^{F-1}\Vert {𝒗}^{f+1}-{𝒗}^f{\Vert }^2.& \text{<span class="tml-eqn"></span>}\end{array}$$

DreamPose [80] demonstrates that a pretrained image diffusion model can be adapted for fashion video generation by conditioning on pose sequences extracted from a driving video. CatVTON [81] simplifies the pipeline by using concatenation-based conditioning, directly concatenating the garment image with the noised person image along the channel dimension, eliminating the need for a separate warping network.

Connections and Synthesis

Video diffusion models do not exist in isolation. They draw upon, extend, and illuminate ideas from nearly every major topic covered in this book. In this section, we make these connections explicit, linking the techniques developed throughout this chapter to the theoretical foundations established in earlier parts. Each connection is presented as an insight box that identifies the specific relationship and points to the relevant chapter for further study.

Variational Inference

Optimal Transport and Flow Matching

Generative Adversarial Networks

Memory and Long-Context Processing

Continual Learning

Agentic AI and World Models

Normalizing Flows

Neural Network Engineering

Connections Map

fig:vdiff:connection-map visualises the connections between video diffusion and the other topics covered in this book.

Open Problems and Future Directions

Despite the remarkable progress documented in this chapter, video diffusion models remain far from the ideal video generation system envisioned in our desiderata (The video generation desiderata). In this section, we identify five fundamental open problems that define the frontier of the field. For each problem, we provide a formal statement, discuss why it is hard, and outline promising directions of attack.

Open Problem 1: Real-Time Generation

Current state-of-the-art video diffusion models require minutes to generate a few seconds of video on high-end GPUs. Interactive applications (video games, virtual reality, live communication) require generation at or near the video frame rate, which for standard video means 24–60 frames per second.

Achieving this conjecture requires roughly a $\mathrm{1,000}\times$ speedup over current systems, which must come from a combination of sources:

1. Architectural efficiency: Replacing quadratic attention with linear-time alternatives (state-space models, linear attention, sparse attention patterns) could reduce the per-step cost by $10$–$100\times$ for long videos.

2. Fewer sampling steps: Consistency distillation, adversarial distillation, and rectified flow already reduce the step count from 50–100 to 1–4, providing a $10$–$50\times$ speedup.

3. Temporal caching: Reusing attention maps and intermediate activations across nearby frames can reduce the effective number of full network evaluations by $2$–$5\times$.

4. Hardware specialisation: Custom accelerators designed for the specific computation patterns of diffusion models (e.g., streaming denoising with pipelined attention) could provide an additional $2$–$5\times$ improvement.

The product of these factors ($100\times 25\times 3\times 3\approx \mathrm{22,500}$) suggests that the conjecture is not obviously impossible, but each factor is at the optimistic end of current estimates.

Open Problem 2: Minute-Scale Temporal Consistency

Generating videos longer than 10–15 seconds with consistent characters, environments, and narratives remains extremely challenging. Current approaches use autoregressive chunking (where each new chunk is conditioned on the end of the previous one), but this introduces accumulated drift: small inconsistencies compound over time, leading to “identity morphing” where characters gradually change appearance, and “environment shifting” where backgrounds drift between shots.

This problem is fundamentally difficult because it requires the model to maintain a persistent “world state” across a context window that is orders of magnitude larger than current training clip lengths. Promising approaches include:

1. Hierarchical planning: first generating a coarse storyboard at low temporal resolution, then filling in details at full resolution conditioned on the storyboard;

2. Persistent memory modules: maintaining an explicit memory bank of character appearances, object states, and scene layouts that is queried and updated throughout generation;

3. Test-time optimisation: periodically optimising the generation to enforce consistency constraints (identity matching, background alignment) between distant frames.

Open Problem 3: Physical Grounding

Current video diffusion models learn an implicit model of physics from training data: they produce videos where objects usually fall downward, water usually flows downhill, and rigid bodies usually maintain their shape. However, they have no explicit physical constraints and routinely produce physically impossible outputs (objects that pass through each other, fluids that flow upward, shadows that are inconsistent with the light source).

This problem sits at the intersection of generative modelling and physics-informed machine learning. The difficulty is twofold: (i) the physical constraints are defined in pixel space (or a physical state space) but the diffusion process operates in latent space, so the constraints must be “lifted” through the decoder; and (ii) hard constraints on the support of a learned distribution are fundamentally harder to enforce than soft penalties in the loss function.

Possible approaches include:

1. Guided sampling: Using a differentiable physics simulator as a guidance function during the denoising process, steering the trajectory toward physically valid outputs at each step;

2. Constrained latent spaces: Training the video autoencoder such that its latent space inherently respects physical constraints (e.g., by training with a physics-based reconstruction loss);

3. Hybrid architectures: Combining a neural diffusion model (for appearance) with a symbolic physics engine (for dynamics), generating the physical state trajectory first and then rendering it into video.

Open Problem 4: Compositional Generation

Generating videos with multiple interacting objects, each with distinct attributes and behaviours, remains a significant weakness of current models. A prompt such as “a red ball bounces off a blue cube while a green sphere rolls in from the left” requires the model to:

1. correctly instantiate three distinct objects;

2. assign the correct colour to each;

3. animate each object with physically plausible motion;

4. model the interaction (the bounce) between the ball and cube;

5. maintain the identity and attributes of each object throughout the video.

Current models frequently fail at one or more of these sub-tasks, exhibiting attribute leakage (the red ball becomes blue), object merging (two objects fuse into one), or interaction failure (the ball passes through the cube).

Compositional generation requires the model to maintain a form of object-centric representation within the diffusion process, a capability that monolithic architectures do not naturally support. Promising directions include slot-based diffusion models (where each object has its own latent “slot” that interacts with other slots through learned dynamics), layout-conditioned generation (where bounding boxes or segmentation masks constrain where each object appears), and compositional score functions (where the score for a multi-object scene is decomposed into per-object scores that are combined at inference time).

Open Problem 5: Unified Multimodal Generation

The ultimate vision for generative AI is a single model that can produce any combination of modalities: video with synchronised audio and subtitles, 3D scenes that can be explored from any viewpoint, and interactive environments that respond to user actions. Current systems treat each modality (video, audio, text, 3D) with separate models or separate heads, limiting the coherence of cross-modal outputs.

This is perhaps the most ambitious open problem in generative AI. Progress will likely require:

1. A unified tokenisation scheme that maps all modalities into a shared latent space;

2. A shared backbone architecture (likely a transformer) that can process tokens from all modalities with appropriate inductive biases for each;

3. Training procedures that balance the learning signals from different modalities, preventing any single modality from dominating;

4. Evaluation frameworks that measure cross-modal consistency, not just per-modal quality.

Exercises