22 Diffusion Language Models

Introduction and Motivation

Language generation has been dominated by autoregressive (AR) models. From early recurrent neural networks to modern Transformers such as GPT [11], LLaMA [12], and their successors, the dominant paradigm is simple: generate text one token at a time, left to right, by factoring the joint probability as (AR Factorization)$$\begin{array}{ccc}& p(𝒙)=p(x^1)\prod _{i=2}^L{p}_{\theta }(x^i|x^1,\dots ,x^{i-1}),& \text{<span class="tml-eqn"></span>}\end{array}$$ where $𝒙=(x^1,x^2,\dots ,x^L)$ is a sequence of tokens from a vocabulary $𝒱$ of size $K$. This factorization is exact by the chain rule of probability; no approximation is needed. Training is straightforward via teacher forcing, and generation proceeds token by token.

Yet this very simplicity conceals fundamental limitations. Consider the following:

• Sequential bottleneck. Generating $L$ tokens requires $L$ serial forward passes through the network. Each pass is memory-bandwidth bound on modern GPUs, leading to poor hardware utilization. Even with billions of FLOPS available, the GPU sits mostly idle waiting for memory.

• Left-to-right commitment. Once a token is generated, it cannot be revised. The model must commit to each word before seeing the rest of the sentence. Humans do not write this way; we draft, revise, and refine.

• The reversal curse. Autoregressive models trained on “A is B” struggle to answer “What is B?” with “A”, because the left-to-right factorization never conditions on the latter part of a sequence to predict the former [1].

• No native infilling. Filling in a blank in the middle of a sentence requires special training modifications (fill-in-the-middle) that are not part of the natural AR framework.

Meanwhile, in the domain of images, diffusion models have achieved spectacular success. Models like DDPM [13], Stable Diffusion [14], and DALL-E [15] generate images by iteratively denoising from pure noise, producing all pixels simultaneously rather than one at a time. The quality of generated images has surpassed that of GANs in many settings. This raises a natural question:

This chapter tells the story of how this question has been progressively answered. We begin by reviewing the key background: diffusion models for images, and the classical masked autoregressive models that foreshadow many ideas in diffusion language models. We then identify the fundamental challenge (language is discrete, not continuous) and show how this challenge has been addressed through discrete diffusion processes, culminating in the modern masked diffusion language models (MDLMs). We trace the development from D3PM [16] through MDLM [17] and SEDD [18] to the large-scale LLaDA [5], LLaDA 2.0 [19], and Mercury [7] models. Along the way, we present detailed mathematical derivations, TikZ diagrams of the key processes, and discussions of the practical advantages and challenges.

Background: Autoregressive and Masked Models

Before diving into diffusion, it is instructive to review the classical masked and autoregressive generative models for sequences. These models share deep connections with diffusion language models, and understanding them provides essential intuition.

Autoregressive Models: NADE, MADE, PixelRNN, PixelCNN

The autoregressive factorization is the backbone of many generative models. The key design choice is how to parameterize each conditional $p_{\theta }(x^i|x^{<i})$.

NADE (Neural Autoregressive Density Estimation).

NADE [20] models each conditional using a single-hidden-layer neural network that shares parameters across positions. For a $D$-dimensional binary vector $𝒙$: $$\begin{array}{cccc}& p(𝒙)& =\prod _{d=1}^{D}p(x_d|{𝒙}_{<d}),& \text{<span class="tml-eqn"></span>}\\ & {𝒉}_d& =\sigma (𝒄+{𝐖}_{:,<d}{𝒙}_{<d}),& \text{<span class="tml-eqn"></span>}\\ & p(x_d=1|{𝒙}_{<d})& =\sigma ({𝒃}_d+{𝐕}_{d,:}{𝒉}_d),& \text{<span class="tml-eqn"></span>}\end{array}$$ where $\sigma$ is the sigmoid function, $𝐖$ and $𝐕$ are weight matrices, and $𝒄$ and $𝒃$ are bias vectors. The crucial feature is that the hidden activation ${𝒉}_d$ depends only on the preceding variables ${𝒙}_{<d}$, enforcing the autoregressive property.

MADE (Masked Autoencoder for Distribution Estimation).

MADE [21] achieves the autoregressive property in a single forward pass by applying carefully designed binary masks to the weight matrices of a standard autoencoder. Each unit in the network is assigned a number $m(k)\in \{1,\dots ,D-1\}$ indicating which input dimensions it is allowed to depend on. The mask for a weight connecting unit $k$ in layer $\ell$ to unit $k^{\prime }$ in layer $\ell +1$ is: $$\begin{array}{ccc}& {𝐌}_{k^{\prime },k}^{\ell }=\mathrm{𝟏}[m^{\ell +1}(k^{\prime })\ge m^{\ell }(k)].& \text{<span class="tml-eqn"></span>}\end{array}$$ The output mask ensures $p(x_d|\cdot )$ depends only on ${𝒙}_{<d}$: $$\begin{array}{ccc}& {𝐌}_{d,k}^{\text{out}}=\mathrm{𝟏}[d>m^L(k)].& \text{<span class="tml-eqn"></span>}\end{array}$$

PixelRNN and PixelCNN.

For image generation, PixelRNN [22] applies the autoregressive factorization to pixels, scanning the image in raster order (top to bottom, left to right): $$\begin{array}{ccc}& p(𝒙)=\prod _{i=1}^{N^2}p(x_i|x_1,\dots ,x_{i-1}),& \text{<span class="tml-eqn"></span>}\end{array}$$ where each pixel $x_i$ takes values in $\{0,1,\dots ,255\}$ (modeled via a softmax over 256 categories). PixelRNN uses LSTM layers to capture long-range dependencies along the raster scan. PixelCNN [23] replaces the recurrence with masked convolutions, convolutional filters that are zeroed out for “future” pixels, ensuring the autoregressive property.

Order-Agnostic Autoregressive Models and XLNet

An important precursor to diffusion language models is the order-agnostic autoregressive model. XLNet [10] proposed training an autoregressive model over all possible permutations of the input sequence: $$\begin{array}{ccc}& {ℒ}_{\text{XLNet}}={𝔼}_{\pi \sim \text{Uniform}(S_L)}[-\sum _{i=1}^L\log p_{\theta }(x_{\pi (i)}|x_{\pi (1)},\dots ,x_{\pi (i-1)})],& \text{<span class="tml-eqn"></span>}\end{array}$$ where $S_L$ is the set of all permutations of $\{1,\dots ,L\}$. At each training step, a random permutation is sampled, and the model learns to predict each token given the preceding tokens in that permutation order.

This is significant because:

• Unlike BERT, XLNet is a proper generative model: it can assign exact likelihoods to sequences.

• Unlike standard AR models, it sees bidirectional context during training (through permuted orders).

• The expected loss over all permutations gives equal weight to every possible conditioning pattern.

The following table summarizes the key properties of these model families:

Masked Language Modeling (BERT)

A different approach to modeling sequences, which will prove central to diffusion language models, is masked language modeling (MLM), introduced by BERT [3]. Rather than factoring the distribution autoregressively, BERT randomly masks a fraction of tokens (typically 15%) and trains a bidirectional Transformer to predict the masked tokens from the unmasked context: $$\begin{array}{ccc}& {ℒ}_{\text{MLM}}=-{𝔼}_{𝒙\sim p_{\text{data}}}{𝔼}_{\text{mask}}\left[\sum _{i\in \text{masked}}\log p_{\theta }(x^i|{𝒙}_{\text{unmasked}})\right].& \text{<span class="tml-eqn"></span>}\end{array}$$

BERT was originally designed as a representation learner, not a generative model. However, the connection between masking and diffusion turns out to be profound. As we will show in Section Masked Diffusion Language Models (MDLM), the training objective of masked diffusion language models is precisely a weighted average of MLM losses at different masking rates. BERT's MLM can be viewed as training a diffusion model at a single, fixed noise level.

Recap: Diffusion Models for Images

We briefly recap the continuous diffusion framework for images, which forms the foundation for understanding the discrete extensions needed for language. This material is covered in detail in earlier chapters; here we highlight the aspects most relevant to the discrete setting.

Forward Process (Adding Noise)

Given a data point ${𝒙}_0\sim p_{\text{data}}$, the forward process adds Gaussian noise over $T$ steps: $$\begin{array}{cccc}& q({𝒙}_t|{𝒙}_{t-1})& =𝒩({𝒙}_t;\sqrt{1-{\beta }_t}{𝒙}_{t-1},{\beta }_t𝐈),& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${\beta }_1,\dots ,{\beta }_T$ is a noise schedule. Defining ${\alpha }_t=1-{\beta }_t$ and ${\bar{\alpha }}_t={\prod }_{s=1}^t{\alpha }_s$, the marginal at any time $t$ is available in closed form: (Continuous Forward Marginal)$$\begin{array}{ccc}& q({𝒙}_t|{𝒙}_0)=𝒩({𝒙}_t;\sqrt{{\bar{\alpha }}_t}{𝒙}_0,(1-{\bar{\alpha }}_t)𝐈).& \text{<span class="tml-eqn"></span>}\end{array}$$ As $t\to T$ with appropriate scheduling, ${\bar{\alpha }}_T\approx 0$ and ${𝒙}_T\approx 𝒩(\mathrm{𝟎},𝐈)$.

Reverse Process (Denoising)

The reverse process generates data by denoising: $$\begin{array}{ccc}& p_{\theta }({𝒙}_{t-1}|{𝒙}_t)=𝒩({𝒙}_{t-1};{𝝁}_{\theta }({𝒙}_t,t),{\sigma }_t^2𝐈).& \text{<span class="tml-eqn"></span>}\end{array}$$ The true reverse posterior, conditioned on ${𝒙}_0$, is also Gaussian: $$\begin{array}{ccc}& q({𝒙}_{t-1}|{𝒙}_t,{𝒙}_0)=𝒩({𝒙}_{t-1};{\widetilde{𝝁}}_t({𝒙}_t,{𝒙}_0),{\tilde{\beta }}_t𝐈),& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${\widetilde{𝝁}}_t$ and ${\tilde{\beta }}_t$ are known functions of ${𝒙}_t$, ${𝒙}_0$, and the noise schedule.

Training Objective

The ELBO decomposes into a sum of KL divergences: $$\begin{array}{cccc}& \log p({𝒙}_0)& \ge {𝔼}_q[\log p_{\theta }({𝒙}_0|{𝒙}_1)& \\ & & -\sum _{t=2}^T{D}_{\text{KL}}(q({𝒙}_{t-1}|{𝒙}_t,{𝒙}_0)& \\ & & \Vert p_{\theta }({𝒙}_{t-1}|{𝒙}_t))& \\ & & -D_{\text{KL}}(q({𝒙}_T|{𝒙}_0)\Vert p({𝒙}_T)\left)\right].& \text{<span class="tml-eqn"></span>}\end{array}$$ With the ${𝒙}_0$-prediction parameterization, this simplifies to a reweighted MSE loss. For the $ϵ$-prediction parameterization (DDPM), the loss becomes: $$\begin{array}{ccc}& {ℒ}_{\text{simple}}={𝔼}_{t,{𝒙}_0,𝝐}[\Vert 𝝐-{𝝐}_{\theta }({𝒙}_t,t){\Vert }^2].& \text{<span class="tml-eqn"></span>}\end{array}$$

Why Continuous Diffusion Does Not Directly Work for Language

Language tokens live in a discrete space $𝒱=\{1,2,\dots ,K\}$. Adding Gaussian noise to a one-hot vector produces a continuous vector that no longer corresponds to any valid token. Several attempts have been made to apply continuous diffusion to text by embedding tokens into a continuous space [24][25], but these approaches face fundamental difficulties:

1. Embedding-space mismatch. Rounding a continuous diffusion output back to the nearest token embedding introduces errors that accumulate during sampling.

2. No natural metric. Unlike pixel values where nearby values are similar (pixel 128 is close to 129), tokens have no natural ordering or distance metric. The word “cat” is not “close to” “bat” in any meaningful geometric sense that Gaussian noise respects.

3. High dimensionality. Modern vocabularies have $K=\mathrm{32,000}$ to $\mathrm{150,000}$ tokens. Operating in this high-dimensional discrete space via continuous relaxations is wasteful.

Discrete Diffusion: The D3PM Framework

The first principled framework for diffusion in discrete spaces was D3PM (Structured Denoising Diffusion Models in Discrete State-Spaces) [16]. D3PM generalizes DDPM to categorical data by replacing Gaussian transitions with categorical transitions defined by stochastic matrices.

Setup

Consider a single token $x_0\in \{1,2,\dots ,K\}$, represented as a one-hot vector in ${\mathrm{ℝ}}^K$. The forward process is a Markov chain defined by transition matrices ${𝐐}_t\in {\mathrm{ℝ}}^{K\times K}$, where entry ${[{𝐐}_t]}_{ij}=q(x_t=j|x_{t-1}=i)$.

The marginal at time $t$ given the initial data $x_0$ is computed via the cumulative product: (D3PM Marginal)$$\begin{array}{cccc}& q(x_t|x_0)& =\mathrm{Categorical}(x_t;x_0{\overline{𝐐}}_t),\text{where}{\overline{𝐐}}_t={𝐐}_1{𝐐}_2\cdots {𝐐}_t.& \text{<span class="tml-eqn"></span>}\end{array}$$

This is the discrete analogue of the Gaussian marginal . Just as $\sqrt{{\bar{\alpha }}_t}$ controls how much of the original signal remains in continuous diffusion, ${\overline{𝐐}}_t$ controls how much of the original token identity is preserved.

Types of Transition Matrices

D3PM considers three types of transition matrices, each inducing a different corruption process:

1. Uniform Diffusion.

$$\begin{array}{ccc}& {𝐐}_t^{\text{unif}}=(1-{\beta }_t)𝐈+\frac{{\beta }_t}{K}𝟙{𝟙}^{\top },& \text{<span class="tml-eqn"></span>}\end{array}$$ where $𝟙$ is the all-ones vector. Each token either stays the same (with probability $1-{\beta }_t$) or transitions uniformly to any of the $K$ states (with probability ${\beta }_t/K$). The stationary distribution is uniform over all tokens.

2. Absorbing State Diffusion.

(Absorbing Transition)$$\begin{array}{ccc}& {[{𝐐}_t^{\text{abs}}]}_{ij}=\{\begin{array}{cc}1-{\beta }_t& \text{if }i=j\text{ and }i\ne [\text{ᴍᴀsᴋ}]\\ {\beta }_t& \text{if }j=[\text{ᴍᴀsᴋ}]\text{ and }i\ne [\text{ᴍᴀsᴋ}]\\ 1& \text{if }i=j=[\text{ᴍᴀsᴋ}]\\ 0& \text{otherwise}\end{array}& \text{<span class="tml-eqn"></span>}\end{array}$$ Tokens either stay the same or transition to a special absorbing state $[\text{ᴍᴀsᴋ}]$. Once masked, a token remains masked. The stationary distribution is a point mass on $[\text{ᴍᴀsᴋ}]$.

3. Discretized Gaussian Diffusion.

$$\begin{array}{ccc}& {[{𝐐}_t^{\text{gauss}}]}_{ij}\propto \exp (-c|i-j{|}^2),& \text{<span class="tml-eqn"></span>}\end{array}$$ for some constant $c$. This imposes ordinal structure: nearby states are more likely transitions. This is suitable for data with natural ordering, such as discretized pixel values.

Reverse Posterior

Using Bayes' theorem, the true reverse posterior (conditioned on knowing $x_0$) is: (D3PM Reverse Posterior)$$\begin{array}{cccc}& q(x_{t-1}|x_t,x_0)& =\frac{q(x_t|x_{t-1})q(x_{t-1}|x_0)}{q(x_t|x_0)}.& \text{<span class="tml-eqn"></span>}\end{array}$$ In categorical/matrix form: (D3PM Reverse Categorical)$$\begin{array}{ccc}& q(x_{t-1}|x_t,x_0)=\mathrm{Categorical}(x_{t-1};\frac{x_t{𝐐}_t^{\top }\odot x_0{\overline{𝐐}}_{t-1}}{{\sum }_j{[x_t{𝐐}_t^{\top }\odot x_0{\overline{𝐐}}_{t-1}]}_j}),& \text{<span class="tml-eqn"></span>}\end{array}$$ where $\odot$ denotes the elementwise (Hadamard) product. This is the discrete analogue of the Gaussian posterior in continuous DDPM.

$x_0$-Parameterization and Training

The model learns $p_{\theta }(x_{t-1}|x_t)$ using the $x_0$-parameterization: a neural network predicts a distribution over clean data ${\tilde{p}}_{\theta }(x_0|x_t)$, and then the reverse transition is computed by plugging this prediction into the Bayes posterior: (D3PM X0 Param)$$\begin{array}{ccc}& p_{\theta }(x_{t-1}|x_t)=\sum _{{\hat{x}}_0}q(x_{t-1}|x_t,{\hat{x}}_0){\tilde{p}}_{\theta }({\hat{x}}_0|x_t).& \text{<span class="tml-eqn"></span>}\end{array}$$

The training objective is the variational lower bound (VLB): (D3PM VLB)$$\begin{array}{cccc}& {ℒ}_{\text{VLB}}={𝔼}_{x_0}[& \underset{{ℒ}_T}{\underbrace{D_{\text{KL}}(q(x_T|x_0)\Vert p(x_T))}}& \\ & & +\sum _{t=2}^T\underset{{ℒ}_t}{\underbrace{{𝔼}_{q(x_t|x_0)}\left[D_{\text{KL}}\right(q(x_{t-1}|x_t,x_0)\Vert p_{\theta }(x_{t-1}|x_t)\left)\right]}}& \\ & & +\underset{{ℒ}_0}{\underbrace{(-{𝔼}_{q(x_1|x_0)}[\log p_{\theta }(x_0|x_1)])}}].& \text{<span class="tml-eqn"></span>}\end{array}$$

Since all distributions are categorical, the KL divergences have closed-form expressions: $$\begin{array}{ccc}& D_{\text{KL}}(\mathrm{Categorical}(𝒑)\Vert \mathrm{Categorical}(𝒒))=\sum _{k=1}^K{p}_k\log \frac{p_k}{q_k}.& \text{<span class="tml-eqn"></span>}\end{array}$$

Masked Diffusion Language Models (MDLM)

While D3PM provides a general framework for discrete diffusion, it was MDLM (Simple and Effective Masked Diffusion Language Models) [17] that demonstrated that a careful specialization to the absorbing state (masking) process, combined with a simplified training objective, could dramatically close the gap with autoregressive models. MDLM was published at NeurIPS 2024 and represents a key turning point for the field.

Forward Process: Masking

MDLM focuses exclusively on absorbing-state diffusion. For a sequence ${𝒙}_0=(x_0^1,\dots ,x_0^L)$, each token is independently either kept or masked: (MDLM Forward)$$\begin{array}{ccc}& q(z_t^i|x_0^i)=\mathrm{Categorical}(z_t^i;{\alpha }_t{x}_0^i+(1-{\alpha }_t)𝒎),& \text{<span class="tml-eqn"></span>}\end{array}$$ where $𝒎$ is the one-hot vector for the $[\text{ᴍᴀsᴋ}]$ token, ${\alpha }_t\in [0,1]$ is a decreasing noise schedule with ${\alpha }_0=1$ (no masking) and ${\alpha }_1=0$ (fully masked), and $t\in [0,1]$ is continuous time.

In words: at time $t$, each token independently survives with probability ${\alpha }_t$ or is masked with probability $1-{\alpha }_t$.

Simplified Reverse Posterior

For absorbing-state diffusion, the reverse posterior simplifies dramatically compared to the general D3PM case: (MDLM Reverse Posterior)$$\begin{array}{ccc}& q(z_s^i|z_t^i,x_0^i)=\{\begin{array}{cc}\mathrm{Categorical}(z_s^i;z_t^i)& \text{if }{z}_t^i\ne [\text{ᴍᴀsᴋ}]\\ \mathrm{Categorical}(z_s^i;{\displaystyle \frac{(1-{\alpha }_s)𝒎+({\alpha }_s-{\alpha }_t)x_0^i}{1-{\alpha }_t}})& \text{if }{z}_t^i=[\text{ᴍᴀsᴋ}]\end{array}& \text{<span class="tml-eqn"></span>}\end{array}$$ for $0\le s<t\le 1$.

The SUBS Parameterization

The learned reverse process substitutes the neural network prediction ${𝒙}_{\theta }({𝒛}_t,t)$ for the true ${𝒙}_0$: $$\begin{array}{ccc}& p_{\theta }(z_s^i|{𝒛}_t)=q(z_s^i|z_t^i,x_0^i=x_{\theta }^i({𝒛}_t,t)).& \text{<span class="tml-eqn"></span>}\end{array}$$ Two structural constraints are imposed on the network output:

1. Zero masking probability: The network never predicts $[\text{ᴍᴀsᴋ}]$ as a clean token: $⟨x_{\theta }^i({𝒛}_t,t),𝒎⟩=0$. In practice, the logit for $[\text{ᴍᴀsᴋ}]$ is set to $-\infty$ before the softmax.

2. Carry-over unmasking: If $z_t^i\ne [\text{ᴍᴀsᴋ}]$, the output simply copies the input: $z_s^i=z_t^i$.

Rao-Blackwellized Training Objective

The main contribution of MDLM is a dramatically simplified training objective obtained by analytically computing expectations (Rao-Blackwellization). Starting from the D3PM-style ELBO with complex KL divergences, the carry-over and zero-masking constraints allow the diffusion loss to simplify to:

This is simply a weighted cross-entropy loss evaluated only at masked positions. Compared to the D3PM parameterization, this has lower variance because expectations are computed analytically rather than estimated via sampling.

Continuous-Time Limit

Taking $T\to \infty$, the discrete sum becomes an integral:

Noise Schedule Invariance

A remarkable property emerges through the change of variables $\gamma =\log (1-{\alpha }_t)$: (MDLM Schedule Invariant)$$\begin{array}{ccc}& {ℒ}_{\text{NELBO}}^{\infty }=-{𝔼}_q{\int }_{-\infty }^0\sum _{\ell =1}^L\log ⟨x_{\theta }^{\ell }({𝒛}_{\gamma },\gamma ),x_0^{\ell }⟩d\gamma .& \text{<span class="tml-eqn"></span>}\end{array}$$

This eliminates one degree of hyperparameter freedom: any reasonable noise schedule converges to the same objective.

Connection to Masked Language Modeling

For a sequence of length $L$, the MDLM objective can be written as: $$\begin{array}{ccc}& {ℒ}_{\text{NELBO}}^{\infty }=-{𝔼}_q{\int }_0^1\frac{{\alpha }_t^{\prime }}{1-{\alpha }_t}\sum _{\ell =1}^L\log ⟨x_{\theta }^{\ell }({𝒛}_t,t),x_0^{\ell }⟩dt.& \text{<span class="tml-eqn"></span>}\end{array}$$ At each time $t$, the masking rate is $1-{\alpha }_t$, and the loss is the cross-entropy of predicting masked tokens, which is precisely masked language modeling (MLM) at that masking rate. The integral averages over all masking rates from 0 to 1, weighted by $\frac{{\alpha }_t^{\prime }}{1-{\alpha }_t}$.

Score Entropy Discrete Diffusion (SEDD)

An alternative approach to discrete diffusion, developed concurrently with MDLM, is SEDD (Score Entropy Discrete Diffusion) [18], which won the ICML 2024 Best Paper award. Instead of parameterizing the reverse process directly, SEDD learns ratios of probabilities, the discrete analogue of the score function.

The Concrete Score

In continuous diffusion, the score ${\nabla }_{𝒙}\log p_t(𝒙)$ is the key object. In discrete spaces, there are no gradients. SEDD defines the concrete score as the ratio: $$\begin{array}{ccc}& s_{\theta }{(𝒙,t)}_{𝒚}\approx \frac{p_t(𝒚)}{p_t(𝒙)},& \text{<span class="tml-eqn"></span>}\end{array}$$ for states $𝒚$ that are “neighbors” of $𝒙$ (differing at a single position). This is the discrete analogue of the Stein score, since: $$\begin{array}{ccc}& {\nabla }_{𝒙}\log p(𝒙)=\frac{{\nabla }_{𝒙}p(𝒙)}{p(𝒙)}.& \text{<span class="tml-eqn"></span>}\end{array}$$ A key advantage is that normalizing constants cancel in ratios: $$\begin{array}{ccc}& \frac{p(𝒚)}{p(𝒙)}=\frac{e^{f(𝒚)}/Z}{e^{f(𝒙)}/Z}=\frac{e^{f(𝒚)}}{e^{f(𝒙)}}.& \text{<span class="tml-eqn"></span>}\end{array}$$

Continuous-Time Markov Chain (CTMC)

SEDD operates in continuous time. The forward process is governed by a rate matrix $𝐑$: $$\begin{array}{ccc}& \frac{d{p}_t}{dt}=𝐑p_t,p_0=p_{\text{data}}.& \text{<span class="tml-eqn"></span>}\end{array}$$

For absorbing diffusion, the rate matrix is: $$\begin{array}{ccc}& R(i,j)=\{\begin{array}{cc}{\sigma }^{\prime }(t)& \text{if }j=[\text{ᴍᴀsᴋ}]\text{ and }i\ne [\text{ᴍᴀsᴋ}]\\ -{\sigma }^{\prime }(t)& \text{if }i=j\text{ and }i\ne [\text{ᴍᴀsᴋ}]\\ 0& \text{otherwise}\end{array}& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${\sigma }^{\prime }(t)=d\sigma (t)/dt$ is the rate of masking. Each non-mask token transitions to $[\text{ᴍᴀsᴋ}]$ at rate ${\sigma }^{\prime }(t)$.

Score Entropy Loss

The score entropy loss for estimating the concrete score is: (Score Entropy)$$\begin{array}{ccc}& {ℒ}_{\text{SE}}={𝔼}_{x\sim p_t}[\sum _{y\sim x}{s}_{\theta }{(𝒙,t)}_{𝒚}-\frac{p_t(𝒚)}{p_t(𝒙)}\log s_{\theta }{(𝒙,t)}_{𝒚}],& \text{<span class="tml-eqn"></span>}\end{array}$$ where $y\sim x$ denotes neighbors of $x$. This is:

• Convex in $s_{\theta }$, ensuring stable optimization.

• Non-negative, equaling zero if and only if $s_{\theta }{(𝒙,t)}_{𝒚}=p_t(𝒚)/p_t(𝒙)$.

• The discrete analogue of the Fisher divergence used in continuous score matching.

Detailed Derivation: From D3PM to MDLM

The transition from D3PM to MDLM involves several mathematical simplifications that are instructive to derive in full detail. We walk through each step, starting from the general D3PM ELBO and arriving at the clean MDLM objective.

Step 1: Specializing to the Absorbing Process

Recall from the general D3PM framework that the reverse posterior is given by . For the absorbing transition matrix , the cumulative product ${\overline{𝐐}}_t$ has a particularly clean form. For a single token: $$\begin{array}{ccc}& {\overline{𝐐}}_t=\prod _{s=1}^t{𝐐}_s^{\text{abs}}.& \text{<span class="tml-eqn"></span>}\end{array}$$ Since each step either keeps the token (with probability $1-{\beta }_s$) or masks it (with probability ${\beta }_s$), the probability that a token survives to time $t$ is: $$\begin{array}{ccc}& {\alpha }_t\stackrel{\mathrm{def}}{=}\prod _{s=1}^t(1-{\beta }_s).& \text{<span class="tml-eqn"></span>}\end{array}$$ The marginal is then: (Absorbing Marginal)$$\begin{array}{ccc}& q(x_t|x_0)=\{\begin{array}{cc}{\alpha }_t& \text{if }{x}_t=x_0\text{ (token survives)}\\ 1-{\alpha }_t& \text{if }{x}_t=[\text{ᴍᴀsᴋ}]\text{ (token is masked)}\\ 0& \text{otherwise}\end{array}& \text{<span class="tml-eqn"></span>}\end{array}$$ This is exactly the MDLM forward process written in discrete time.

Step 2: Computing the Reverse Posterior Explicitly

We now derive from first principles. Using Bayes' theorem: $$\begin{array}{cccc}& q(x_{t-1}|x_t,x_0)& =\frac{q(x_t|x_{t-1})q(x_{t-1}|x_0)}{q(x_t|x_0)}.& \text{<span class="tml-eqn"></span>}\end{array}$$

Case 1: $x_t\ne [\text{ᴍᴀsᴋ}]$.

If $x_t$ is a real token (not masked), then $x_t=x_0$ (the only way a real token appears at time $t$ is if it was never masked). In this case, $x_{t-1}$ must also equal $x_0$: $$\begin{array}{ccc}& q(x_{t-1}=x_0|x_t=x_0,x_0)=1.& \text{<span class="tml-eqn"></span>}\end{array}$$ This is the “carry-over” property: unmasked tokens remain fixed.

Case 2: $x_t=[\text{ᴍᴀsᴋ}]$.

We need to compute: $$\begin{array}{ccc}& q(x_{t-1}|x_t=[\text{ᴍᴀsᴋ}],x_0)=\frac{q(x_t=[\text{ᴍᴀsᴋ}]|x_{t-1})q(x_{t-1}|x_0)}{q(x_t=[\text{ᴍᴀsᴋ}]|x_0)}.& \text{<span class="tml-eqn"></span>}\end{array}$$

The denominator is $q(x_t=[\text{ᴍᴀsᴋ}]|x_0)=1-{\alpha }_t$ from .

Sub-case 2a: $x_{t-1}=x_0$ (token was unmasked at $t-1$, then masked at $t$).

$$\begin{array}{cccc}& q(x_{t-1}=x_0|x_t=[\text{ᴍᴀsᴋ}],x_0)& =\frac{q([\text{ᴍᴀsᴋ}]|x_0)\cdot q(x_0|x_0)}{1-{\alpha }_t}=\frac{{\beta }_t\cdot {\alpha }_{t-1}}{1-{\alpha }_t}.& \text{<span class="tml-eqn"></span>}\end{array}$$ Since ${\alpha }_t={\alpha }_{t-1}(1-{\beta }_t)$, we have ${\beta }_t{\alpha }_{t-1}={\alpha }_{t-1}-{\alpha }_t$, so: (Posterior Unmask)$$\begin{array}{ccc}& q(x_{t-1}=x_0|x_t=[\text{ᴍᴀsᴋ}],x_0)=\frac{{\alpha }_{t-1}-{\alpha }_t}{1-{\alpha }_t}.& \text{<span class="tml-eqn"></span>}\end{array}$$

Sub-case 2b: $x_{t-1}=[\text{ᴍᴀsᴋ}]$ (token was already masked at $t-1$).

(Posterior Staymask)$$\begin{array}{cccc}& q(x_{t-1}=[\text{ᴍᴀsᴋ}]|x_t=[\text{ᴍᴀsᴋ}],x_0)& =\frac{q([\text{ᴍᴀsᴋ}]|[\text{ᴍᴀsᴋ}])\cdot q([\text{ᴍᴀsᴋ}]|x_0)}{1-{\alpha }_t}=\frac{1\cdot (1-{\alpha }_{t-1})}{1-{\alpha }_t}.& \text{<span class="tml-eqn"></span>}\end{array}$$

We can verify these sum to 1: $$\begin{array}{ccc}& \frac{{\alpha }_{t-1}-{\alpha }_t}{1-{\alpha }_t}+\frac{1-{\alpha }_{t-1}}{1-{\alpha }_t}=\frac{{\alpha }_{t-1}-{\alpha }_t+1-{\alpha }_{t-1}}{1-{\alpha }_t}=\frac{1-{\alpha }_t}{1-{\alpha }_t}=1.✓& \text{<span class="tml-eqn"></span>}\end{array}$$

Combining both cases yields exactly .

Step 3: From KL Divergence to Cross-Entropy

The D3PM VLB contains terms of the form: $$\begin{array}{ccc}& {ℒ}_t={𝔼}_{q(x_t|x_0)}\left[D_{\text{KL}}(q(x_{t-1}|x_t,x_0)\Vert p_{\theta }(x_{t-1}|x_t))\right].& \text{<span class="tml-eqn"></span>}\end{array}$$

For the absorbing process, when $x_t\ne [\text{ᴍᴀsᴋ}]$, both the true and learned reverse posteriors are deterministic (keeping $x_t$ unchanged), so the KL is zero. The non-trivial contribution comes only from $x_t=[\text{ᴍᴀsᴋ}]$: $$\begin{array}{cccc}& {ℒ}_t& =\mathrm{Pr}[x_t=[\text{ᴍᴀsᴋ}]|x_0]\cdot D_{\text{KL}}(q(x_{t-1}|[\text{ᴍᴀsᴋ}],x_0)\Vert p_{\theta }(x_{t-1}|[\text{ᴍᴀsᴋ}]))& \text{<span class="tml-eqn"></span>}\\ & & =(1-{\alpha }_t)\cdot D_{\text{KL}}(q(\cdot )\Vert p_{\theta }(\cdot )).& \text{<span class="tml-eqn"></span>}\end{array}$$

With the SUBS parameterization and the zero-masking-probability constraint, the reverse distribution at a masked position takes the form: $$\begin{array}{ccc}& p_{\theta }(x_{t-1}|x_t=[\text{ᴍᴀsᴋ}])=\{\begin{array}{cc}\frac{1-{\alpha }_{t-1}}{1-{\alpha }_t}& \text{if }{x}_{t-1}=[\text{ᴍᴀsᴋ}]\\ \frac{{\alpha }_{t-1}-{\alpha }_t}{1-{\alpha }_t}\cdot p_{\theta }^v(x_0=v|x_t)& \text{if }{x}_{t-1}=v\ne [\text{ᴍᴀsᴋ}]\end{array}& \text{<span class="tml-eqn"></span>}\end{array}$$ where $p_{\theta }^v(x_0=v|x_t)$ is the network's predicted probability for token $v$.

The KL divergence between the true and learned posteriors at a masked position can be expanded: $$\begin{array}{cccc}& & D_{\text{KL}}(q(x_{t-1}|[\text{ᴍᴀsᴋ}],x_0)\Vert p_{\theta }(x_{t-1}|[\text{ᴍᴀsᴋ}]))& \\ & & =\frac{1-{\alpha }_{t-1}}{1-{\alpha }_t}\log \frac{(1-{\alpha }_{t-1})/(1-{\alpha }_t)}{(1-{\alpha }_{t-1})/(1-{\alpha }_t)}+\frac{{\alpha }_{t-1}-{\alpha }_t}{1-{\alpha }_t}\log \frac{({\alpha }_{t-1}-{\alpha }_t)/(1-{\alpha }_t)}{({\alpha }_{t-1}-{\alpha }_t)\cdot p_{\theta }^{x_0}(x_0|x_t)/(1-{\alpha }_t)}& \text{<span class="tml-eqn"></span>}\\ & & =\frac{{\alpha }_{t-1}-{\alpha }_t}{1-{\alpha }_t}\log \frac{1}{p_{\theta }^{x_0}(x_0|x_t)}& \text{<span class="tml-eqn"></span>}\\ & & =-\frac{{\alpha }_{t-1}-{\alpha }_t}{1-{\alpha }_t}\log p_{\theta }(x_0|x_t).& \text{<span class="tml-eqn"></span>}\end{array}$$

This is the Rao-Blackwellization: the KL divergence at masked positions reduces to a weighted negative log-probability of the true token. The weighting $\frac{{\alpha }_{t-1}-{\alpha }_t}{1-{\alpha }_t}$ is exactly the coefficient in the MDLM loss .

Step 4: Summing Over Positions and Time

For a sequence of length $L$ with independent masking, summing over all positions and time steps: $$\begin{array}{cccc}& \sum _{t=1}^T{ℒ}_t& =-\sum _{t=1}^T{𝔼}_{q({𝒙}_t|{𝒙}_0)}[\frac{{\alpha }_{t-1}-{\alpha }_t}{1-{\alpha }_t}\sum _{\ell =1}^L\mathrm{𝟏}[x_t^{\ell }=[\text{ᴍᴀsᴋ}]]\cdot \log p_{\theta }(x_0^{\ell }|{𝒙}_t)].& \text{<span class="tml-eqn"></span>}\end{array}$$

Taking $T\to \infty$ and using ${\alpha }_{t-1}-{\alpha }_t\to -{\alpha }_t^{\prime }dt$ (where ${\alpha }_t^{\prime }=d{\alpha }_t/dt<0$): $$\begin{array}{ccc}& {ℒ}_{\text{diffusion}}=-{\int }_0^1{𝔼}_q[\frac{-{\alpha }_t^{\prime }}{1-{\alpha }_t}\sum _{\ell =1}^L\mathrm{𝟏}[x_t^{\ell }=[\text{ᴍᴀsᴋ}]]\cdot \log p_{\theta }(x_0^{\ell }|{𝒙}_t)]dt.& \text{<span class="tml-eqn"></span>}\end{array}$$ Since ${\alpha }_t$ is decreasing, ${\alpha }_t^{\prime }<0$, so $-{\alpha }_t^{\prime }>0$ and the weight is positive. This recovers the MDLM continuous-time NELBO .

Step 5: The Change of Variables to Schedule Invariance

The schedule invariance result follows from the substitution $\gamma =\log (1-{\alpha }_t)$, which yields $d\gamma =\frac{-{\alpha }_t^{\prime }}{1-{\alpha }_t}dt$. The integral becomes: $$\begin{array}{ccc}& {ℒ}_{\text{diffusion}}=-{\int }_{-\infty }^0{𝔼}_q[\sum _{\ell =1}^L\mathrm{𝟏}[x_{\gamma }^{\ell }=[\text{ᴍᴀsᴋ}]]\cdot \log p_{\theta }(x_0^{\ell }|{𝒙}_{\gamma })]d\gamma .& \text{<span class="tml-eqn"></span>}\end{array}$$

The key observation is that $\gamma$ absorbs the entire schedule-dependent weighting, leaving an integral that depends only on the masking probability $1-{\alpha }_t=e^{\gamma }$ (since $\gamma =\log (1-{\alpha }_t)$). Two different schedules ${\alpha }_t$ and ${\alpha }_t^{\prime }$ that sweep through the same range of masking probabilities will produce the same integral, just traversed at different speeds.

Worked Example: MDLM on a 3-Token Vocabulary

To make these derivations concrete, consider a toy example with vocabulary $𝒱=\{\text{A},\text{B},[\text{ᴍᴀsᴋ}]\}$ ($K=3$ including mask) and sequence length $L=2$.

The Continuous-Time Markov Chain Perspective

The discrete-time formulations of D3PM and MDLM have natural continuous-time analogues via continuous-time Markov chains (CTMCs). This perspective, developed in Campbell et al. [26] and refined in SEDD [18], provides elegant theoretical properties and connects discrete diffusion to the score-based framework.

Rate Matrices and the Master Equation

A CTMC over a finite state space $\{1,\dots ,K\}$ is characterized by a rate matrix (or generator) $𝐑(t)\in {\mathrm{ℝ}}^{K\times K}$, where:

• $R_{ij}(t)\ge 0$ for $i\ne j$ (transition rate from state $i$ to state $j$)

• $R_{ii}(t)=-{\sum }_{j\ne i}{R}_{ij}(t)$ (rows sum to zero)

The probability distribution ${𝒑}_t=(p_t(1),\dots ,p_t(K))$ evolves according to the master equation (Kolmogorov forward equation): (Master Equation)$$\begin{array}{ccc}& \frac{d{𝒑}_t}{dt}={𝒑}_t𝐑(t).& \text{<span class="tml-eqn"></span>}\end{array}$$

This is the discrete-space analogue of the Fokker-Planck equation in continuous diffusion. The solution is: $$\begin{array}{ccc}& {𝒑}_t={𝒑}_0\exp \left({\int }_0^t𝐑(s)ds\right),& \text{<span class="tml-eqn"></span>}\end{array}$$ where the matrix exponential gives the transition probability matrix.

The Absorbing Rate Matrix

For masked diffusion, the rate matrix is: (Absorbing RATE)$$\begin{array}{ccc}& R_{ij}(t)=\{\begin{array}{cc}{\sigma }^{\prime }(t)& \text{if }j=[\text{ᴍᴀsᴋ}]\text{ and }i\ne [\text{ᴍᴀsᴋ}]\\ -{\sigma }^{\prime }(t)& \text{if }i=j\text{ and }i\ne [\text{ᴍᴀsᴋ}]\\ 0& \text{otherwise}\end{array}& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${\sigma }^{\prime }(t)=d\sigma (t)/dt>0$ controls the masking rate.

In matrix form, for a 3-state system $\{1,2,[\text{ᴍᴀsᴋ}]\}$: $$\begin{array}{ccc}& 𝐑(t)={\sigma }^{\prime }(t)\left(\begin{array}{ccc}-1& 0& 1\\ 0& -1& 1\\ 0& 0& 0\end{array}\right).& \text{<span class="tml-eqn"></span>}\end{array}$$

The masking probability at time $t$ is: $$\begin{array}{ccc}& \mathrm{Pr}[x_t=[\text{ᴍᴀsᴋ}]|x_0\ne [\text{ᴍᴀsᴋ}]]=1-e^{-\bar{\sigma }(t)},\text{where }\bar{\sigma }(t)={\int }_0^t{\sigma }^{\prime }(s)ds.& \text{<span class="tml-eqn"></span>}\end{array}$$ Comparing with MDLM's ${\alpha }_t$: we have ${\alpha }_t=e^{-\bar{\sigma }(t)}$, or equivalently $\bar{\sigma }(t)=-\log {\alpha }_t$.

The Reverse CTMC

The time-reversed CTMC has rate matrix: (Reverse RATE)$$\begin{array}{ccc}& {\bar{R}}_{ij}(t)=\frac{p_t(j)}{p_t(i)}{R}_{ji}(t).& \text{<span class="tml-eqn"></span>}\end{array}$$ This is the discrete analogue of Anderson's reverse-time SDE. The ratio $p_t(j)/p_t(i)$ is exactly the concrete score, which SEDD learns to estimate.

For the absorbing process, the reverse rate simplifies: the only non-trivial transitions are from $[\text{ᴍᴀsᴋ}]$ to real tokens, with rate: $$\begin{array}{ccc}& {\bar{R}}_{[\text{ᴍᴀsᴋ}],v}(t)=\frac{p_t(v)}{p_t([\text{ᴍᴀsᴋ}])}\cdot {\sigma }^{\prime }(t).& \text{<span class="tml-eqn"></span>}\end{array}$$

Large Language Diffusion Models (LLaDA)

With the theoretical foundations from MDLM and SEDD, the next question was: can masked diffusion models scale to the size and capability of modern LLMs? LLaDA (Large Language Diffusion Models) [5] provided a resounding “yes,” demonstrating that an 8-billion parameter diffusion model can match or exceed autoregressive models on many benchmarks.

Core Thesis

LLaDA challenges the prevailing assumption that autoregressive formulations are necessary for emergent LLM capabilities such as in-context learning, instruction following, and chain-of-thought reasoning. The authors argue that these capabilities arise from the interplay between Transformers, model scale, data scale, and Fisher consistency, a statistical property guaranteeing that the model can recover the true data distribution with sufficient data and capacity, rather than from the left-to-right factorization specifically.

Forward and Reverse Process

LLaDA uses the same masking-based forward process as MDLM, but with a clearer presentation in terms of continuous time $t\in [0,1]$:

Forward process (token-level):

(Llada Forward)$$\begin{array}{ccc}& q_{t|0}(x_t^i|x_0^i)=\{\begin{array}{cc}1-t& \text{if }{x}_t^i=x_0^i\\ t& \text{if }{x}_t^i=[\text{ᴍᴀsᴋ}]\end{array}& \text{<span class="tml-eqn"></span>}\end{array}$$ At $t=0$, no tokens are masked; at $t=1$, all tokens are masked. This is the simplest possible masking schedule: ${\alpha }_t=1-t$.

Reverse process (token-level):

For $0\le s<t\le 1$, the reverse transition has four cases: (Llada Reverse)$$\begin{array}{ccc}& q_{s|t}(x_s^i|{𝒙}_t)=\{\begin{array}{cc}1& \text{if }{x}_t^i\ne [\text{ᴍᴀsᴋ}]\text{ and }{x}_s^i=x_t^i\text{(stay unmasked)}\\ 0& \text{if }{x}_t^i\ne [\text{ᴍᴀsᴋ}]\text{ and }{x}_s^i\ne x_t^i\text{(cannot change)}\\ {\displaystyle \frac{s}{t}}& \text{if }{x}_t^i=[\text{ᴍᴀsᴋ}]\text{ and }{x}_s^i=[\text{ᴍᴀsᴋ}]\text{(stay masked)}\\ {\displaystyle \frac{t-s}{t}}\cdot q_{0|t}(x_s^i|{𝒙}_t)& \text{if }{x}_t^i=[\text{ᴍᴀsᴋ}]\text{ and }{x}_s^i\ne [\text{ᴍᴀsᴋ}]\text{(unmask)}\end{array}& \text{<span class="tml-eqn"></span>}\end{array}$$

Training Objective

The LLaDA training loss is: (Llada LOSS)$$\begin{array}{ccc}& ℒ(\theta )=-{𝔼}_{t\sim U(\mathrm{0,1}],{𝒙}_0\sim p_{\text{data}},{𝒙}_t\sim q_{t|0}}[\frac{1}{t}\sum _{i=1}^L\mathrm{𝟏}[x_t^i=[\text{ᴍᴀsᴋ}]]\cdot \log p_{\theta }(x_0^i|{𝒙}_t)].& \text{<span class="tml-eqn"></span>}\end{array}$$

The $1/t$ weighting is theoretically motivated: it ensures that the loss is a valid upper bound on the negative log-likelihood: $$\begin{array}{ccc}& -{𝔼}_{{𝒙}_0\sim p_{\text{data}}}[\log p_{\theta }({𝒙}_0)]\le ℒ(\theta ).& \text{<span class="tml-eqn"></span>}\end{array}$$

Architecture

LLaDA 8B uses essentially the same Transformer architecture as LLaMA 3 8B, with one critical difference:

The key architectural change is removing the causal attention mask, allowing the model to attend to all positions bidirectionally. This is essential for the diffusion formulation, where the model needs to use both left and right context to predict masked tokens. However, it means that the KV cache optimization, crucial for fast autoregressive inference, is not directly applicable.

Supervised Fine-Tuning for Diffusion Models

For instruction following, LLaDA uses a modified SFT loss where only the response tokens are subject to masking: (Llada SFT)$$\begin{array}{ccc}& {ℒ}_{\text{SFT}}(\theta )=-{𝔼}_{t,{𝒑}_0,{𝒓}_0,{𝒓}_t}[\frac{1}{t}\sum _{i=1}^{L^{\prime }}\mathrm{𝟏}[r_t^i=[\text{ᴍᴀsᴋ}]]\cdot \log p_{\theta }(r_0^i|{𝒑}_0,{𝒓}_t)],& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${𝒑}_0$ are the prompt tokens (always unmasked), ${𝒓}_0$ are the response tokens (subject to masking), and $L^{\prime }$ is the response length.

Experimental Results: How Good Are Diffusion Language Models?

Before describing the sampling algorithms, let us examine the empirical evidence for diffusion language models to understand how they compare with autoregressive models across various tasks.

Language Modeling Perplexity (MDLM)

MDLM [17] provides the first strong evidence that masked diffusion can approach autoregressive performance. On the One Billion Word benchmark (LM1B) with 110M parameter models:

MDLM achieves a 17% improvement over SEDD (the prior best diffusion model) and comes within $\sim$10% of the AR baseline. The progression from D3PM ($\le 76.90$) to MDLM ($\le 23.00$) demonstrates the power of the simplified Rao-Blackwellized objective.

Benchmark Results at Scale (LLaDA 8B)

LLaDA 8B [5], trained on 2.3 trillion tokens, provides the first comparison of diffusion models with modern LLMs at the 8B parameter scale:

Remarkably, LLaDA significantly outperforms LLaMA3 on math benchmarks (GSM8K: 70.3 vs. 48.7; MATH: 31.4 vs. 16.0) despite using only 2.3T tokens compared to LLaMA3's 15T tokens. This suggests that the bidirectional architecture may have advantages for mathematical reasoning.

Post-SFT Performance

After supervised fine-tuning, LLaDA 8B Instruct (using only SFT, no RLHF/DPO) achieves:

These results are competitive with AR models that use RLHF alignment, suggesting that the diffusion training paradigm provides strong instruction-following capabilities even without explicit preference optimization.

Representation Quality (MDLM on GLUE)

A key question is whether the diffusion training objective degrades the representation quality of encoder-only models. MDLM tests this by fine-tuning a BERT model with the MDLM objective and evaluating on GLUE:

Fine-tuning BERT with the MDLM objective does not degrade representation quality; it actually slightly improves it (82.1 vs. 81.6 average) while adding generative capabilities. This demonstrates that diffusion training and representation learning are compatible.

Scaling Laws for Diffusion Language Models

LLaDA demonstrates that diffusion language models exhibit scaling behavior comparable to autoregressive models. Testing across FLOPs from ${10}^{20}$ to ${10}^{23}$ on six benchmarks, diffusion models follow the same general scaling trend as ARMs.

An important finding from the scaling experiments is that likelihood metrics alone are insufficient predictors of downstream task performance for diffusion models. This contrasts with AR models, where lower perplexity reliably predicts better downstream performance. For diffusion models, the ELBO (which is an upper bound on negative log-likelihood) does not always correlate with task-specific metrics, suggesting that evaluation strategies for diffusion models need further development.

Sampling Algorithms and Remasking Strategies

Generation in diffusion language models works by iteratively unmasking tokens. Starting from a fully masked sequence ${𝒓}_1=([\text{ᴍᴀsᴋ}],\dots ,[\text{ᴍᴀsᴋ}])$, the model progressively reveals tokens over $N$ steps. The key design choice is the remasking strategy: after the model predicts all masked tokens, which predictions should be kept and which should be re-masked for refinement?

Random Remasking

The simplest approach randomly selects which tokens to remask:

Low-Confidence Remasking

A more effective strategy is to remask the tokens that the model is least confident about:

Block Diffusion: Semi-Autoregressive Generation

Block diffusion [27] interpolates between fully parallel diffusion and fully sequential autoregressive generation. The sequence is divided into blocks, and blocks are generated left-to-right (autoregressively), while tokens within each block are generated via diffusion (in parallel).

Formally, for a sequence divided into blocks $𝒙=({𝒙}_{\le 1},{𝒙}_{\le 2},\dots ,{𝒙}_{\le B})$, the block diffusion loss is: $$\begin{array}{ccc}& {ℒ}_{\text{BDLM}}(\theta )=-𝔼[\frac{{\alpha }_t^{\prime }}{1-{\alpha }_t}\sum _{k=1}^B\sum _i\mathrm{𝟏}[x_{t,k}^i=[\text{ᴍᴀsᴋ}]]\cdot \log p_{\theta }(x_{0,k}^i|{𝒙}_{0,<k},{𝒙}_{t,k})].& \text{<span class="tml-eqn"></span>}\end{array}$$ The block size controls the trade-off:

• Block size = 1: fully autoregressive

• Block size = $L$: fully diffusion-based

• Intermediate: semi-autoregressive

Detailed Analysis of Masking Schedules

The masking schedule ${\alpha }_t$ determines how quickly tokens are corrupted during the forward process. While the continuous-time NELBO is schedule-invariant at convergence, the schedule significantly affects training dynamics, variance, and finite-step sampling quality.

Common Schedules

Linear Schedule.

${\alpha }_t=1-t$ for $t\in [0,1]$. This is the simplest choice and is used in LLaDA. The masking rate increases linearly, and the weight $\frac{-{\alpha }_t^{\prime }}{1-{\alpha }_t}=\frac{1}{t}$ diverges as $t\to 0$ (few tokens masked), placing heavy emphasis on near-complete sequences.

Cosine Schedule.

${\alpha }_t={\cos }^2(\pi t/2)$. Inspired by the cosine schedule for continuous diffusion [28], this provides a smoother transition with less aggressive masking at the start and end: $$\begin{array}{ccc}& \frac{-{\alpha }_t^{\prime }}{1-{\alpha }_t}=\frac{\pi \sin (\pi t/2)\cos (\pi t/2)}{1-{\cos }^2(\pi t/2)}=\frac{\pi \cos (\pi t/2)}{\sin (\pi t/2)}=\pi \cot (\pi t/2).& \text{<span class="tml-eqn"></span>}\end{array}$$

Geometric (Exponential) Schedule.

${\alpha }_t=e^{-\sigma t}$ for some rate parameter $\sigma >0$: $$\begin{array}{ccc}& \frac{-{\alpha }_t^{\prime }}{1-{\alpha }_t}=\frac{\sigma e^{-\sigma t}}{1-e^{-\sigma t}}.& \text{<span class="tml-eqn"></span>}\end{array}$$

Effect on Training Variance

At each training step, we sample a time $t\sim U(0,1]$ and compute the loss at that masking level. The variance of the gradient estimator depends on the schedule through the weight $w(t)=\frac{-{\alpha }_t^{\prime }}{1-{\alpha }_t}$:

• When $w(t)$ varies widely, the gradient estimator has high variance.

• The linear schedule has $w(t)=1/t$, which diverges as $t\to 0$, leading to occasionally very large gradients when a nearly-complete sequence is sampled.

• Low-discrepancy time sampling (e.g., stratified sampling of $t$) can reduce this variance significantly [6].

Mask Ratio Bandwidth

LLaDA 2.0 introduces mask ratio bandwidth: clipping the noise schedule to $[{\alpha }_{\min },{\alpha }_{\max }]$ instead of $[0,1]$, which avoids the extreme masking rates that cause high variance: $$\begin{array}{ccc}& {\tilde{\alpha }}_t={\alpha }_{\min }+({\alpha }_{\max }-{\alpha }_{\min })\cdot {\alpha }_t.& \text{<span class="tml-eqn"></span>}\end{array}$$ This is equivalent to restricting $t$ to a subinterval that excludes very small and very large masking rates.

Training Algorithms in Detail

We present the complete training algorithms for both pre-training and supervised fine-tuning of diffusion language models.

Detailed Sampling Procedures

We present the complete sampling algorithms, including the newly introduced ancestral sampling variant and the semi-autoregressive extension.

Ancestral Sampling (Principled but Slow)

The most principled sampling method follows the learned reverse process exactly:

At each step, each masked token either stays masked (with probability $s/t$) or is unmasked by sampling from the predicted distribution. This follows the exact reverse posterior .

Caching Optimization

When no new tokens are unmasked in a step (which can happen, especially with few remaining masked tokens), the forward pass can be skipped entirely. Furthermore, if the model does not use time conditioning (as suggested by Remark Remark 12), consecutive steps with the same set of unmasked tokens produce identical predictions, enabling a $\sim$2$\times$ speedup.

Semi-Autoregressive (SAR) Generation

For sequences longer than the context window $L$, MDLM introduces a sliding-window approach:

Advantages over Autoregressive LLMs

Diffusion language models offer several structural advantages over their autoregressive counterparts. These advantages are not merely theoretical; they have been demonstrated empirically in recent large-scale models.

Parallel Generation

The most obvious advantage is parallelism. In autoregressive generation, producing $L$ tokens requires $L$ serial forward passes, each of which reads the full KV cache but produces only a single token. This makes AR inference memory-bandwidth bound on modern GPUs, leading to poor hardware utilization.

Diffusion models, in contrast, process and refine all tokens simultaneously in each denoising step. If the model requires $N$ denoising steps (typically $N\ll L$), the total number of forward passes is $N$ rather than $L$. Each forward pass has much higher arithmetic intensity (ratio of compute to memory operations), better saturating modern GPU compute units.

Bidirectionality and the Reversal Curse

Autoregressive models factor the distribution left-to-right, which creates a fundamental asymmetry. If trained on “A is B,” the model learns $p(\text{B}|\text{A})$ but not $p(\text{A}|\text{B})$. This is the reversal curse [1].

Diffusion language models are inherently bidirectional: they use full attention (no causal mask), conditioning on both left and right context simultaneously. This means they naturally learn both directions.

Native Fill-in-the-Middle

Because diffusion models condition bidirectionally, fill-in-the-middle (FIM) is natively supported without special training modifications. Given a prefix and suffix, the model can generate the middle by treating the prefix and suffix as unmasked context and the middle as the masked region to be diffused.

LLaDA achieves 73.8% on HumanEval-FIM versus LLaMA3's 73.3%, despite LLaMA3 requiring specific FIM training.

Iterative Refinement and Error Correction

Perhaps the most intriguing advantage is the ability to self-correct. In autoregressive generation, once a token is emitted, it cannot be changed. An early mistake propagates through the entire remaining sequence. Diffusion models, through the remasking mechanism, can revisit and revise earlier decisions.

In low-confidence remasking (Algorithm alg:confidence_remasking), the model naturally identifies positions where it is uncertain and re-predicts them with additional context. This provides a built-in form of error correction that autoregressive models lack.

Flexible Generation Order

Autoregressive models are committed to a single generation order (left-to-right). Diffusion models can generate tokens in any order; the order is determined dynamically by the confidence scores and remasking strategy. This flexibility allows the model to:

• Generate high-certainty tokens first (e.g., function signatures before implementations).

• Fill in details after establishing the overall structure.

• Naturally handle tasks like code completion where the generation order is not left-to-right.

Classifier-Free Guidance for Diffusion Language Models

Classifier-free guidance (CFG) [29], a technique that dramatically improves conditional generation quality in image diffusion models, can also be applied to diffusion language models.

CFG for Conditional Text Generation

In the image domain, CFG interpolates between conditional and unconditional predictions: $$\begin{array}{ccc}& {\widetilde{𝝐}}_{\theta }({𝒙}_t,t,c)=(1+w){𝝐}_{\theta }({𝒙}_t,t,c)-w{𝝐}_{\theta }({𝒙}_t,t),& \text{<span class="tml-eqn"></span>}\end{array}$$ where $w>0$ is the guidance strength and $c$ is the condition (e.g., text prompt).

For masked diffusion language models, the analogous formulation operates on the predicted token logits. Given a prompt $𝒑$ and a masked response ${𝒓}_t$, the guided logits are: $$\begin{array}{ccc}& {\tilde{\ell }}_{\theta }^i=(1+w){\ell }_{\theta }^i(𝒑,{𝒓}_t)-w{\ell }_{\theta }^i({𝒓}_t),& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${\ell }_{\theta }^i(𝒑,{𝒓}_t)$ are the conditional logits (with prompt) and ${\ell }_{\theta }^i({𝒓}_t)$ are the unconditional logits (prompt replaced with masks or dropped).

Comparison with AR Temperature Scaling

In autoregressive models, the analogue of CFG is temperature scaling: $$\begin{array}{ccc}& p_{\tau }(x^i|{𝒙}^{<i})\propto p_{\theta }{(x^i|{𝒙}^{<i})}^{1/\tau },& \text{<span class="tml-eqn"></span>}\end{array}$$ where $\tau <1$ sharpens the distribution (more deterministic) and $\tau >1$ flattens it (more diverse).

CFG in diffusion models is more powerful than simple temperature scaling because it leverages the difference between conditional and unconditional predictions, which captures information about what the prompt specifically requests. Temperature scaling merely adjusts the sharpness without such semantic grounding.

Challenges: Generation Speed and What Makes It Slow

Despite the advantages, diffusion language models face significant challenges related to generation speed and efficiency. Understanding these challenges is crucial for appreciating the engineering innovations in models like Mercury and LLaDA 2.0.

The Cost of Multiple Denoising Steps

While diffusion models process all tokens in parallel per step, they require multiple denoising steps (typically 16–256) to generate high-quality output. Each step involves a full forward pass through the Transformer. If the model uses $N=64$ steps and the autoregressive model generates $L=512$ tokens:

• AR model: 512 forward passes, each producing 1 token (but using efficient KV cache)

• Diffusion model: 64 forward passes, each processing all 512 tokens

The AR model benefits enormously from the KV cache: after the first pass, each subsequent pass only processes a single new token, reusing cached key-value pairs. The diffusion model must process the full sequence at each step (since bidirectional attention cannot use incremental caching), making each individual forward pass much more expensive.

No KV Cache

The fundamental incompatibility between bidirectional attention and the KV cache is the single biggest efficiency challenge:

• AR models with KV cache: After the prefill phase, each token generation requires attention computation proportional to $O(L)$ (attending to all cached tokens), but the computation is done only for the new token.

• Diffusion models: Each denoising step requires full self-attention over all $L$ tokens, with cost $O(L^2)$, because the attention pattern changes at every step (different tokens are masked/unmasked).

Quality-Speed Trade-off

The number of denoising steps $N$ directly controls the trade-off between generation quality and speed:

• More steps: better quality (more refinement iterations), slower generation.

• Fewer steps: faster generation, but tokens may not be sufficiently refined.

Finding the optimal $N$ is an active research challenge. LLaDA shows that results are relatively insensitive to $N$ within a reasonable range, but extremely few steps (e.g., $N=1$) significantly degrade quality.

Arithmetic Intensity Analysis

To understand the speed difference between AR and diffusion models quantitatively, consider the arithmetic intensity (AI), the ratio of floating-point operations to bytes transferred from memory: $$\begin{array}{ccc}& \text{AI}=\frac{\text{FLOPs}}{\text{Bytes transferred}}.& \text{<span class="tml-eqn"></span>}\end{array}$$

AR inference (generation phase).

After the prefill, each token generation step involves:

• Reading the full KV cache: $O(L\cdot d_{\text{model}})$ bytes

• Computing attention for 1 query token against $L$ keys: $O(L\cdot d_{\text{model}})$ FLOPs

• AI $\approx O(1)$, memory-bandwidth bound

Diffusion inference (each denoising step).

Each step involves:

• Reading model weights: $O(d_{\text{model}}^2)$ bytes (same as AR)

• Computing attention for $L$ query tokens: $O(L^2\cdot d_{\text{model}})$ FLOPs

• AI $\approx O(L)$, compute bound for long sequences

The key insight is that diffusion inference has $L\times$ higher arithmetic intensity than AR generation-phase inference. On modern GPUs (H100 with 3.35 PF of compute but only 3.35 TB/s bandwidth), the breakeven AI is approximately 1000 FLOPs/byte. AR inference typically achieves AI $\approx 1$–10, leaving $>99\%$ of compute idle. Diffusion inference can achieve AI $\approx L$, much better utilizing the available compute.

Speculative Decoding and Fast Inference

Several techniques have been developed to accelerate diffusion language model inference, drawing inspiration from both the diffusion model literature and the autoregressive speculative decoding paradigm.

Confidence-Aware Parallel (CAP) Decoding

Introduced in LLaDA 2.0 [19], CAP training adds an auxiliary confidence loss that minimizes entropy selectively on correctly predicted tokens: $$\begin{array}{ccc}& ℒ(\theta )={ℒ}_{\text{SFT}}(\theta )+\lambda {ℒ}_{\text{conf}}(\theta ).& \text{<span class="tml-eqn"></span>}\end{array}$$ The confidence loss sharpens the model's probability distributions for tokens it can predict well, enabling more tokens to be unmasked in a single step. This effectively reduces the required number of denoising steps while maintaining accuracy.

With CAP training, LLaDA 2.0-flash achieves 535 tokens/second, a 2.1$\times$ speedup over comparable AR baselines at 237–256 tokens/second.

Adaptive Step Count

Mercury [7] uses a dynamic step count that adjusts based on output complexity:

• Simple structured outputs: $\sim$8 denoising steps

• Complex reasoning: 16–20 denoising steps

This allows the system to automatically navigate the speed/quality trade-off without manual tuning.

Speculative Decoding for Diffusion

In autoregressive models, speculative decoding uses a small “draft” model to propose multiple tokens in parallel, which are then verified by the large model. An analogous approach for diffusion models:

1. A fast, small diffusion model generates a draft sequence in few steps.

2. The large model refines the draft in one or two additional denoising steps.

3. The refinement step leverages the bidirectional context to correct errors in the draft.

This naturally fits the diffusion paradigm because the refinement step is exactly what the model already does: denoising a partially correct sequence is a special case of denoising a partially masked sequence.

Scaling Up: LLaDA 2.0 and Mercury

The most recent frontier has been scaling diffusion language models to compete with state-of-the-art LLMs. Two prominent efforts are LLaDA 2.0 and Mercury.

LLaDA 2.0: From 8B to 100B via AR Conversion

LLaDA 2.0 [19] introduces a key insight: rather than training diffusion language models from scratch (which faces a maturity gap in data and infrastructure compared to AR models), convert pre-trained AR models into diffusion models.

Three-Phase Block-Level Training.

The conversion uses a Warmup-Stable-Decay (WSD) scheme:

1. Warmup: Progressively increase the block size from 1 (fully AR) to 4096 (full sequence MDLM): $1\to 4\to 32\to 64\to 4096$.

2. Stable: Full-sequence MDLM training at block size 4096 for deep bidirectional understanding.

3. Decay: Gradually reduce block size from 4096 back to 32 for efficient inference.

Mixture-of-Experts Architecture.

LLaDA 2.0 uses MoE (Mixture-of-Experts) architectures:

Post-Training Pipeline.

LLaDA 2.0 introduces several innovations for post-training:

1. Complementary masking: Generates two antithetical training instances from a single sequence: one uses a random mask, the other its logical inverse. Every token appears unmasked exactly once per pair, doubling effective data utilization.

2. Mask ratio bandwidth: Clips the noise schedule to $[{\alpha }_{\min },{\alpha }_{\max }]$ instead of $[0,1]$, avoiding extreme masking rates that induce high gradient variance.

3. DPO for diffusion: Adapts Direct Preference Optimization using the ELBO as a surrogate for the intractable log-likelihood: $$\begin{array}{ccc}& {ℒ}_{\text{DPO}}(\theta )=-{𝔼}_{(𝒄,{𝒙}_w,{𝒙}_l)}\left[\log \sigma \left(\beta [\mathrm{\Delta }ℬ({𝒙}_w|𝒄)-\mathrm{\Delta }ℬ({𝒙}_l|𝒄)]\right)\right],& \text{<span class="tml-eqn"></span>}\end{array}$$ where $\mathrm{\Delta }ℬ(𝒙|𝒄)={ℬ}_{\theta }(𝒙|𝒄)-{ℬ}_{{\theta }_{\text{ref}}}(𝒙|𝒄)$ is the difference in ELBO between the current and reference model.

Results.

LLaDA 2.0-flash (100B total, 6.1B active) achieves an overall average of 73.18 across 47 benchmarks, competitive with Qwen3-30B (73.60) and its AR base model Ling-flash-2.0 (72.15). Notable strengths include:

• HumanEval: 94.51% (vs. 93.29% for Qwen3-30B)

• MBPP: 88.29% (vs. 86.65% for Qwen3-30B)

• AIME 2025: 60.00% (competitive with 61.88%)

Mercury: Ultra-Fast Commercial-Scale Diffusion

Mercury [7] from Inception Labs represents the first commercial-scale diffusion language model, designed for maximum throughput.

Architecture.

Mercury uses a standard Transformer architecture; the key difference is the training loss (diffusion denoising instead of next-token prediction) and the generation algorithm (iterative denoising instead of sequential sampling). The architecture choice is explicitly stated to be “orthogonal to the fact that Mercury models are diffusion-based.”

Key Speed Results.

Mercury Coder Mini is approximately 5–18$\times$ faster than comparable models while maintaining competitive quality.

Why So Fast?

The fundamental advantage comes from the diffusion generation paradigm:

1. Higher arithmetic intensity: Each denoising step processes many tokens in parallel, better saturating GPU compute units.

2. Fewer total forward passes: Even with $\sim$10 denoising steps for 100 tokens, that is 10$\times$ fewer forward passes than AR.

3. Proprietary inference engine: Custom kernels optimized for parallel denoising workloads with dynamically batched sampling.

Theoretical Results

The theoretical understanding of discrete diffusion has advanced significantly alongside the empirical progress. We survey key results that provide convergence guarantees and illuminate the connections between discrete and continuous diffusion.

Convergence Bounds for Discrete Diffusion

Equivalence to All-Order Autoregressive

A remarkable theoretical result connects absorbing discrete diffusion to autoregressive modeling:

Proof Sketch.

The key idea is a change of variables from continuous time to the masking probability $\lambda (t)=1-e^{-\bar{\sigma }(t)}=1-{\alpha }_t$. The continuous-time NELBO is: $$\begin{array}{ccc}& ℒ=-{\int }_0^{\infty }{𝔼}_{q({𝒙}_t|{𝒙}_0)}[{\sigma }^{\prime }(t)\sum _{\ell }\mathrm{𝟏}[x_t^{\ell }=[\text{ᴍᴀsᴋ}]]\cdot \frac{\log p_{\theta }(x_0^{\ell }|{𝒙}_t)}{1-{\alpha }_t}]dt.& \text{<span class="tml-eqn"></span>}\end{array}$$ Under the change of variables to $\lambda =1-{\alpha }_t$, the integral over $\lambda \in [0,1]$ can be decomposed combinatorially. At masking level $\lambda$, each subset $S$ of positions is masked with probability: $$\begin{array}{ccc}& \mathrm{Pr}[\text{masked positions}=S]={\lambda }^{|S|}{(1-\lambda )}^{L-|S|}.& \text{<span class="tml-eqn"></span>}\end{array}$$ Integrating over $\lambda$ from 0 to 1 and summing over all possible subsets $S$ (with appropriate combinatorial weighting), the integral naturally decomposes into a sum over all $L!$ permutations. For each permutation $\pi$, the contribution is exactly the negative log-likelihood under the autoregressive factorization in order $\pi$: $$\begin{array}{ccc}& {\int }_0^1\sum _{S\subseteq [L]}\mathrm{Pr}[\text{masked}=S]\sum _{\ell \in S}\frac{-\log p_{\theta }(x_0^{\ell }|{𝒙}_0^{[L]\setminus S})}{|S|}d\lambda =\frac{1}{L!}\sum _{\pi \in S_L}\sum _{i=1}^L-\log p_{\theta }(x_0^{\pi (i)}|{𝒙}_0^{\pi (<i)}).& \text{<span class="tml-eqn"></span>}\end{array}$$ The uniform weighting over permutations emerges naturally from the Beta integral ${\int }_0^1{\lambda }^{k-1}{(1-\lambda )}^{L-k}d\lambda =B(k,L-k+1)=\frac{(k-1)!(L-k)!}{L!}$.

Parallels Between Discrete and Continuous Diffusion

The following table summarizes the deep mathematical parallels:

DPO and Alignment for Diffusion Language Models

Aligning language models with human preferences is critical for safe and useful deployment. While RLHF and DPO are well-established for autoregressive models, adapting them to diffusion models poses unique challenges.

The Challenge: Intractable Log-Likelihood

Standard DPO requires computing the log-likelihood of responses under the model: $$\begin{array}{ccc}& {ℒ}_{\text{DPO}}=-{𝔼}_{(𝒄,{𝒙}_w,{𝒙}_l)}\left[\log \sigma \left(\beta [\log \frac{p_{\theta }({𝒙}_w|𝒄)}{p_{\text{ref}}({𝒙}_w|𝒄)}-\log \frac{p_{\theta }({𝒙}_l|𝒄)}{p_{\text{ref}}({𝒙}_l|𝒄)}]\right)\right].& \text{<span class="tml-eqn"></span>}\end{array}$$

For autoregressive models, $\log p_{\theta }(𝒙|𝒄)={\sum }_i\log p_{\theta }(x^i|{𝒙}^{<i},𝒄)$ is tractable (just sum the per-token log-probabilities). For diffusion models, $\log p_{\theta }(𝒙)$ requires marginalizing over all possible denoising trajectories, which is generally intractable.

ELBO as a Surrogate

LLaDA 2.0 solves this by substituting the ELBO for the log-likelihood. The block diffusion ELBO provides a tractable lower bound: $$\begin{array}{ccc}& {ℬ}_{\text{BDLM}}(\theta ,𝒙|𝒄)={𝔼}_{t,{𝒙}_t}[\frac{{\alpha }_t^{\prime }}{1-{\alpha }_t}\sum _k\sum _i\mathrm{𝟏}[x_{t,k}^i=[\text{ᴍᴀsᴋ}]]\cdot \log p_{\theta }(x_k^i|𝒄,{𝒙}_{<k},{𝒙}_{t,k})].& \text{<span class="tml-eqn"></span>}\end{array}$$

The diffusion DPO loss becomes: $$\begin{array}{ccc}& {ℒ}_{\text{DPO-diff}}(\theta )=-𝔼\left[\log \sigma \left(\beta [\mathrm{\Delta }ℬ({𝒙}_w|𝒄)-\mathrm{\Delta }ℬ({𝒙}_l|𝒄)]\right)\right],& \text{<span class="tml-eqn"></span>}\end{array}$$ where $\mathrm{\Delta }ℬ(𝒙|𝒄)={ℬ}_{\theta }(𝒙|𝒄)-{ℬ}_{{\theta }_{\text{ref}}}(𝒙|𝒄)$.

Complementary Masking for Efficient DPO

Computing the ELBO requires estimating the loss over random masking patterns, which introduces variance. LLaDA 2.0's complementary masking technique is particularly valuable for DPO:

• For each preference pair $({𝒙}_w,{𝒙}_l)$, generate a random mask $𝒎$.

• Compute the loss for both $𝒎$ and its complement $\overline{𝒎}=1-𝒎$.

• Average the two estimates.

This ensures every token appears in its unmasked state exactly once per pair, reducing variance in the ELBO estimate and making the DPO training more stable.

Putting It All Together: A Comparison

We conclude with a comprehensive comparison of the key models discussed in this chapter:

Conclusion and Open Problems

Diffusion language models have progressed from a theoretical curiosity to competitive alternatives to autoregressive LLMs in just a few years. The key insight, that masking-based discrete diffusion is equivalent to weighted masked language modeling, provides both theoretical elegance and practical simplicity. Models like LLaDA, LLaDA 2.0, and Mercury have demonstrated that diffusion models can match AR models on benchmarks while offering unique advantages in parallelism, bidirectionality, and iterative refinement.

Open Problems.

1. Closing the quality gap. While diffusion models are competitive, the largest AR models (GPT-4, Claude) still lead on the hardest reasoning and coding benchmarks. Can diffusion models close this gap?

2. RLHF and alignment. DPO has been adapted for diffusion (via ELBO surrogates), but full RLHF with PPO remains unexplored. How should reward models interact with the iterative denoising process?

3. Length generalization. Diffusion models require specifying the output length in advance. How can we make the output length adaptive without sacrificing the parallel generation advantage?

4. Efficient attention for diffusion. The incompatibility with KV cache is a fundamental limitation. Can we design attention mechanisms that maintain bidirectionality while enabling incremental computation?

5. Multimodal diffusion. Diffusion already excels at images. Can a single diffusion model handle both language and vision in a unified framework?

6. Online / streaming generation. Autoregressive models naturally support streaming (tokens appear as they are generated). Diffusion models produce all tokens at once at the end. Can diffusion models support streaming via progressive unmasking with partial output?

7. Reasoning with diffusion. The iterative refinement of diffusion is reminiscent of the “think, draft, revise” process of reasoning. Can this be formalized into a chain-of-thought-like framework for diffusion models?

8. Theoretical optimality. Is there a fundamental information-theoretic advantage to diffusion over autoregressive generation for certain distribution classes? Or are the advantages purely computational?

Continuous Diffusion Approaches for Text: What Went Wrong?

Before discrete diffusion models succeeded, several attempts were made to apply continuous diffusion to text. Understanding why these approaches fell short provides useful context.

Diffusion-LM

Diffusion-LM [24] operates by embedding each token into a continuous vector space and then applying standard Gaussian diffusion to the sequence of embeddings. The forward process adds Gaussian noise to the embeddings: $$\begin{array}{ccc}& q({𝒛}_t|{𝒛}_0)=𝒩({𝒛}_t;\sqrt{{\bar{\alpha }}_t}{𝒛}_0,(1-{\bar{\alpha }}_t)𝐈),& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${𝒛}_0=(\text{Embed}(x_0^1),\dots ,\text{Embed}(x_0^L))\in {\mathrm{ℝ}}^{L\times d}$ is the sequence of token embeddings.

The Rounding Problem.

After denoising, the continuous output ${\widehat{𝒛}}_0$ must be mapped back to discrete tokens. Diffusion-LM uses a learned rounding function: $$\begin{array}{ccc}& {\hat{x}}^i={\mathrm{arg\hspace{0.17em}min}}_{v\in 𝒱}\Vert {\hat{z}}_0^i-\text{Embed}(v){\Vert }_2.& \text{<span class="tml-eqn"></span>}\end{array}$$ This nearest-neighbor rounding introduces errors because:

1. Small errors in the continuous space can map to completely different tokens.

2. The embedding space is high-dimensional, and nearest-neighbor search is noisy.

3. Errors compound across the sequence, and each rounding error changes the context for other tokens.

Results.

On LM1B, Diffusion-LM achieved perplexity $\le 118.62$, dramatically worse than the discrete approaches (MDLM: $\le 23.00$). The continuous relaxation fundamentally loses the discrete structure of language.

DiffuSeq and Continuous Text Generation

DiffuSeq [25] applied continuous diffusion to sequence-to-sequence tasks (machine translation, summarization). While achieving reasonable results on these constrained tasks, it still suffered from the rounding problem and was significantly slower than autoregressive baselines.

Lessons Learned

The failure of continuous approaches and the success of discrete ones teach an important lesson:

This lesson extends to other discrete domains. DNA sequences, protein sequences, and molecular graphs all benefit from discrete diffusion approaches rather than continuous relaxations. MDLM demonstrated competitive results on DNA modeling, confirming the broad applicability of the discrete framework.

Applications Beyond Standard Text Generation

Diffusion language models have unique properties that make them particularly well-suited for certain applications.

Code Generation and Completion

Code has natural “fill-in-the-middle” structure: developers often write function signatures and docstrings before implementations. Diffusion models excel at this because:

• Bidirectional context: The model sees both the function signature (above) and the test cases (below) when filling in the implementation.

• Iterative refinement: Complex code can be drafted and refined, allowing the model to ensure consistency across the implementation.

• Native FIM: No special training is needed, as fill-in-the-middle is the natural operating mode.

Mercury Coder's strong performance on coding benchmarks (HumanEval: 90.0%, MBPP: 76.6% for Small variant) validates this advantage.

Controlled and Constrained Generation

Diffusion models naturally support constrained generation: fixing certain tokens and generating the rest. For example:

• Template filling: Given a template “Dear [NAME], Thank you for your [REASON]. We will [ACTION].” the diffusion model fills in the blanks using bidirectional context.

• Lexically constrained generation: Requiring specific keywords to appear in the output can be enforced by initializing those tokens as unmasked.

• Style transfer: Keeping content words fixed while regenerating function words can change the style of text while preserving meaning.

Biological Sequence Design

MDLM has been applied to DNA sequence modeling with competitive results. The masking-based approach is natural for biological sequences because:

• Biological sequences are discrete (nucleotides or amino acids).

• The function of a position depends on both flanking regions (bidirectional context).

• Protein engineering often involves fixing certain regions (e.g., active sites) while optimizing others, naturally expressed as constrained diffusion.

Machine Translation and Editing

Non-autoregressive translation (NAT) is a long-standing goal in machine translation. Diffusion models provide a principled framework for NAT:

1. Encode the source sentence $𝒔$ as context.

2. Initialize the target as fully masked: ${𝒓}_1=([\text{ᴍᴀsᴋ}],\dots ,[\text{ᴍᴀsᴋ}])$.

3. Run the reverse diffusion process conditioned on $𝒔$.

This allows the translation to be generated in parallel, with iterative refinement ensuring fluency and accuracy. The bidirectional context is particularly valuable for translation, where word order may differ between source and target languages.

Historical Timeline and Development

The development of diffusion language models has been remarkably rapid. We present a timeline highlighting the key milestones:

Exercises