21 Audio and Music Generation

Why Audio Generation?

Consider a ten-second clip of a jazz trio: a piano outlines a chord progression, a bass walks through the changes, and brushes dance on a snare drum. The corresponding digital waveform contains roughly 441,000 samples (at 44.1,kHz), each carrying amplitude information at 16-bit resolution. To a generative model, this clip is a point in ${\mathrm{ℝ}}^{\mathrm{441,000}}$, and the space of perceptually valid jazz trio recordings occupies a vanishingly thin manifold within that vast ambient space. The challenge of audio generation is to learn the structure of this manifold well enough to draw new points from it on demand.

Why should we care? The practical applications are abundant. Text-to-speech systems convert written language to natural sounding utterances, powering virtual assistants and accessibility tools. Text-to-music models compose novel audio given natural-language prompts such as “a melancholic cello solo over soft rain.” Sound effect generation synthesises Foley audio for film and game production. Audio inpainting fills gaps in corrupted recordings, and voice conversion transforms the timbre of a speaker while preserving linguistic content. These tasks share a common mathematical substrate: learning a conditional or unconditional distribution over waveforms and sampling from it efficiently.

The Dimensionality of Sound

The raw dimensionality of audio signals dwarfs that of the images we encountered in earlier chapters, yet remains below the extreme scales of video. tab:audiogen:dimensionality makes this concrete.

A single second of CD-quality mono audio contains 44,100 floating-point values. A three-minute pop song therefore lives in a space of dimension $d\approx 7.9\times {10}^6$; in stereo, this doubles to $1.6\times {10}^7$. It is duration, not instantaneous richness, that does the damage: a ten-second music clip ($4.4\times {10}^5$) is actually a little smaller than a single $512\times 512$ RGB image ($7.9\times {10}^5$), but the three-minute song is a full order of magnitude larger, and its stereo version twenty times larger. An image generator is asked for one such object; a music generator is asked for a signal that keeps being coherent for minutes.

A Brief History of Audio Generation

The history of neural audio generation can be divided into four overlapping eras, each defined by the dominant generative paradigm. We summarise them here to provide context for the technical sections that follow; the reader should consult the original papers for full details.

Era I: Concatenative and parametric synthesis (pre-2016).

Classical text-to-speech systems either concatenated recorded speech segments from large databases or drove parametric vocoders with hand-engineered acoustic features. The quality ceiling was unmistakable: concatenative systems sounded choppy at segment boundaries, and parametric vocoders produced a characteristic “buzzy” timbre that listeners found unnatural.

Era II: Autoregressive waveform models (2016–2019).

DeepMind's WaveNet [13] demonstrated that an autoregressive model predicting one sample at a time could produce strikingly natural speech. The model factorised the joint distribution as (Wavenet Factor)$$\begin{array}{ccc}& p(𝒙)=\prod _{n=1}^{L}p(x_n|x_1,\dots ,x_{n-1}),& \text{<span class="tml-eqn"></span>}\end{array}$$ with each conditional implemented by a dilated causal convolution stack. The catch was speed: generating one second of 16,kHz audio required 16,000 sequential forward passes. Parallel WaveNet [14] and WaveRNN [15] addressed inference cost, but the autoregressive bottleneck remained a theme.

Era III: GAN and diffusion vocoders (2019–2022).

GAN-based vocoders such as MelGAN [16], HiFi-GAN [17], and UnivNet [18] replaced autoregressive generation with feed-forward synthesis, achieving real-time waveform generation on a single GPU\@. Concurrently, diffusion-based vocoders (DiffWave [19], WaveGrad [20]) applied the denoising diffusion framework of 19 to the audio domain. These models serve as the “decoder” in many two-stage generation pipelines.

Era IV: Foundation models for audio (2023–present).

The convergence of large language models, neural audio codecs, and latent diffusion has produced a new generation of systems capable of generating high-fidelity, long-form audio from text prompts. AudioLDM [21] applies latent diffusion to mel-spectrograms. MusicGen [22] generates music autoregressively over the discrete tokens of an EnCodec codec. Stable Audio [23] operates a latent diffusion transformer in the continuous latent space of a variational autoencoder. Commercial systems such as Suno and Udio generate full songs with vocals, approaching (and sometimes passing) human-level quality in listener studies.

Chapter Roadmap

This chapter develops the mathematical and architectural foundations of modern audio generation systems. The narrative is organised into a progression from signal-level building blocks to complete generation pipelines.

1. Audio signal processing (Audio Signal Processing Foundations). We formalise the Short-Time Fourier Transform, the mel scale, and the mel-spectrogram, establishing the standard input representations used throughout the chapter.

2. Neural audio vocoders (Neural Audio Vocoders). We study HiFi-GAN and related architectures that convert mel-spectrograms back into high-fidelity waveforms, connecting their discriminator designs to the GAN theory of 9.

3. Neural audio codecs (Neural Audio Codecs). We develop Residual Vector Quantisation and its application in SoundStream, EnCodec, and DAC, which compress audio into sequences of discrete tokens amenable to language-model-style generation.

4. CLAP and contrastive audio–text alignment (CLAP: Contrastive Language–Audio Pretraining). We formalise the contrastive loss that learns a joint embedding space for audio and text, enabling text-conditioned generation.

5. Autoregressive generation over codec tokens (later sections). We study MusicLM and MusicGen, which treat audio generation as a sequence modelling problem over discrete codec codes.

6. Latent diffusion and flow matching for audio (later sections). We develop AudioLDM, Stable Audio, and related systems that apply continuous generative models in compressed latent spaces.

7. Evaluation, ethics, and open problems (later sections). We discuss Fréchet Audio Distance, perceptual metrics, watermarking, and the societal implications of generative audio.

Throughout, we assume familiarity with the fundamentals of generative adversarial networks (9), variational autoencoders (15), diffusion models (19), and normalising flows (18). Readers seeking background on these topics should consult the corresponding chapters before proceeding.

Audio Signal Processing Foundations

Before we can generate audio, we must understand how audio is represented, transformed, and compressed. This section develops the mathematical machinery that every subsequent section relies upon: the Short-Time Fourier Transform (STFT), the mel frequency scale, and the mel-spectrogram. These are not mere engineering details; they encode deep facts about human perception and signal structure that profoundly shape the design of generative models.

Digital Audio Signals

A continuous-time audio signal is a real-valued function of time, $s:\mathrm{ℝ}\to \mathrm{ℝ}$. Digital audio is obtained by sampling $s$ at a fixed rate $f_s$ (the sampling rate or sample rate), producing a discrete sequence (Sampling)$$\begin{array}{ccc}& x[n]=s\left(\frac{n}{f_s}\right),n=0,1,\dots ,L-1,& \text{<span class="tml-eqn"></span>}\end{array}$$ where $L=\lfloor T\cdot f_s\rfloor$ is the total number of samples and $T$ is the signal duration in seconds. We collect these samples into a vector $𝒙={(x[0],x[1],\dots ,x[L-1])}^{\top }\in {\mathrm{ℝ}}^L$.

Standard sample rates include 8,kHz (telephony), 16,kHz (speech processing), 22.05,kHz (compressed music), 44.1,kHz (CD-quality), and 48,kHz (professional audio and video). The choice of $f_s$ determines both the bandwidth and the dimensionality of the generative modelling problem.

The Short-Time Fourier Transform

The Fourier transform reveals the frequency content of a signal, but it provides no information about when those frequencies occur. For audio generation, temporal localisation is essential: a piano note that begins at time $t=2$,s and ends at $t=3$,s must be placed correctly in the output waveform. The Short-Time Fourier Transform (STFT) resolves this tension by applying the Fourier transform to short, overlapping windows of the signal.

Each column of $𝐗$ is the discrete Fourier transform (DFT) of a windowed segment of $𝒙$. The window function $w$ (commonly Hann, Hamming, or Blackman) tapers the segment smoothly to zero at its edges, suppressing spectral leakage from discontinuities at the segment boundaries.

The STFT is invertible (under mild conditions on the window), which is crucial: a vocoder must be able to reconstruct a waveform from a modified spectrogram. The inverse STFT (iSTFT) recovers the signal by applying the inverse DFT to each frame, multiplying by a synthesis window, and overlap-adding the results.

For the inverse to be exact, the analysis and synthesis windows must satisfy a completeness condition.

The STFT decomposes the signal into three informative representations, each useful in different generative contexts.

The Mel Scale and Mel-Spectrogram

The linear frequency axis of the STFT does not match human auditory perception. We perceive pitch differences on a roughly logarithmic scale: the perceptual distance between 100,Hz and 200,Hz (one octave) is much greater than that between 5,000,Hz and 5,100,Hz, even though the absolute difference in hertz is the same. The mel scale provides a frequency warping that better reflects this perceptual nonlinearity.

To construct a mel-spectrogram, we apply a bank of triangular filters, spaced uniformly on the mel scale, to the power (or magnitude) spectrogram.

fig:audiogen:mel-filterbank illustrates the mel filterbank and the pipeline from waveform to mel-spectrogram.

Representation Taxonomy

Modern audio generation systems operate on one of several representations, each offering a different trade-off between reconstruction fidelity, dimensionality, and suitability for generative modelling. We classify the main options here for reference; subsequent sections develop each in detail.

1. Raw waveform ($𝒙\in {\mathrm{ℝ}}^L$). The highest-fidelity representation, but extremely high-dimensional and temporally fine-grained. Direct waveform models (WaveNet, SampleRNN) operate here but face severe computational costs.

2. Mel-spectrogram (${𝐒}_{\log }\in {\mathrm{ℝ}}^{F_{\mathrm{mel}}\times M}$). A compressed, perceptually motivated 2D representation. Since $M\approx L/H$, the number of stored values falls by a factor of $H/F_{\mathrm{mel}}$ relative to the waveform (for $H=320$ and $F_{\mathrm{mel}}=80$, a factor of 4), but reconstruction requires a vocoder to estimate phase and fine spectral detail.

3. Discrete codec tokens ($𝒄\in {\{1,\dots ,V\}}^{Q\times T_c}$). A sequence of integer codes produced by a neural audio codec (SoundStream, EnCodec, DAC) with $Q$ quantisation levels and $T_c$ time frames. This is the most compressed representation, enabling language-model-style generation, but introduces quantisation artefacts.

4. Continuous latents ($𝒛\in {\mathrm{ℝ}}^{d\times T_z}$). The bottleneck representation of a continuous (non-quantised) autoencoder. Latent diffusion and flow matching models operate here, combining the compression benefits of codecs with the smoothness of continuous spaces.

The ordering deserves comment, because it is not the one a reader might expect. A continuous codec latent is less compressed than a mel-spectrogram in this configuration - 96,000 numbers against 60,000 - and yet it is the better representation to generate in. Compression and modelling difficulty are two different axes, and the rest of this chapter is largely the story of how they came apart.

Neural Audio Vocoders

A vocoder is a system that converts a compact acoustic representation (classically, linear predictive coding parameters; in modern systems, a mel-spectrogram or similar feature) into a waveform. In the context of audio generation, the vocoder is the final stage that transforms a generated spectrogram into an audible signal. Its quality sets the ceiling for the overall system: a perfect generative model paired with a poor vocoder will produce poor audio.

Neural vocoders have evolved rapidly since WaveNet first demonstrated that a neural network could synthesise perceptually natural waveforms. In this section, we focus on HiFi-GAN [17], which has become the de facto standard vocoder in modern audio generation pipelines due to its combination of high fidelity, fast inference, and training stability. We develop its architecture in detail, connecting its multi-scale and multi-period discriminator designs to the GAN theory developed in 9.

The Vocoding Problem

Formally, the vocoding problem is a conditional generation task.

HiFi-GAN: Generator Architecture

The HiFi-GAN generator is a fully convolutional feed-forward network that progressively upsamples the mel-spectrogram to the waveform sampling rate through a cascade of transposed convolution blocks, each followed by a multi-receptive-field fusion (MRF) module.

Multi-Period and Multi-Scale Discriminators

The adversarial training of HiFi-GAN relies on two complementary discriminator families, each designed to capture different structural properties of audio waveforms. The design of these discriminators reflects a deep understanding of audio signal structure and connects directly to the GAN theory developed in 9 and the Wasserstein framework of 10.

HiFi-GAN Training Losses

The HiFi-GAN training objective combines adversarial losses, feature matching losses, and a spectral reconstruction loss. We develop each component and then state the full training procedure.

Adversarial losses.

Each sub-discriminator $D_k$ (from both MPD and MSD) is trained with the least-squares GAN (LSGAN) objective (see 9): (Hifigan ADV D)$$\begin{array}{cccc}& {ℒ}_{\text{adv}}^{(D_k)}& ={𝔼}_{𝒙}\left[{(D_k(𝒙)-1)}^2\right]+{𝔼}_{\widehat{𝒙}}\left[D_k{(\widehat{𝒙})}^2\right],& \text{<span class="tml-eqn"></span>}\\ & {ℒ}_{\text{adv}}^{(G,D_k)}& ={𝔼}_{\widehat{𝒙}}\left[{(D_k(\widehat{𝒙})-1)}^2\right].& \text{<span class="tml-eqn"></span>}\end{array}$$ The total adversarial loss sums over all sub-discriminators: (Hifigan ADV Total)$$\begin{array}{ccc}& {ℒ}_{\text{adv}}^{(D)}=\sum _k{ℒ}_{\text{adv}}^{(D_k)},{ℒ}_{\text{adv}}^{(G)}=\sum _k{ℒ}_{\text{adv}}^{(G,D_k)}.& \text{<span class="tml-eqn"></span>}\end{array}$$

Feature matching loss.

For each sub-discriminator $D_k$ with $N_k$ intermediate layers, let $D_k^{(l)}(𝒙)\in {\mathrm{ℝ}}^{d_l}$ denote the activation at layer $l$. The feature matching loss encourages the generator to produce waveforms whose internal discriminator representations match those of real audio: (Hifigan FM)$$\begin{array}{ccc}& {ℒ}_{\text{fm}}=\sum _k\sum _{l=1}^{N_k}\frac{1}{d_l}𝔼\left[{\Vert D_k^{(l)}(𝒙)-D_k^{(l)}(\widehat{𝒙})\Vert }_1\right].& \text{<span class="tml-eqn"></span>}\end{array}$$ This provides a stable, multi-scale gradient signal to the generator, analogous to the perceptual loss used in image super-resolution.

Multi-resolution STFT loss.

To accelerate early training and improve fine spectral detail, the generator is additionally supervised with a reconstruction loss computed in the STFT domain at multiple resolutions.

Full training objective.

The generator is trained to minimise (Hifigan FULL)$$\begin{array}{ccc}& {ℒ}_G={ℒ}_{\text{adv}}^{(G)}+{\lambda }_{\text{fm}}{ℒ}_{\text{fm}}+{\lambda }_{\text{rec}}{ℒ}_{\text{rec}},& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${ℒ}_{\text{rec}}$ is the spectral reconstruction term. The original HiFi-GAN takes ${ℒ}_{\text{rec}}={ℒ}_{\text{mel}}$ with ${\lambda }_{\text{fm}}=2$ and ${\lambda }_{\text{mel}}=45$; that 45 is the mel loss weight, consistent with Remark 13, and it should not be carried over to a multi-resolution STFT term. Codec systems keep the same three-term structure but substitute ${ℒ}_{\text{rec}}={ℒ}_{\text{STFT}}$, retuning the weight, which is typically an order of magnitude smaller because the multi-resolution STFT loss is already normalised by signal energy. The discriminators are trained to minimise ${ℒ}_{\text{adv}}^{(D)}$. Training alternates between discriminator and generator updates, following the standard GAN training protocol of 9.

Neural Audio Codecs

The mel-spectrogram is a useful intermediate representation, but it is continuous and high-dimensional (typically $80\times M$ floats). For autoregressive generation with language models, we need a discrete representation: a sequence of tokens from a finite vocabulary, analogous to the word tokens in text. Neural audio codecs provide exactly this, compressing audio into sequences of discrete codes while maintaining high reconstruction quality.

The key innovation that makes neural audio codecs practical is Residual Vector Quantisation (RVQ), which achieves high codebook utilisation and fine-grained quantisation by applying multiple rounds of vector quantisation to the residuals of previous rounds. In this section, we develop RVQ mathematically, then examine three influential codec architectures: SoundStream, EnCodec, and DAC\@.

Vector Quantisation Revisited

We briefly recall the essentials of vector quantisation (VQ) before developing the residual extension. The reader may consult 15 for a thorough treatment of VQ-VAE foundations.

A single vector quantiser with codebook size $V$ can represent at most $V$ distinct vectors, yielding ${\log }_{2}V$ bits per frame. For $V=1024$ (a common choice), this is only 10 bits per frame. At a codec frame rate of 75 frames per second, the resulting bitrate is 750 bits per second, far too low for high-fidelity audio reconstruction. Naively increasing $V$ to, say, $2^{20}$ would provide 20 bits per frame but would make codebook learning intractable: most entries would never be used, and the straight-through gradient estimator would become unreliable.

Residual Vector Quantisation

Residual Vector Quantisation (RVQ) elegantly solves this problem by decomposing the quantisation into a cascade of $Q$ stages, each quantising the residual left by the previous stages.

SoundStream

SoundStream [10] was the first neural audio codec to combine a convolutional encoder-decoder with RVQ, establishing the architectural template that EnCodec and DAC would later refine.

SoundStream is trained with a combination of reconstruction and adversarial losses: (Soundstream LOSS)$$\begin{array}{ccc}& {ℒ}_{\text{SS}}={\lambda }_{\text{recon}}{ℒ}_{\text{recon}}+{\lambda }_{\text{adv}}{ℒ}_{\text{adv}}+{\lambda }_{\text{fm}}{ℒ}_{\text{fm}}+{\lambda }_{\text{vq}}{ℒ}_{\text{vq}},& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${ℒ}_{\text{recon}}$ is a multi-resolution STFT loss (Definition 12), ${ℒ}_{\text{adv}}$ and ${ℒ}_{\text{fm}}$ are adversarial and feature matching losses from a wave-based discriminator, and ${ℒ}_{\text{vq}}$ is the codebook commitment loss: (VQ LOSS)$$\begin{array}{ccc}& {ℒ}_{\text{vq}}=\sum _{q=1}^Q({\Vert \mathrm{sg}[{𝒓}_{q-1}]-{\widehat{𝒓}}_q\Vert }_2^2+\beta {\Vert {𝒓}_{q-1}-\mathrm{sg}[{\widehat{𝒓}}_q]\Vert }_2^2),& \text{<span class="tml-eqn"></span>}\end{array}$$ with $\mathrm{sg}[\cdot ]$ denoting the stop-gradient operator and $\beta$ the commitment cost weight.

EnCodec

EnCodec [24] refines the SoundStream architecture with several improvements: a higher-capacity encoder-decoder, a novel loss balancing mechanism, and support for variable bitrates through codebook dropout.

The loss balancer deserves special attention, as it addresses a practical difficulty in training multi-loss systems.

DAC: Descript Audio Codec

The Descript Audio Codec (DAC) [25] pushes neural codec quality further through three innovations: improved codebook learning, the Snake activation function, and a multi-band discriminator.

Codec Comparison

tab:audiogen:codec-comparison summarises the key design choices across the three major neural audio codecs.

The “variable bitrate” row is worth pausing on, because it is easily misread as a minor feature. SoundStream's headline contribution was quantizer dropout: train with a random number of active RVQ stages, and a single set of weights then serves every rate from $Q=4$ (3,kbps) to $Q=24$ (18,kbps), chosen at inference time. EnCodec adopted the same idea. This is what makes the $Q$ column and the bitrate column consistent with one another - a single fixed-rate codec could not span a range at all - and it is also why a codec's advertised bitrate is a deployment choice rather than an architectural constant.

The RVQ Token Structure

The output of an RVQ codec is not a single flat sequence of tokens but a matrix of codes $𝐂\in {\{1,\dots ,V\}}^{Q\times T_c}$, where $Q$ is the number of quantisation stages and $T_c$ is the number of time frames. This 2D structure poses a fundamental challenge for autoregressive models, which operate on 1D sequences.

There are three principal strategies for linearising the code matrix:

Flat pattern.

Interleave all codes in raster order: $(c_{\mathrm{1,1}},c_{\mathrm{2,1}},\dots ,c_{Q,1},c_{\mathrm{1,2}},c_{\mathrm{2,2}},\dots )$. This yields a sequence of length $Q\cdot T_c$ and preserves all dependencies but requires the model to generate $Q$ tokens per audio frame, which is slow.

Delay pattern.

Introduced by MusicGen [22], the delay pattern offsets each quantisation level by one time step: level $q$ at time $t$ is placed at sequence position $t+q-1$. This allows parallel prediction of codes at different levels for the same time step, reducing the effective sequence length to $T_c+Q-1$ while maintaining causal ordering.

Coarse-to-fine.

First generate all $T_c$ codes at level $q=1$ (the coarsest level), then all codes at level $q=2$ conditioned on level 1, and so on. This hierarchical approach (used by MusicLM and VALL-E) decouples the coarse acoustic structure from the fine detail but requires $Q$ sequential passes through the model.

Training Neural Audio Codecs: A Unified View

All three codecs (SoundStream, EnCodec, DAC) share a common training framework that combines four families of loss functions. We consolidate these here to provide a unified perspective.

From Codecs to Generation

With the codec machinery in place, we can now see how the full audio generation pipeline works in practice. The two-stage paradigm, adopted by virtually all modern systems, is straightforward:

The sections that follow will develop each generative paradigm in detail: contrastive pretraining for conditioning (CLAP: Contrastive Language–Audio Pretraining), autoregressive generation over codec tokens, latent diffusion over continuous representations, and flow matching for efficient synthesis. In each case, the codec (or, alternatively, a continuous autoencoder) provides the compressed representation that makes generation tractable.

Exercises

CLAP: Contrastive Language–Audio Pretraining

Images have CLIP; audio has CLAP\@. The Contrastive Language–Audio Pretraining framework learns a joint embedding space in which audio signals and their textual descriptions are aligned, enabling zero-shot audio classification, text-to-audio retrieval, and, most importantly for this chapter, conditioning signals for generative models. The conceptual parallel with CLIP is deliberate: both methods train dual encoders with an InfoNCE-style contrastive objective, and both produce embeddings that transfer broadly to downstream tasks.

This section formalises the CLAP objective, describes the encoder architectures, and explains why CLAP embeddings have become the default conditioning representation in modern text-to-audio systems. For a thorough treatment of contrastive losses and divergence measures, we refer the reader to 4.

The Audio–Text Alignment Problem

Consider a dataset of $N$ paired examples ${\{({𝒙}_i^{(a)},{𝒙}_i^{(t)})\}}_{i=1}^N$, where ${𝒙}_i^{(a)}\in {\mathrm{ℝ}}^L$ is an audio waveform (or its mel-spectrogram representation ${𝐒}_i\in {\mathrm{ℝ}}^{T_s\times F_s}$) and ${𝒙}_i^{(t)}$ is a natural language caption describing the audio content. The audio and text modalities live in fundamentally different spaces: the audio is a continuous, high-dimensional signal with temporal structure, while the text is a discrete sequence of tokens with compositional semantics. The alignment problem asks us to learn a shared representation in which semantically matched audio–text pairs are close and unmatched pairs are far apart.

The cosine similarity between an audio embedding and a text embedding is (CLAP SIM)$$\begin{array}{ccc}& s_{ij}=\frac{1}{\tau }⟨{𝒆}_i^{(a)},{𝒆}_j^{(t)}⟩=\frac{1}{\tau }\frac{⟨f_{\theta }({𝐒}_i),g_{\varphi }({𝒙}_j^{(t)})⟩}{\Vert f_{\theta }({𝐒}_i)\Vert \Vert g_{\varphi }({𝒙}_j^{(t)})\Vert },& \text{<span class="tml-eqn"></span>}\end{array}$$ where the temperature $\tau$ controls the sharpness of the distribution over similarities. Within a mini-batch of $B$ pairs, we form the $B\times B$ similarity matrix ${𝐒}_{\text{sim}}\in {\mathrm{ℝ}}^{B\times B}$ with entries $s_{ij}$.

Expanding the two terms, we recognise them as the negative log-probability of the correct match under two softmax distributions: (CLAP A2T)$$\begin{array}{cccc}& {ℒ}_{\text{a}\to \text{t}}& =-\frac{1}{B}\sum _{i=1}^B\log \frac{\exp (s_{ii})}{{\sum }_{j=1}^B\exp (s_{ij})},& \text{<span class="tml-eqn"></span>}\\ & {ℒ}_{\text{t}\to \text{a}}& =-\frac{1}{B}\sum _{i=1}^B\log \frac{\exp (s_{ii})}{{\sum }_{j=1}^B\exp (s_{ji})}.& \text{<span class="tml-eqn"></span>}\end{array}$$ Both terms are instances of the categorical cross-entropy, where the target is the identity permutation: sample $i$ should match sample $i$.

This is a definition, not a result. Each summand is a cosine similarity between unit vectors, so by Cauchy–Schwarz it lies in $[-1,1]$, and therefore so does their mean; the range is an immediate consequence of ${\ell }_2$ normalisation and says nothing about how well the score tracks perceptual alignment. The score attains $1$ only when every generated audio embedding coincides with its text embedding, and is $0$ when the paired embeddings are orthogonal in the joint space. Orthogonality of one pair of vectors is a geometric statement; it is not the same as the two embeddings being statistically uncorrelated, which would be a claim about an expectation over a distribution of pairs. The same quantity is restated for evaluation purposes in Definition 66, where it is normalised per pair rather than per test set.

Audio and Text Encoders

The two arms of the CLAP architecture can be instantiated with a variety of backbone networks. The key design constraint is that both encoders must produce embeddings of the same dimension $d$, so that cosine similarity is well defined.

Audio encoder.

The audio encoder operates on mel-spectrogram inputs $𝐒\in {\mathrm{ℝ}}^{T_s\times F_s}$. Several backbone choices have been explored in the literature:

1. CNN14 (from PANNs): A 14-layer convolutional network originally designed for audio tagging. It processes the mel-spectrogram through a sequence of convolutional blocks with batch normalisation and ReLU activations, followed by global average pooling to produce a fixed-dimensional vector ${𝒉}^{(a)}\in {\mathrm{ℝ}}^{d_a}$.

2. HTS-AT (Hierarchical Token-Semantic Audio Transformer): A Swin-Transformer-based architecture that treats the mel-spectrogram as a 2D “image” and applies hierarchical windowed self-attention. The [CLS] token output serves as the audio representation ${𝒉}^{(a)}\in {\mathrm{ℝ}}^{d_a}$.

3. HTSAT-BERT hybrid: A variant that combines convolutional feature extraction at early layers with Transformer-based global attention at later layers, seeking to capture both local spectral patterns and long-range temporal dependencies.

After the backbone produces ${𝒉}^{(a)}\in {\mathrm{ℝ}}^{d_a}$, a learned linear projection maps it to the shared embedding dimension: (CLAP Audio PROJ)$$\begin{array}{ccc}& {𝒆}^{(a)}=\frac{{𝐖}_a{𝒉}^{(a)}+{𝒃}_a}{\Vert {𝐖}_a{𝒉}^{(a)}+{𝒃}_a\Vert },{𝐖}_a\in {\mathrm{ℝ}}^{d\times d_a},{𝒃}_a\in {\mathrm{ℝ}}^d.& \text{<span class="tml-eqn"></span>}\end{array}$$ The ${\ell }_2$ normalisation ensures that all audio embeddings lie on the unit hypersphere ${𝕊}^{d-1}$.

Text encoder.

The text encoder processes a tokenised caption ${𝒙}^{(t)}=(w_1,w_2,\dots ,w_L)$ through a pre-trained language model. Common choices include:

1. BERT (Bidirectional Encoder Representations from Transformers): The [CLS] token representation from the final Transformer layer is used as ${𝒉}^{(t)}\in {\mathrm{ℝ}}^{d_t}$.

2. RoBERTa: An optimised BERT variant with improved pre-training procedures. Like BERT, it produces a [CLS] representation that is projected to the shared space.

3. GPT-2: For autoregressive text encoders, the embedding of the final token (or a special [EOS] token) serves as the sequence-level representation.

As with the audio side, a linear projection and normalisation map the text representation to the shared space: (CLAP TEXT PROJ)$$\begin{array}{ccc}& {𝒆}^{(t)}=\frac{{𝐖}_t{𝒉}^{(t)}+{𝒃}_t}{\Vert {𝐖}_t{𝒉}^{(t)}+{𝒃}_t\Vert },{𝐖}_t\in {\mathrm{ℝ}}^{d\times d_t},{𝒃}_t\in {\mathrm{ℝ}}^d.& \text{<span class="tml-eqn"></span>}\end{array}$$

Training Data and Applications

The quality of a CLAP model depends critically on the diversity and scale of its training data. Unlike the image domain, where billions of image–text pairs can be scraped from the web, audio–text pairs are substantially scarcer.

Datasets.

The principal audio-captioning datasets used for CLAP training include:

The WavCaps dataset represents an interesting strategy: rather than paying human annotators, the creators used a large language model (ChatGPT) to generate captions from audio metadata and weak labels. This approach trades annotation precision for scale, a trade-off that has proven effective for contrastive pre-training where the loss is robust to moderate label noise.

CLAP for conditioning.

In text-to-audio generation systems, CLAP embeddings serve as the conditioning signal that steers the generative process. Given a text prompt ${𝒙}^{(t)}$ at inference time, the pre-trained text encoder produces ${𝒆}^{(t)}=g_{\varphi }({𝒙}^{(t)})/\Vert g_{\varphi }({𝒙}^{(t)})\Vert$, which is then injected into the generative model via cross-attention or FiLM layers (see Conditioning Mechanisms for Audio Generation). This approach decouples the text understanding from the audio generation: the CLAP encoder “understands” the text, and the generative model “creates” the audio.

CLAP for evaluation.

As formalised in Definition 23, the CLAP score measures semantic alignment between generated audio and conditioning text. It has become a standard evaluation metric alongside the Fréchet Audio Distance (FAD) and the Kullback–Leibler divergence over audio event labels. While FAD measures distributional fidelity (how realistic the generated audio sounds), the CLAP score measures prompt adherence (how well the generated audio matches the textual description).

Autoregressive Music Generation

Diffusion models have dominated recent text-to-audio generation, but an equally powerful paradigm treats audio generation as a language modelling problem. The idea is deceptively simple: tokenise the audio signal into a sequence of discrete codes using a neural codec (see Neural Audio Codecs), then train a Transformer to predict these codes autoregressively, one token at a time. The approach inherits the full machinery of large language models: causal attention, nucleus sampling, KV-cache acceleration, and instruction-following via cross-attention to text encoders.

This section traces the evolution from early approaches such as Jukebox to the modern MusicGen framework, focusing on the mathematical challenge of modelling multi-codebook sequences and the computational trade-offs inherent in autoregressive audio generation.

From Jukebox to Modern Autoregressive Models

The Jukebox model (Dhariwal et al., 2020) was a landmark demonstration that autoregressive Transformers could generate minutes of coherent music with vocals, instruments, and stylistic variation. As discussed in Case Study: Jukebox and Music Generation, Jukebox used a hierarchical VQ-VAE to compress raw audio waveforms into discrete tokens at multiple temporal resolutions, then trained separate autoregressive Transformers at each level of the hierarchy.

While Jukebox proved the concept, it suffered from severe practical limitations. Generating a single minute of music required hours of computation on high-end GPUs, because the VQ-VAE tokens operated at relatively high temporal rates (e.g., one token per $8$ waveform samples at the finest level), producing extremely long sequences. The hierarchical generation scheme also introduced error accumulation: artefacts at coarse levels propagated and amplified through finer levels.

The shift to neural codec tokens (from EnCodec, SoundStream, and related models) resolved much of this inefficiency. Neural codecs compress audio far more aggressively than the VQ-VAE approach used by Jukebox, producing token rates of $50$–$75$ tokens per second at each codebook level rather than thousands of tokens per second. However, neural codecs introduce a new challenge: the residual vector quantisation (RVQ) scheme produces multiple parallel codebooks, and the autoregressive model must decide how to serialise these multi-codebook tokens into a single sequence.

The Codebook Flattening Problem

Let $𝐂\in {\{1,\dots ,V\}}^{Q\times T}$ denote the token matrix produced by a neural codec with $T$ time steps, $Q$ codebook levels and codebook size $V$. The entry $c_t^{(q)}$ is the discrete code assigned to time step $t$ at codebook level $q$. The autoregressive model must define a total ordering on these $T\times Q$ tokens. Three principal patterns have emerged, each with distinct trade-offs in sequence length, modelling fidelity, and generation speed.

The interleaved pattern is the most faithful to the true joint distribution: it makes no conditional independence assumptions, since each token is conditioned on the complete history. However, the $Q$-fold increase in sequence length directly translates to $Q$-fold slower generation and $Q$-fold higher memory consumption for the KV-cache.

The delayed pattern exploits the observation that the coarser codebook levels ($q=1$) carry the most information about the audio structure, while finer levels ($q>1$) refine acoustic details that depend primarily on the coarse codes at nearby time steps. By delaying finer codebooks, the model ensures that when predicting $c_t^{(q)}$, it has already seen $c_t^{(1)},\dots ,c_t^{(q-1)}$ from the same time step, as well as all codes from earlier time steps. It is worth being precise about when those tokens appear: by (Delayed), the level-$q$ code of frame $t$ is emitted at autoregressive step $t+q-1$, so the frame-$t$ codes $c_t^{(1)},c_t^{(2)},\dots ,c_t^{(q-1)}$ are emitted at steps $t,t+1,\dots ,t+q-2$ respectively - one per step, not all at step $t$. A single frame of the code matrix is therefore completed only after $Q$ consecutive steps, which is exactly why the schedule costs $Q-1$ extra steps in total.

The parallel pattern is the fastest at generation time but makes a strong conditional independence assumption: given the past, the $Q$ codes at time $t$ are predicted independently. This ignores the hierarchical dependency structure of RVQ, where finer codes depend on the residual left by coarser codes.

MusicGen: Architecture and Training

MusicGen (Copet et al., 2023) is the most prominent example of a modern autoregressive music generation system built on neural codec tokens. It uses EnCodec as its audio tokeniser and a decoder-only Transformer with the delayed pattern as its core generation engine.

Token representation.

MusicGen operates on EnCodec tokens at $32$ kHz with $Q=4$ codebook levels and a frame rate of $50$ Hz (one frame per $20$ ms). Each codebook has $V=2048$ entries. A $30$-second music clip is thus represented as a token matrix $𝐂\in {\{1,\dots ,2048\}}^{4\times 1500}$.

Under the delayed pattern (Definition 25), the autoregressive sequence has length $T+Q-1=1503$. At each step $t$, the model predicts four tokens (one per codebook level) in parallel, with codebook $q$ predicting the token for time step $t-q+1$.

Transformer architecture.

The MusicGen Transformer is a standard decoder-only architecture with causal self-attention, as studied in 17. The model uses $N_L=48$ layers, hidden dimension $d=2048$, $32$ attention heads, and a context length of $1503$ tokens. At each position, the input to the Transformer is the sum of $Q$ codebook embeddings: (Musicgen Input)$$\begin{array}{ccc}& {𝒉}_t=\sum _{q=1}^Q{𝐄}^{(q)}{𝒆}_{c_{t-q+1}^{(q)}}+{𝒑}_t,& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${𝐄}^{(q)}\in {\mathrm{ℝ}}^{d\times V}$ is the embedding matrix for codebook $q$, ${𝒆}_c\in {\mathrm{ℝ}}^V$ is a one-hot vector for code $c$, and ${𝒑}_t\in {\mathrm{ℝ}}^d$ is the positional embedding for step $t$.

Text conditioning.

MusicGen uses a pre-trained T5 encoder to convert text descriptions into a sequence of conditioning embeddings. These are injected into the Transformer via cross-attention at every layer. Let ${𝐇}^{(l)}\in {\mathrm{ℝ}}^{T_{\text{seq}}\times d}$ denote the hidden states at layer $l$ and ${𝐂}_{\text{text}}\in {\mathrm{ℝ}}^{T_{\text{text}}\times d}$ the T5 embeddings projected to dimension $d$. The cross-attention operation at each layer is (Musicgen Crossattn)$$\begin{array}{ccc}& \text{CrossAttn}({𝐇}^{(l)},{𝐂}_{\text{text}})=\mathrm{softmax}\left(\frac{𝐐{𝐊}^{\top }}{\sqrt{d_h}}\right)𝐕,& \text{<span class="tml-eqn"></span>}\end{array}$$ where $𝐐={𝐇}^{(l)}{𝐖}_Q$, $𝐊={𝐂}_{\text{text}}{𝐖}_K$, $𝐕={𝐂}_{\text{text}}{𝐖}_V$, and $d_h=d/H$ is the per-head dimension. This follows the standard cross-attention mechanism studied in Conditioning Mechanisms.

Melody conditioning via chromagram.

A distinctive feature of MusicGen is its ability to condition on a reference melody. The melody is represented as a chromagram, which encodes the distribution of musical pitch classes over time.

The chromagram captures harmonic content while being invariant to timbre, dynamics, and instrumentation. Two performances of the same melody on different instruments will produce similar chromagrams. MusicGen discretises the chromagram by taking, at each frame, the $\mathrm{arg\hspace{0.17em}max}$ over the twelve pitch-class bins (after suppressing the percussive component of the signal), which yields one dominant pitch-class index per frame. This index sequence is embedded and provided to the Transformer as an additional conditioning signal alongside the text embeddings. No separate learned codebook is involved: the “quantisation” is the argmax over an existing twelve-way axis, and keeping only the dominant bin is deliberate, since retaining the full chroma vector lets the model copy the reference almost verbatim rather than treating it as a melodic constraint.

The Autoregressive Bottleneck

Despite the improvements brought by neural codecs and the delayed pattern, autoregressive audio generation faces a fundamental computational bottleneck: generation is inherently sequential. Each token depends on all previously generated tokens, so the model must be evaluated once per time step, with no opportunity for temporal parallelism during inference.

Sequential generation speed.

For a sequence of length $L$, autoregressive generation requires $L$ forward passes through the Transformer. With a KV-cache, each forward pass processes a single new position, so its cost scales as $O(d^2+L_{\text{cache}}\cdot d)$, where $d$ is the model dimension and $L_{\text{cache}}$ is the current cache length. Summing over the $L$ steps, during which the cache grows from $1$ to $L$, gives a total of (AR COST Order)$$\begin{array}{ccc}& O(L\cdot d^2+L^2\cdot d).& \text{<span class="tml-eqn"></span>}\end{array}$$ It is worth reading this expression carefully, because it is easy to mislabel it as “quadratic in the sequence length”. The two terms are of comparable size only when $L\sim d$; below that the linear term dominates and the total cost is, to a good approximation, linear in $L$. With the constants worked out in Proposition 13, the crossover for the MusicGen configuration ($d=2048$) sits at $L=12d=\mathrm{24,576}$ tokens, which is more than eight minutes of audio at $50$ Hz. For all sequence lengths of practical interest in this chapter the quadratic attention term is a correction, not the leading cost.

KV-cache memory.

The KV-cache stores the key and value tensors for all previous positions, consuming (KV Memory)$$\begin{array}{ccc}& {\text{Memory}}_{\text{KV}}=2\cdot N_L\cdot L\cdot d\cdot \text{bytes-per-element}& \text{<span class="tml-eqn"></span>}\end{array}$$ bytes. For MusicGen's configuration ($N_L=48$, $d=2048$, $L=1503$, $\text{float16}$), this is $2\times 48\times 1503\times 2048\times 2=\mathrm{591,003,648}$ bytes per batch element, that is $564$ MiB (equivalently $591$ MB in decimal units - the distinction matters when the two are within ten per cent of a device's memory budget). Generating long audio clips (e.g., $5$ minutes) would require $L\approx \mathrm{15,000}$, pushing KV-cache memory to $5.49$ GiB ($5.90$ GB) per sample. For a treatment of memory-efficient attention mechanisms that can mitigate this cost, see 24.

Conditioning Mechanisms for Audio Generation

A generative model that produces audio unconditionally is of limited practical use: we nearly always want to control what is generated. The conditioning signal might be a text description (“a jazz piano solo with brushed drums”), a melody hummed by the user, a chord progression, song lyrics, a reference audio clip whose style should be transferred, or even a video whose soundtrack must be synchronised.

This section catalogues the principal conditioning modalities for audio generation, then analyses the mathematical mechanisms through which conditioning information is injected into the generative model. Many of these mechanisms are shared with image and video generation (Conditioning Mechanisms); here we focus on the adaptations and considerations specific to the audio domain.

The Conditioning Landscape

Table 1 summarises the principal conditioning modalities used in modern audio generation systems.

Modality	Representation	Control Level	Example Systems
Text description	CLAP / T5 embeddings	Semantic	MusicGen, AudioLDM2
Melody	Chromagram tokens	Pitch contour	MusicGen (melody mode)
Chords	Chord sequence embeddings	Harmonic structure	SingSong, Jukebox
Lyrics	Phoneme / text alignment	Vocal content	Jukebox, VoiceCraft
Audio reference	Codec / CLAP embedding	Style / timbre	MusicLM, AudioLDM
Video	CLIP / temporal features	Temporal sync	SpecVQGAN, Diff-Foley
MIDI	Discrete note events	Note-level control	MIDI-DDSP

The richness of this conditioning landscape is a distinguishing feature of audio generation compared to image generation, where text is the dominant conditioning modality. In audio, the temporal structure of the output creates opportunities for temporally aligned conditioning signals (melody, chords, lyrics, video) that have no direct analogue in the image domain.

Cross-Attention Conditioning

Cross-attention is the most common mechanism for injecting conditioning information into Transformer-based audio generators. The mechanism is identical in form to the cross-attention used in image and video diffusion models (Conditioning Mechanisms), but the conditioning and generation modalities differ.

The attention weights $𝐀=\mathrm{softmax}(𝐐{𝐊}^{\top }/\sqrt{d_h})\in {\mathrm{ℝ}}^{T_{\text{seq}}\times T_c}$ form a soft alignment matrix between the audio generation positions and the conditioning positions. Each row of $𝐀$ is a probability distribution over the conditioning tokens, determining how much each conditioning token influences the corresponding audio generation position.

FiLM Conditioning

Feature-wise Linear Modulation (FiLM) provides a computationally efficient alternative to cross-attention for injecting conditioning information. Rather than attending to a sequence of conditioning tokens, FiLM conditions by applying a learned affine transformation to the hidden features.

The FiLM mechanism is closely related to the Adaptive Layer Normalisation (AdaLN) used in diffusion Transformers. In AdaLN, the affine parameters of layer normalisation are conditioned on the time step and class label; FiLM generalises this to arbitrary conditioning signals.

Classifier-Free Guidance for Audio

Classifier-free guidance (CFG) was originally developed for class-conditional image generation and later extended to text-to-image and text-to-video diffusion models (see Classifier-Free Guidance for Video). Its application to audio generation follows the same mathematical framework, but the practical considerations differ in important ways.

Review of CFG.

During training, the conditioning signal $𝒄$ is dropped with probability $p_{\text{drop}}$ (typically $0.1$ to $0.2$), replaced by a null conditioning token ${𝒄}_{⌀}$. At inference, the model's output (score function, velocity field, or logits, depending on the generative framework) is extrapolated along the conditioning direction: (CFG)$$\begin{array}{ccc}& \widetilde{𝝐}({𝒙}_t,𝒄)=(1+w){𝝐}_{\theta }({𝒙}_t,𝒄)-w{𝝐}_{\theta }({𝒙}_t,{𝒄}_{⌀}),& \text{<span class="tml-eqn"></span>}\end{array}$$ where $w\ge 0$ is the guidance scale. Note the convention: $w=0$ makes (CFG) collapse to ${𝝐}_{\theta }({𝒙}_t,𝒄)$, that is to ordinary conditional generation with no guidance at all; it is $w=-1$ that would recover the unconditional prediction ${𝝐}_{\theta }({𝒙}_t,{𝒄}_{⌀})$. Increasing $w$ above $0$ extrapolates past the conditional prediction, amplifying the influence of the conditioning signal at the cost of reduced sample diversity.

Audio-specific guidance scales.

A notable empirical observation is that the optimal guidance scales reported for audio generation are lower than those reported for image generation. Part of that gap is bookkeeping rather than substance: the image figures usually quoted, $7.5$ to $15$, are values of $s$ and so correspond to $w\in [6.5,14]$, which shrinks the apparent gap to roughly a factor of two to three. The remainder is real, and reflects how audible over-guidance is: the artefacts catalogued at the end of this subsection (harmonic distortion, spectral peaking, temporal repetition) appear at guidance strengths that would still look benign in an image. Text-to-audio systems therefore operate in the range $w\in [2,7]$, with the precise choice depending strongly on the task; Remark 38 gives per-task figures, from $w\in [1,3]$ for speech up to $w\in [3,7]$ for music.

It is tempting to explain the audio/image difference by observing that audio captions are vaguer than image prompts (“birds singing near a stream” versus “a red car parked on a snowy street”). That observation is correct, but it does not by itself settle the direction of the effect, and the reader should be wary of it: vagueness can be read as leaving room for perceptually valid variation, hence arguing for less guidance, or as under-determining the target, hence arguing for more. Both readings appear in the literature. What the evidence actually supports is the narrower claim above - that audio degrades audibly at lower guidance strengths than images degrade visibly - together with the per-task ordering in Remark 38, in which the tasks with the most intrinsic structure (speech) tolerate the least guidance.

Multi-Modal Conditioning

Real-world audio generation tasks often require simultaneous conditioning on multiple modalities. A user might specify a text description (“upbeat pop song”), provide a melody (hummed reference), and indicate a desired style (audio of a reference track). The generation model must fuse these heterogeneous signals into a coherent conditioning representation.

Conditioning fusion strategies.

Three principal strategies exist for combining multiple conditioning modalities:

1. Concatenation: The conditioning embeddings from different modalities are concatenated along the sequence dimension before being provided to cross-attention. If the text embeddings are ${𝐂}_{\text{text}}\in {\mathrm{ℝ}}^{T_t\times d}$ and the melody embeddings are ${𝐂}_{\text{mel}}\in {\mathrm{ℝ}}^{T_m\times d}$, the combined conditioning is (Concat COND)$$\begin{array}{ccc}& {𝐂}_{\text{combined}}=[{𝐂}_{\text{text}};{𝐂}_{\text{mel}}]\in {\mathrm{ℝ}}^{(T_t+T_m)\times d}.& \text{<span class="tml-eqn"></span>}\end{array}$$ The attention mechanism then learns to attend to the appropriate modality at each generation position.

2. Separate cross-attention: Each conditioning modality is injected through its own cross-attention layer, and the outputs are summed: (Separate Crossattn)$$\begin{array}{ccc}& {𝒉}_t^{\text{cond}}={\text{CrossAttn}}_{\text{text}}({𝒉}_t,{𝐂}_{\text{text}})+{\text{CrossAttn}}_{\text{mel}}({𝒉}_t,{𝐂}_{\text{mel}}).& \text{<span class="tml-eqn"></span>}\end{array}$$ This allows independent attention patterns for each modality but increases the parameter count and computational cost.

3. Hierarchical conditioning: Global conditioning signals (style, genre) are injected via FiLM, while sequential conditioning signals (text, melody) are injected via cross-attention. This leverages the strengths of each mechanism.

Spectrogram-Domain Generation

Before neural codecs and latent diffusion models became the dominant paradigm, an influential line of work explored generating audio by treating mel spectrograms as images and applying image generation techniques directly. This approach, most famously demonstrated by Riffusion, is conceptually elegant: it repurposes the extensive machinery developed for image synthesis (U-Net denoisers, latent diffusion, classifier-free guidance) without any audio-specific architectural modifications.

This section analyses the mathematics of spectrogram-domain generation, its surprising effectiveness, and the fundamental limitations that ultimately motivated the shift to latent codec spaces.

Riffusion: Image Diffusion Meets Audio

Riffusion (Forsgren and Martiros, 2022) was a proof-of-concept demonstration that fine-tuning Stable Diffusion on mel spectrogram images could produce musically coherent audio. The approach consists of three stages:

1. Data preparation. Each audio clip is converted to a mel spectrogram $𝐒\in {\mathrm{ℝ}}^{T_s\times F_s}$, which is then rescaled and formatted as a single-channel (or three-channel, with the spectrogram duplicated) image of fixed resolution (e.g., $512\times 512$).

2. Fine-tuning. The pre-trained Stable Diffusion model, including its latent autoencoder and U-Net denoiser, is fine-tuned on the spectrogram “images” with corresponding text captions. The VAE encoder learns to compress spectrograms into the same latent space, and the U-Net learns to denoise spectrograms rather than natural images.

3. Inversion. The generated spectrogram is converted back to a waveform using a spectrogram inversion algorithm (e.g., Griffin–Lim or a neural vocoder such as HiFi-GAN).

Formally, the Riffusion generation process can be described as follows. Let ${ℰ}_{\text{img}}$ and ${𝒟}_{\text{img}}$ denote the pre-trained (and fine-tuned) Stable Diffusion VAE encoder and decoder, and let ${𝝐}_{\theta }$ be the fine-tuned U-Net denoiser. Given a text prompt $𝒄$, the generation pipeline is: (Riffusion Sample)$$\begin{array}{cccc}& {𝒛}_T& \sim \mathrm{Normal}(0,𝐈),& \\ & {𝒛}_0& =\text{DDPM-Sample}({𝒛}_T,{𝝐}_{\theta },𝒄),& \\ & \widehat{𝐒}& ={𝒟}_{\text{img}}({𝒛}_0),& \\ & \widehat{𝒙}& =\text{Vocoder}(\widehat{𝐒}),& \text{<span class="tml-eqn"></span>}\end{array}$$ where $\widehat{𝐒}\in {\mathrm{ℝ}}^{T_s\times F_s}$ is the generated mel spectrogram and $\widehat{𝒙}\in {\mathrm{ℝ}}^L$ is the reconstructed waveform.

Spectrogram-to-waveform inversion.

The critical step in the Riffusion pipeline is converting the generated mel spectrogram back to a waveform. This inversion is inherently ill-posed: the mel spectrogram discards phase information and compresses the linear frequency axis, so multiple waveforms can produce the same mel spectrogram.

The classical approach is the Griffin–Lim algorithm, which iteratively searches for a phase field consistent with the given magnitudes by alternating between the time-domain and frequency-domain constraints. Starting from a random phase estimate ${\varphi }^{(0)}$, the algorithm iterates: (Griffin LIM)$$\begin{array}{cccc}& X^{(k)}[m,n]& =\sqrt{S[m,n]}{e}^{i{\varphi }^{(k)}[m,n]},& \\ & x^{(k)}(n)& =\text{Overlap-Add}(\text{IFFT}(X^{(k)})),& \\ & {\varphi }^{(k+1)}[m,n]& =\angle \text{STFT}(x^{(k)})[m,n],& \text{<span class="tml-eqn"></span>}\end{array}$$ where $S[m,n]$ is the magnitude spectrogram (obtained by inverting the mel filterbank) and $\angle$ denotes the phase angle. After $K$ iterations (typically $K=32$ to $64$), the reconstructed waveform $x^{(K)}$ approximately satisfies the magnitude constraints.

It is important to be clear about what this procedure does and does not achieve. Each iteration monotonically decreases the inconsistency between $X^{(k)}$ and the STFT of its own overlap-added inverse, and that is the only quantity it decreases. It does not decrease the distance to the original phase, and in general it does not recover it. Running the iteration on a signal whose true phase is known makes the point sharply: the magnitude error falls by an order of magnitude while the phase error of Definition 32 stays pinned at the value a uniformly random phase would score. Griffin–Lim converges to a consistent phase, not the phase; the resulting waveform has roughly the right spectral envelope and the wrong fine structure, which is exactly what the characteristic “phasiness” of Griffin–Lim output sounds like.

Modern systems replace Griffin–Lim with neural vocoders (HiFi-GAN, BigVGAN, Vocos) that directly map mel spectrograms to waveforms using a generative neural network. These produce substantially higher-quality audio, as they learn to predict plausible phase patterns from the spectrogram structure.

Mathematics of Spectrogram Diffusion

When we apply diffusion models to mel spectrograms, the data space is ${\mathrm{ℝ}}^{T_s\times F_s}$ rather than the image space ${\mathrm{ℝ}}^{H\times W\times 3}$. The forward noising process and reverse denoising process follow the same mathematical framework as image diffusion, but the structure of the data introduces domain-specific considerations.

The key question is: what does the denoiser learn about the structure of mel spectrograms? Natural audio spectrograms exhibit a rich set of statistical regularities:

1. Horizontal continuity: Harmonic components of sustained sounds create horizontal ridges in the spectrogram that persist across many time frames.

2. Harmonic spacing: A fundamental frequency $f_0$ produces harmonics at $2f_0,3f_0,\dots$, which appear as evenly spaced horizontal lines (on a linear frequency axis) or logarithmically spaced lines (on a mel axis).

3. Onset transients: Percussive events produce vertical broadband features spanning many frequency bins in a single time frame.

4. Spectral envelope: The overall energy distribution across frequencies follows smooth, slowly varying envelopes determined by the source and resonance characteristics.

The diffusion model learns these regularities implicitly through the training data. The U-Net architecture is particularly well suited to capturing both local features (individual harmonics) and global structure (musical form, temporal dynamics) through its multi-scale encoder–decoder structure.

Limitations of Spectrogram-Domain Generation

Despite its conceptual elegance, spectrogram-domain generation suffers from several fundamental limitations that have driven the field toward latent codec spaces.

Phase loss.

The most fundamental limitation is the loss of phase information. A mel spectrogram represents only the magnitude of the STFT, discarding the phase component. This means that the diffusion model generates magnitude-only representations, and the phase must be estimated during the inversion step.

A useful reference point on this scale is the value achieved by guessing. If $\hat{\mathrm{\Phi }}[m,k]-{\mathrm{\Phi }}^{\ast }[m,k]$ is uniform on $[-\pi ,\pi ]$ then $𝔼[2(1-\cos \mathrm{\Delta })]=2$, so ${ℰ}_{\text{phase}}=2$ is what a phase field carrying no information about the target scores. This is the number to compare against when assessing an inversion algorithm, and, as noted above, it is roughly where Griffin–Lim sits however long it is run.

Even state-of-the-art neural vocoders introduce subtle phase artefacts, particularly in regions of the spectrogram with rapid temporal variation (transients) or closely spaced harmonics. These artefacts manifest as “buzziness,” “phasiness,” or reduced clarity in the reconstructed audio.

Frequency resolution.

The mel frequency scale provides perceptually motivated frequency resolution: high resolution at low frequencies (where pitch perception is acute) and lower resolution at high frequencies (where the ear is less sensitive to fine spectral detail). However, this compression means that the generated spectrogram has limited control over the fine frequency structure of the audio.

Limited duration.

Spectrogram-domain generation inherits the resolution constraints of the underlying image generation model. Standard diffusion models operate on fixed-resolution images (e.g., $512\times 512$), which limits the spectrogram to a fixed time–frequency grid. For a hop size of $H=512$ at $f_s=\mathrm{44,100}$ Hz, a $512$-frame spectrogram represents only $512\times 512/44100\approx 5.9$ seconds of audio. Generating longer clips requires either: (i) sequential generation with overlap and crossfade, which introduces stitching artefacts; or (ii) higher-resolution generation, which is computationally expensive and may not be supported by the architecture.

Exercises

Latent Diffusion for Audio

The preceding sections developed two pillars of modern audio generation: contrastive embeddings that align audio and text (CLAP: Contrastive Language–Audio Pretraining), and autoregressive models that treat audio tokens as a language (Autoregressive Music Generation). We now turn to a third pillar, one that has arguably had the greatest impact on text-to-audio quality and diversity: latent diffusion models (LDMs). The core idea, introduced for images in 19 and now adapted to audio, is to decompose generation into three conceptually distinct stages:

1. Compress: an autoencoder maps the audio signal, or its mel-spectrogram, to a compact latent space of much lower dimension.

2. Diffuse: a diffusion model generates novel latent representations conditioned on text, tags, or other control signals.

3. Decode: the autoencoder's decoder (often augmented by a neural vocoder) maps the generated latent back to a waveform.

This three-stage decomposition is not merely an engineering convenience; it offers precise mathematical advantages. By learning to generate in a compressed latent space rather than in raw waveform or spectrogram space, the diffusion model operates in a domain where the data manifold has lower intrinsic dimension, the signal-to-noise ratio is more favourable, and the computational cost per denoising step is dramatically reduced.

In this section, we formalise the three-stage pipeline (The Three-Stage Pipeline), study AudioLDM, the system that established the paradigm (AudioLDM: The Foundational System), then examine AudioLDM2's architectural innovations (AudioLDM2: Language-Aligned Latents), and finally develop MusicLDM's beat-synchronous data augmentation (MusicLDM: Beat-Synchronous Augmentation).

The Three-Stage Pipeline

We begin with a precise formulation of the three-stage architecture.

The mathematical advantage of the three-stage decomposition can be quantified through the lens of the evidence lower bound (ELBO). For a standard diffusion model operating on mel spectrograms, the variational bound decomposes as (ELBO Pixel)$$\begin{array}{ccc}& \log p(𝐒)\ge {𝔼}_{q({𝒛}_{1:T}|𝐒)}\left[\log \frac{p({𝒛}_T){\prod }_{t=1}^T{p}_{\theta }({𝒛}_{t-1}|{𝒛}_t)}{{\prod }_{t=1}^{T}q({𝒛}_t|{𝒛}_{t-1})}\right].& \text{<span class="tml-eqn"></span>}\end{array}$$ When the diffusion operates in the latent space of a pretrained autoencoder, we instead lower-bound $\log p(𝐒)$ in two stages: (ELBO Latent 1)$$\begin{array}{cccc}& \log p(𝐒)& \ge {𝔼}_{{𝒛}_0=ℰ(𝐒)}[\log p_{\theta }({𝒛}_0)+\log p_{𝒟}(𝐒|{𝒛}_0)],& \text{<span class="tml-eqn"></span>}\\ & \log p_{\theta }({𝒛}_0)& \ge {\text{ELBO}}_{\text{diffusion}}({𝒛}_0),& \text{<span class="tml-eqn"></span>}\end{array}$$ where $p_{𝒟}(𝐒|{𝒛}_0)$ captures the reconstruction quality of the autoencoder. The key insight is that ${\text{ELBO}}_{\text{diffusion}}$ involves denoising in the low-dimensional latent space, which is computationally cheaper and statistically easier to learn.

Autoencoder design for audio latent diffusion.

The audio autoencoder is typically a variational autoencoder (VAE) trained on mel spectrograms. The encoder produces a mean and log-variance for each latent position: (VAE Encode)$$\begin{array}{ccc}& ℰ(𝐒)=𝝁+𝝈\odot 𝝐,𝝁,\log {𝝈}^2=\text{EncoderNet}(𝐒),𝝐\sim \mathrm{Normal}(0,𝐈),& \text{<span class="tml-eqn"></span>}\end{array}$$ where the reparametrisation trick enables gradient-based training. The VAE is trained with a combination of reconstruction loss and KL regularisation: (VAE LOSS)$$\begin{array}{ccc}& {ℒ}_{\text{VAE}}=\underset{\text{reconstruction}}{\underbrace{{\Vert 𝐒-\widehat{𝐒}\Vert }_1+{\lambda }_{\text{stft}}{ℒ}_{\text{MR-STFT}}(𝐒,\widehat{𝐒})}}+\underset{\text{regularisation}}{\underbrace{{\lambda }_{\text{KL}}{D}_{\text{KL}}(q(𝒛|𝐒)\Vert \mathrm{Normal}(0,𝐈))}},& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${ℒ}_{\text{MR-STFT}}$ is the multi-resolution STFT loss (HiFi-GAN Training Losses) and ${\lambda }_{\text{KL}}$ is a small weighting coefficient (typically ${10}^{-6}$ to ${10}^{-4}$) to prevent posterior collapse while keeping the latent space smooth.

AudioLDM: The Foundational System

AudioLDM, introduced by Liu et al. (2023), was the first system to demonstrate that the latent diffusion paradigm from image generation could be transferred wholesale to audio, achieving state-of-the-art text-to-audio results with a remarkably simple architecture. The system consists of three components: a pretrained audio VAE, a U-Net-based diffusion model conditioned on CLAP embeddings, and a HiFi-GAN vocoder.

Architecture.

The AudioLDM pipeline instantiates Definition 33 as follows. The audio VAE is trained on mel spectrograms of size $𝐒\in {\mathrm{ℝ}}^{T_s\times F_s}$ (typically $T_s=1024$, $F_s=64$ for 10-second clips at 16,kHz with a hop size of 160 samples). The encoder compresses by a factor of $4\times$ along the time axis and $8\times$ along the frequency axis, producing latents ${𝒛}_0\in {\mathrm{ℝ}}^{256\times 8}$. The U-Net follows the architecture from Stable Diffusion (Rombach et al., 2022), adapted for the two-dimensional latent spectrogram shape: it uses 2D convolutions (treating time and compressed-frequency as spatial axes), self-attention at the bottleneck resolution, and cross-attention layers that attend to the conditioning embeddings.

CLAP conditioning.

The conditioning embedding $𝒄$ is obtained from a pretrained CLAP model (CLAP: Contrastive Language–Audio Pretraining). Given a text prompt ${𝒙}^{(t)}$, the CLAP text encoder produces a global embedding ${𝒄}_{\text{global}}=g_{\varphi }({𝒙}^{(t)})\in {\mathrm{ℝ}}^d$. This embedding is injected into the U-Net via cross-attention: (Audioldm Crossattn)$$\begin{array}{ccc}& \text{CrossAttn}(𝒉,{𝒄}_{\text{global}})=\mathrm{softmax}\left(\frac{(𝒉{𝐖}_Q){({𝒄}_{\text{global}}{𝐖}_K)}^{\top }}{\sqrt{d_h}}\right)({𝒄}_{\text{global}}{𝐖}_V),& \text{<span class="tml-eqn"></span>}\end{array}$$ where $𝒉$ represents the spatial features at a given U-Net layer and the CLAP embedding is treated as a sequence of length 1. This is equivalent to the image-domain cross-attention mechanism studied in 19, with CLAP replacing CLIP as the conditioning encoder.

AudioLDM2: Language-Aligned Latents

AudioLDM2 (Liu et al., 2024) addresses a fundamental limitation of the original AudioLDM: the CLAP embedding is a single global vector, which provides only a coarse, non-temporal conditioning signal. A text prompt like “a piano plays a gentle melody, then thunder rumbles” describes a temporal sequence of events, but a single CLAP vector cannot represent this ordering. AudioLDM2 introduces two architectural innovations to enable richer, temporally structured conditioning: an AudioMAE-based representation learning module and a Language-Oriented Audio (LOA) generation framework.

AudioMAE representation.

AudioLDM2 replaces the single CLAP vector with a sequence of latent vectors derived from an AudioMAE (Audio Masked Autoencoder). The AudioMAE is a vision transformer trained on mel spectrograms using the masked autoencoding objective: random patches of the spectrogram are masked, and the model learns to reconstruct them from the visible patches. After pretraining, the AudioMAE encoder produces a sequence of patch embeddings that capture both local spectral features and global temporal structure.

Language-Oriented Audio (LOA) generation.

The central innovation of AudioLDM2 is the LOA module, which bridges the gap between language representations and audio latent representations. LOA consists of a GPT-2-style autoregressive model that takes as input a sequence-level language representation (from FLAN-T5 or CLAP) and outputs a sequence of AudioMAE-compatible embeddings. These embeddings serve as the conditioning signal for the latent diffusion model, replacing the single CLAP vector of AudioLDM\@.

Formally, let ${𝒄}_{\text{lang}}\in {\mathrm{ℝ}}^{L_t\times d_t}$ be the output of the language encoder (FLAN-T5) for a text prompt, where $L_t$ is the sequence length and $d_t$ is the language embedding dimension. The LOA module generates a sequence of AudioMAE-like tokens: (LOA)$$\begin{array}{ccc}& {𝒉}_i=\text{LOA-GPT}({𝒄}_{\text{lang}},{𝒉}_1,\dots ,{𝒉}_{i-1}),i=1,\dots ,N_a,& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${𝒉}_i\in {\mathrm{ℝ}}^{d_m}$ is the $i$-th generated audio token and $N_a$ is the target sequence length (matching the temporal resolution of the diffusion model's conditioning input).

The generated sequence ${𝐇}_{\text{LOA}}=({𝒉}_1,\dots ,{𝒉}_{N_a})\in {\mathrm{ℝ}}^{N_a\times d_m}$ then serves as the cross-attention key and value for the latent diffusion U-Net: (Audioldm2 Crossattn)$$\begin{array}{ccc}& \text{CrossAttn}(𝒇,{𝐇}_{\text{LOA}})=\mathrm{softmax}\left(\frac{(𝒇{𝐖}_Q){({𝐇}_{\text{LOA}}{𝐖}_K)}^{\top }}{\sqrt{d_h}}\right)({𝐇}_{\text{LOA}}{𝐖}_V),& \text{<span class="tml-eqn"></span>}\end{array}$$ where $𝒇$ denotes the spatial features at a given U-Net layer. Unlike AudioLDM's single-vector conditioning, this provides spatially varying conditioning: different temporal positions in the latent spectrogram attend to different audio tokens, enabling the model to generate temporally structured outputs.

MusicLDM: Beat-Synchronous Augmentation

Music generation poses unique challenges compared to general audio generation. Music has strong temporal structure: beats, bars, phrases, and sections create hierarchical rhythmic patterns that listeners perceive and expect. MusicLDM (Chen et al., 2024) addresses this by introducing beat-synchronous mixup, a data augmentation strategy that creates new training examples by mixing pairs of music clips at beat boundaries. The insight is that mixing at metrically meaningful points produces musically coherent augmentations, whereas mixing at arbitrary time points produces artefacts that degrade generation quality.

The key property of beat-synchronous mixup is that the two clips' pulses coincide, so the mixture is heard as one piece of music at one tempo. This contrasts with naive mixup at a random offset, which superimposes two beat grids at an arbitrary phase and is heard as two recordings playing over each other.

Diffusion Architectures for Audio

The previous section established the three-stage latent diffusion pipeline and studied systems built around U-Net denoisers. In this section, we examine the architectural design space more broadly. The central question is: what neural network architecture should serve as the denoising backbone for audio diffusion? Two families dominate the landscape: the U-Net, inherited from image diffusion (19), and the Diffusion Transformer (DiT), adapted from the vision transformer paradigm (Video Diffusion Transformer (DiT)). Each architecture offers distinct mathematical trade-offs in terms of inductive bias, scalability, and conditioning flexibility.

We study three influential systems that explore different points in this design space: Stable Audio, which introduces timing conditioning and, in its later releases, moves to a DiT backbone (Stable Audio: Timing Conditioning, and the Move to a DiT); Noise2Music, which uses a cascaded U-Net architecture (Noise2Music: Cascaded Generation); and Moûsai, which combines diffusion with efficient U-Net variants (Moûsai: Efficient Diffusion for Long Audio).

U-Net vs. DiT: The Architectural Divide

The U-Net and the DiT represent fundamentally different inductive biases for the denoising function ${𝝐}_{\theta }:{\mathrm{ℝ}}^{T_z\times C_z}\times [0,T]\times {\mathrm{ℝ}}^{d_c}\to {\mathrm{ℝ}}^{T_z\times C_z}$.

U-Net.

The U-Net processes the latent spectrogram through an encoder-decoder structure with skip connections at matching resolutions. At each resolution level $\ell$, the encoder applies a sequence of residual blocks and (optionally) self-attention: (UNET Block)$$\begin{array}{ccc}& {𝒉}^{(\ell +1)}=\text{Downsample}(\text{ResBlock}({𝒉}^{(\ell )},t)+\text{SelfAttn}({𝒉}^{(\ell )})+\text{CrossAttn}({𝒉}^{(\ell )},𝒄)),& \text{<span class="tml-eqn"></span>}\end{array}$$ where the timestep $t$ is injected via adaptive group normalisation (AdaGN), and the conditioning $𝒄$ enters through cross-attention. The decoder mirrors this structure with upsampling operations and skip connections from the encoder. The U-Net's multi-scale structure is a natural fit for spectrograms, which exhibit different statistical properties at different time-frequency scales.

DiT.

The Diffusion Transformer treats the latent representation as a sequence of patches and processes it with a standard transformer encoder. The latent ${𝒛}_t\in {\mathrm{ℝ}}^{T_z\times C_z}$ is first patchified into a sequence of $N_p$ tokens, each of dimension $d_{\text{model}}$: (DIT Patchify)$$\begin{array}{ccc}& {𝒛}_t\stackrel{\stackrel{}{\text{patchify}}}{\to }({𝒑}_1,\dots ,{𝒑}_{N_p}),{𝒑}_i\in {\mathrm{ℝ}}^{d_{\text{model}}}.& \text{<span class="tml-eqn"></span>}\end{array}$$ Each transformer block then applies self-attention and a feed-forward network with adaptive layer normalisation (adaLN) for timestep and conditioning injection: (DIT Block)$$\begin{array}{ccc}& {𝒉}_i^{\prime }={𝒉}_i+{\gamma }_1(t,𝒄)\odot \text{SelfAttn}(\text{LN}({𝒉}_i;{\beta }_1(t,𝒄))),& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${\gamma }_1,{\beta }_1$ are scale and shift parameters predicted from the timestep and conditioning embeddings. The DiT's advantage is scalability: it has no resolution-specific components, and its compute scales linearly with the number of layers for a fixed sequence length, making it straightforward to scale to billions of parameters.

Stable Audio: Timing Conditioning, and the Move to a DiT

Stable Audio (Evans et al., 2024) introduces two innovations that have since been widely copied: explicit timing conditioning, which enables variable-length generation up to 95 seconds, and a waveform-domain VAE that bypasses the mel-spectrogram representation entirely.

Waveform VAE.

Unlike AudioLDM, which compresses mel spectrograms, Stable Audio trains its VAE directly on raw waveforms. The encoder operates on the waveform $𝒙\in {\mathrm{ℝ}}^L$ (at 44.1,kHz) and produces a continuous latent sequence ${𝒛}_0\in {\mathrm{ℝ}}^{T_z\times C_z}$ with a temporal compression factor of $2048\times$ (from 44.1,kHz to approximately 21.5,Hz). This extreme compression is achieved through a series of strided 1D convolutions: (Stable Audio VAE)$$\begin{array}{ccc}& {𝒛}_0={ℰ}_{\text{wav}}(𝒙),T_z=\lfloor \frac{L}{{\prod }_i{s}_i}\rfloor ,& \text{<span class="tml-eqn"></span>}\end{array}$$ where $s_1,s_2,\dots$ are the stride factors of successive convolutional layers, with ${\prod }_i{s}_i=2048$.

Note that $2048$ is the compression of the clock, not of the data. The encoder widens as it shortens: one scalar sample per $1/44100$ s becomes a vector of $C_z=64$ channels per $1/21.533$ s. In the units of Proposition 20, the compression ratio is a ratio of value counts, $$r=\frac{\mathrm{44,100}}{21.533\times 64}=32,$$ which is what governs the per-step cost, and which sits squarely in the $r\in [16,64]$ band of Remark 27. Quoting $2048$ as though it were $r$ overstates the saving by a factor of $64$.

Architecture details of the Stable Audio Open DiT.

The DiT released with Stable Audio Open uses $N_L=24$ transformer layers with hidden dimension $d=1536$ and 24 attention heads, for a total of approximately 1.06B parameters. (The 95-second timing-conditioned system discussed above is a U-Net of about 907M parameters; only the conditioning interface is shared.) Conditioning is provided through three channels:

1. Text embeddings from a CLAP text encoder, injected via cross-attention at every transformer block.

2. Timing embeddings $({𝒆}_{{\tau }_s},{𝒆}_{{\tau }_d})$, injected via adaLN alongside the diffusion timestep.

3. Diffusion timestep embedding ${𝒆}_t$, also injected via adaLN.

The combined conditioning vector for adaLN is (Stable Audio COND)$$\begin{array}{ccc}& {𝒄}_{\text{adaLN}}=\text{MLP}({𝒆}_t+{𝒆}_{{\tau }_s}+{𝒆}_{{\tau }_d})\in {\mathrm{ℝ}}^d,& \text{<span class="tml-eqn"></span>}\end{array}$$ from which the scale and shift parameters $(\gamma ,\beta )$ are extracted for each transformer layer.

Noise2Music: Cascaded Generation

Noise2Music (Huang et al., 2023) takes a different architectural approach: instead of a single diffusion model in latent space, it uses a cascade of two diffusion models operating at different resolutions in spectrogram space. This mirrors the cascaded approach used in image generation (e.g., DALL-E 2, Imagen) and provides a natural decomposition of the generation problem into coarse structure and fine detail.

Base model.

The Noise2Music base model generates a full 30-second clip at a reduced temporal resolution; in the running example of this subsection we take $T_s^{(\text{lo})}=64$ frames at a $50$ ms hop, which covers $3.2$ seconds and keeps the arithmetic small. It uses a 1D U-Net operating along the time axis, treating the $F_s$ frequency bins as channels. The text conditioning is provided by a frozen T5 encoder whose embeddings are injected via cross-attention. A second large language model, LaMDA, appears elsewhere in the Noise2Music pipeline, but in a different role: it is used offline to pseudo-label unlabelled music with captions, enlarging the paired training set. It is not the conditioning encoder.

Super-resolution model.

The super-resolution model takes the base spectrogram ${𝐒}^{(\text{lo})}$ (bilinearly upsampled to full resolution) as an additional input channel, concatenated with the noisy full-resolution spectrogram at each diffusion step: (SR Input)$$\begin{array}{ccc}& \text{input}=[{𝒛}_t;\text{Upsample}({𝐒}^{(\text{lo})})]\in {\mathrm{ℝ}}^{T_s\times 2F_s}.& \text{<span class="tml-eqn"></span>}\end{array}$$ This concatenation-based conditioning allows the super-resolution model to use the base spectrogram as a structural guide while adding high-frequency detail.

Moûsai: Efficient Diffusion for Long Audio

Moûsai (Schneider et al., 2023) addresses the challenge of generating long music clips (up to 48 seconds) with limited computational resources. Its key insight is that a carefully designed 1D U-Net operating on waveform-domain latents can achieve competitive quality with far fewer parameters than spectrogram-based alternatives.

Cascaded 1D U-Net.

Moûsai uses a two-stage design, but the latent space is its own, not a borrowed neural codec. The first stage is a diffusion magnitude-autoencoder: a spectrogram magnitude is compressed by a factor of roughly $64\times$ into a continuous 1D latent sequence, and a diffusion decoder reconstructs the waveform from it, phase included. The second stage is a latent diffusion model that generates those latents from text. Both stages use 1D temporal convolutions along the time axis.

The 1D U-Net replaces the 2D convolutions of spectrogram-based models with 1D temporal convolutions: (Mousai CONV)$$\begin{array}{ccc}& {𝒉}^{(\ell +1)}[t]=\sigma (\sum _{k=-K}^K{𝐖}_k^{(\ell )}{𝒉}^{(\ell )}[t+k]+{𝒃}^{(\ell )}+\gamma (t_{\text{diff}})\text{LN}({𝒉}^{(\ell )}[t])),& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${𝐖}_k^{(\ell )}$ are the convolution weights at level $\ell$ with kernel half-width $K$, $\sigma$ is a nonlinearity, and $\gamma (t_{\text{diff}})$ is the timestep-dependent scale from adaptive normalisation.

Diffusion in Audio Latent Spaces

The previous two sections described the architectures and systems that instantiate latent audio diffusion. This section examines the diffusion process itself in detail: the forward noising process that corrupts audio latents, the noise schedules that control corruption speed, and the alternative parameterisations of the denoising objective that affect training dynamics and generation quality. While the mathematical framework follows the general theory developed in 19, the audio domain introduces specific considerations related to the structure of audio latent spaces, the multi-scale nature of audio signals, and the interaction between noise schedules and perceptual quality.

The Forward Process in Audio Latent Space

Let ${𝒛}_0\in {\mathrm{ℝ}}^{T_z\times C_z}$ be a clean audio latent obtained from a pretrained encoder $ℰ$. The forward diffusion process is a Markov chain that progressively adds Gaussian noise: (Forward)$$\begin{array}{ccc}& q({𝒛}_t|{𝒛}_{t-1})=\mathrm{Normal}({𝒛}_t;\sqrt{1-{\beta }_t}{𝒛}_{t-1},{\beta }_t𝐈),t=1,\dots ,T,& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${\{{\beta }_t\}}_{t=1}^T$ is the noise schedule with ${\beta }_t\in (0,1)$. Using the reparametrisation trick, we can sample ${𝒛}_t$ directly from ${𝒛}_0$ without iterating through the chain: (Forward Closed)$$\begin{array}{ccc}& q({𝒛}_t|{𝒛}_0)=\mathrm{Normal}({𝒛}_t;\sqrt{{\bar{\alpha }}_t}{𝒛}_0,(1-{\bar{\alpha }}_t)𝐈),& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${\bar{\alpha }}_t={\prod }_{s=1}^t(1-{\beta }_s)$ is the cumulative signal retention coefficient. Equivalently, (Forward Reparam)$$\begin{array}{ccc}& {𝒛}_t=\sqrt{{\bar{\alpha }}_t}{𝒛}_0+\sqrt{1-{\bar{\alpha }}_t}𝝐,𝝐\sim \mathrm{Normal}(0,𝐈).& \text{<span class="tml-eqn"></span>}\end{array}$$

The SNR is the central quantity governing the diffusion process. It determines how much signal information remains at each timestep and directly controls the difficulty of the denoising task. For audio latent spaces, the choice of noise schedule (i.e., how the SNR decreases from $\infty$ to $0$) has a profound impact on generation quality.

Noise Schedules for Audio

The noise schedule ${\{{\beta }_t\}}_{t=1}^T$ (or equivalently, the function $\bar{\alpha }(t)$ in continuous time) determines the rate at which the signal is corrupted. The general theory of noise schedules is developed in Noise Schedules for Video; here we focus on the specific considerations that arise when the diffusion process operates in an audio latent space.

Parameterisation: $𝝐$, ${𝒙}_0$, and $𝒗$epsilon, x0, and v

The denoising network ${𝝐}_{\theta }$ can be trained to predict different targets, each corresponding to a different mathematical parameterisation of the reverse process. While these parameterisations are theoretically equivalent (they all define the same conditional distribution $p_{\theta }({𝒛}_{t-1}|{𝒛}_t)$), they differ significantly in training dynamics, gradient magnitudes, and empirical generation quality.

The three parameterisations are related by linear transformations. Given any one prediction, the other two can be recovered: (Param Conversion 1)$$\begin{array}{cccc}& {𝒛}_0& =\frac{1}{\sqrt{{\bar{\alpha }}_t}}({𝒛}_t-\sqrt{1-{\bar{\alpha }}_t}𝝐),& \text{<span class="tml-eqn"></span>}\\ & 𝝐& =\frac{1}{\sqrt{1-{\bar{\alpha }}_t}}({𝒛}_t-\sqrt{{\bar{\alpha }}_t}{𝒛}_0),& \text{<span class="tml-eqn"></span>}\\ & 𝒗& =\sqrt{{\bar{\alpha }}_t}𝝐-\sqrt{1-{\bar{\alpha }}_t}{𝒛}_0=\frac{\sqrt{{\bar{\alpha }}_t}{𝒛}_t-{𝒛}_0}{\sqrt{1-{\bar{\alpha }}_t}}.& \text{<span class="tml-eqn"></span>}\end{array}$$ The pair $({𝒛}_t,𝒗)$ determines ${𝒛}_0$ and $𝝐$ by the inverse rotation (Param Conversion Inverse)$$\begin{array}{ccc}& {𝒛}_0=\sqrt{{\bar{\alpha }}_t}{𝒛}_t-\sqrt{1-{\bar{\alpha }}_t}𝒗,𝝐=\sqrt{1-{\bar{\alpha }}_t}{𝒛}_t+\sqrt{{\bar{\alpha }}_t}𝒗,& \text{<span class="tml-eqn"></span>}\end{array}$$ which makes the geometry explicit: with $\sqrt{{\bar{\alpha }}_t}=\cos \varphi$ and $\sqrt{1-{\bar{\alpha }}_t}=\sin \varphi$, the map $({𝒛}_0,𝝐)\mapsto ({𝒛}_t,𝒗)$ is a rotation by $\varphi$ in the plane they span, and $𝒗$ is the derivative of ${𝒛}_t$ with respect to $\varphi$ - hence the name “velocity”.

Training Audio Diffusion Models

The preceding sections described what audio diffusion models predict (Section Diffusion in Audio Latent Spaces) and the architectures that implement these predictions (Section Diffusion Architectures for Audio). This section addresses the equally important question of how these models are trained. Training an audio diffusion model is considerably more than minimising the squared-error loss from (EPS Param): in practice, systems employ composite losses that combine multiple objectives, face data challenges unique to the audio domain, and require careful stabilisation techniques to prevent training collapse. We develop each of these topics in turn.

Composite Training Losses

While the core diffusion objective ((Audioldm LOSS)) provides the primary training signal, high-quality audio generation systems augment it with auxiliary losses that encode perceptual and structural priors about audio.

Multi-resolution STFT loss in detail.

The multi-resolution STFT loss is particularly important for audio because it operates directly in the time-frequency domain where human perception is most sensitive. By summing over multiple STFT configurations, it captures perceptual quality at different temporal and spectral resolutions.

Data Challenges in Audio Diffusion

Training audio diffusion models confronts several data challenges that do not arise (or are less severe) in image generation.

Scarcity of paired data.

The largest paired text-to-image datasets contain billions of image-caption pairs (e.g., LAION-5B). By contrast, the largest public audio-caption datasets contain at most hundreds of thousands of pairs. AudioCaps contains approximately 46,000 10-second clips with human-written captions; Clotho contains approximately 6,000 clips, each with five captions. Even with data augmentation (e.g., MusicLDM's beat-synchronous mixup from Definition 35), the total training data for audio models is orders of magnitude smaller than for image models.

Variable audio lengths and silence.

Audio clips in training datasets have highly variable durations (from 1 second to several minutes), and many clips contain significant silence or low-energy segments. Naive training on fixed-length crops can result in a large fraction of training examples being predominantly silent, wasting model capacity on learning the distribution of silence.

Label noise in audio captions.

Audio captions are notoriously noisy. A clip described as “a cat meowing” may also contain background traffic, air conditioning hum, and room reverberation not mentioned in the caption. This label noise directly affects the diffusion model's ability to learn accurate text-to-audio correspondences.

Training Stability and Loss Balancing

When training with composite losses (Definition 41), the relative magnitudes of the different loss terms can vary by several orders of magnitude, leading to training instability. The diffusion denoising loss ${ℒ}_{\text{diff}}$ may have typical values in $[0.01,0.1]$, while the multi-resolution STFT loss ${ℒ}_{\text{MR-STFT}}$ can range from $[1,100]$ depending on the STFT configuration. Without careful balancing, one loss term dominates the gradient signal, and the model effectively ignores the others.

Learning rate warmup and scheduling.

Audio diffusion models are sensitive to the learning rate schedule, particularly during the early stages of training when the model's predictions are far from the target distribution. A common recipe is:

1. Linear warmup over the first 1,000 to 5,000 steps, ramping the learning rate from $0$ to ${\eta }_{\max }={10}^{-4}$.

2. Cosine decay from ${\eta }_{\max }$ to ${\eta }_{\min }={10}^{-6}$ over the remaining training steps.

3. EMA of model weights: maintain an exponential moving average of the model parameters with decay rate $\mu =0.9999$, and use the EMA weights for evaluation and inference.

The EMA weights produce smoother, higher-quality outputs than the raw training weights, particularly for music generation where subtle timbral consistency is important.

The Complete Training Algorithm

We now consolidate the components from the preceding subsections into a complete training algorithm for latent audio diffusion.

Classifier-Free Guidance for Audio

Classifier-free guidance (CFG), first introduced in image diffusion and studied in Classifier-Free Guidance for Audio, is essential for high-quality audio generation. Here we examine the specific training and inference considerations for CFG in the audio setting.

During training, the conditioning embedding ${\widehat{𝒄}}_j$ is replaced with a null embedding ${𝒄}_{⌀}$ with probability $p_{\text{drop}}$. This trains the model to produce both conditional and unconditional outputs. At inference, the guided prediction is (CFG Inference)$$\begin{array}{ccc}& {\widehat{𝒚}}_{\text{guided}}=(1+w)\widehat{𝒚}({𝒛}_t,t,𝒄)-w\widehat{𝒚}({𝒛}_t,t,{𝒄}_{⌀}),& \text{<span class="tml-eqn"></span>}\end{array}$$ where $w\ge 0$ is the guidance scale.

Exercises

Flow Matching for Music Generation

The diffusion models studied in the preceding sections learn to reverse a stochastic noising process: training teaches a network to predict the noise added at each timestep, and generation proceeds by iteratively denoising a sample drawn from pure Gaussian noise. While this framework has produced remarkable audio and music generation systems, it carries an inherent inefficiency. The stochastic trajectories connecting noise to data are highly curved in the latent space, requiring many small denoising steps for accurate traversal. Flow matching, introduced in 18 and connected to optimal transport in 14, offers a compelling alternative: learn a deterministic velocity field that transports noise to data along straighter paths, enabling fewer integration steps and simpler training objectives.

In this section we develop the theory and practice of flow matching for audio and music generation. We begin by recalling the essential ideas from continuous normalising flows and explaining why straighter paths translate directly to faster audio synthesis (From Diffusion to Flow Matching). We then examine two influential systems: MusicFlow, which cascades flow matching models from text to semantic to acoustic tokens (MusicFlow: Cascaded Flow Matching), and MelodyFlow, which adds melody conditioning in a non-autoregressive framework (MelodyFlow: Melody-Conditioned Flow Matching). Finally, we study optimal transport paths tailored to audio latent spaces and the reflow procedure that further straightens learned trajectories (Optimal Transport Paths for Audio).

From Diffusion to Flow Matching

Recall from 18 that a continuous normalising flow (CNF) defines a time-dependent velocity field $v_{\theta }:{\mathrm{ℝ}}^d\times [\mathrm{0,1}]\to {\mathrm{ℝ}}^d$ whose integral curves transport a source distribution $p_0$ to a target distribution $p_1$. Given an initial point ${𝒛}_0\sim p_0$, the generated sample is (CNF ODE)$$\begin{array}{ccc}& {𝒛}_1={𝒛}_0+{\int }_0^1{v}_{\theta }({𝒛}_t,t)\mathrm{d}t,& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${𝒛}_t$ follows the ODE $\mathrm{d}{𝒛}_t/\mathrm{d}t=v_{\theta }({𝒛}_t,t)$. In audio generation, $p_0=\mathrm{Normal}(0,𝐈)$ is a standard Gaussian over the latent space of an audio codec or spectrogram encoder, and $p_1$ is the distribution of clean audio latents.

The flow matching training objective avoids the expensive simulation required by maximum-likelihood training of CNFs. Instead, it constructs conditional probability paths $p_t(𝒛|{𝒛}_0,{𝒛}_1)$ that interpolate between the source and target, and trains $v_{\theta }$ to match the conditional velocity field of these paths. The simplest choice is the linear interpolation path: (Linear Interp)$$\begin{array}{ccc}& {𝒛}_t=(1-t){𝒛}_0+t{𝒛}_1,t\in [0,1],& \text{<span class="tml-eqn"></span>}\end{array}$$ whose conditional velocity field is simply the displacement ${𝒛}_1-{𝒛}_0$, independent of $t$.

To make the comparison concrete, consider the number of function evaluations (NFE) needed to generate a 10-second audio clip at acceptable quality. A typical diffusion model requires 50 to 200 NFE with a DDPM-style sampler, or 20 to 50 NFE with an accelerated sampler such as DPM-Solver. A flow matching model with near-linear trajectories can achieve comparable quality with 5 to 20 NFE, representing a 5 to 10$\times$ speedup in wall-clock generation time.

Gaussian-to-data paths and the audio latent geometry.

Audio latent spaces exhibit structure that is atypical of natural image latent spaces: strong temporal correlations (adjacent latent frames encode overlapping waveform windows), a near-periodic substructure induced by musical rhythm, and highly non-isotropic marginals arising from the varying energy distribution across frequency bands. These properties have two consequences for flow matching. First, the linear interpolation (Linear Interp) may not be the most efficient path, because the intermediate points ${𝒛}_t$ for $t\in (\mathrm{0,1})$ may lie far from the data manifold. Second, the velocity field learned by the flow model inherits the temporal structure of the audio, so transformer architectures with causal or windowed attention along the time axis are natural choices for parameterising $v_{\theta }$.

MusicFlow: Cascaded Flow Matching

MusicFlow applies the flow matching framework to music generation through a cascaded architecture that mirrors the semantic-to-acoustic pipeline of earlier diffusion-based systems, but replaces the diffusion backbone with conditional flow matching at each stage. The cascade consists of two stages:

1. Text $\to$ Semantic. A flow matching model $v_{\theta }^{(\text{sem})}$ maps Gaussian noise to semantic token embeddings, conditioned on a text embedding ${𝒄}_{\text{text}}$ obtained from a pretrained language model (e.g., T5 or CLAP).

2. Semantic $\to$ Acoustic. A second flow matching model $v_{\theta }^{(\text{ac})}$ maps Gaussian noise to acoustic latents (e.g., EnCodec or DAC embeddings), conditioned on the semantic tokens generated in stage (i).

The two stages of MusicFlow are trained independently using (FM LOSS) with different conditioning signals. At inference, text is first encoded by the language model, then passed through the semantic flow model to produce semantic tokens, which in turn condition the acoustic flow model. Both stages use an Euler ODE solver with a small number of steps (typically 10 to 25).

Optimal transport conditional paths.

The linear interpolation in (Linear Interp) pairs each noise sample ${𝒛}_0$ with a data sample ${𝒛}_1$ uniformly at random. This independent coupling can produce crossing paths: two distinct pairs $({𝒛}_0^{(a)},{𝒛}_1^{(a)})$ and $({𝒛}_0^{(b)},{𝒛}_1^{(b)})$ may have trajectories ${𝒛}_t^{(a)}$ and ${𝒛}_t^{(b)}$ that intersect, forcing the velocity field to be multi-valued at the intersection point. Such crossings increase the curvature of the learned velocity field and necessitate more integration steps.

MusicFlow mitigates this by using optimal transport (OT) conditional paths, as described in 14. Within each training minibatch of size $B$, the $B$ noise samples and $B$ data samples are paired by solving a discrete optimal transport problem: (OT Coupling)$$\begin{array}{ccc}& {\pi }^{\ast }={\mathrm{arg\hspace{0.17em}min}}_{\pi \in \mathrm{\Pi }({𝒛}_0^{1:B},{𝒛}_1^{1:B})}\sum _{i,j}{\pi }_{ij}\Vert {𝒛}_0^{(i)}-{𝒛}_1^{(j)}{\Vert }^2,& \text{<span class="tml-eqn"></span>}\end{array}$$ where $\mathrm{\Pi }$ denotes the set of permutation matrices (or, more generally, doubly stochastic matrices). The OT-paired paths are less likely to cross, yielding a smoother velocity field that can be integrated with fewer steps.

MelodyFlow: Melody-Conditioned Flow Matching

While MusicFlow demonstrates the viability of flow matching for text-to-music generation, many creative applications require finer control over the generated output. MelodyFlow extends the flow matching paradigm to accept melody conditioning: a reference pitch contour or hummed melody that guides the harmonic content of the generated music, while text controls higher-level attributes such as genre, instrumentation, and mood.

A distinguishing feature of MelodyFlow is its non-autoregressive architecture. Unlike the autoregressive token-based systems studied in earlier sections, MelodyFlow generates the entire audio segment in parallel by solving the ODE (CNF ODE) from $t=0$ to $t=1$. This parallel generation naturally accommodates global conditioning signals (such as a melody contour spanning the full duration) without the left-to-right bottleneck of autoregressive models.

Conditioning mechanism.

The melody conditioning signal ${𝒄}_{\text{mel}}\in {\mathrm{ℝ}}^{T^{\prime }\times d_{\text{mel}}}$ is a time-frequency representation of the reference melody, typically a chromagram or a pitch-class profile extracted by a pretrained pitch tracker. This is combined with the text embedding ${𝒄}_{\text{text}}\in {\mathrm{ℝ}}^{d_{\text{text}}}$ through cross-attention layers in the flow model's transformer backbone: (Melodyflow Attention)$$\begin{array}{ccc}& {𝒉}_{\ell }={𝒉}_{\ell -1}+\mathrm{CrossAttn}({𝒉}_{\ell -1},{𝒄}_{\text{text}})+\mathrm{CrossAttn}({𝒉}_{\ell -1},{𝒄}_{\text{mel}}),& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${𝒉}_{\ell }$ is the hidden state at layer $\ell$ of the transformer. The two cross-attention modules operate independently, allowing the model to attend to text and melody information at different spatial and temporal scales.

Classifier-free guidance with dual conditions.

When both text and melody conditions are present, MelodyFlow applies a two-scale classifier-free guidance scheme. During training, the text condition is dropped independently with probability $p_{\text{drop}}^{(\text{text})}$ and the melody condition with probability $p_{\text{drop}}^{(\text{mel})}$. At inference, the guided velocity combines three predictions: (DUAL CFG)$$\begin{array}{ccc}& \begin{array}{cc}{\tilde{v}}_{\theta }({𝒛}_t,t)& =v_{\theta }({𝒛}_t,t,⌀,⌀)\\ & +w_{\text{text}}(v_{\theta }({𝒛}_t,t,{𝒄}_{\text{text}},⌀)-v_{\theta }({𝒛}_t,t,⌀,⌀))\\ & +w_{\text{mel}}(v_{\theta }({𝒛}_t,t,{𝒄}_{\text{text}},{𝒄}_{\text{mel}})-v_{\theta }({𝒛}_t,t,{𝒄}_{\text{text}},⌀)),\end{array}& \text{<span class="tml-eqn"></span>}\end{array}$$ where $w_{\text{text}}$ and $w_{\text{mel}}$ are separate guidance scales. Setting $w_{\text{mel}}=0$ recovers text-only generation. Setting $w_{\text{text}}=0$ with $w_{\text{mel}}>0$ removes the guidance on the text condition, but not the text itself: both evaluations in the melody term, $v_{\theta }({𝒛}_t,t,{𝒄}_{\text{text}},{𝒄}_{\text{mel}})$ and $v_{\theta }({𝒛}_t,t,{𝒄}_{\text{text}},⌀)$, still receive ${𝒄}_{\text{text}}$, so the text prompt continues to shape the output. Melody-only generation requires dropping ${𝒄}_{\text{text}}$ from the conditioning arguments as well, that is, replacing the melody term by $v_{\theta }({𝒛}_t,t,⌀,{𝒄}_{\text{mel}})-v_{\theta }({𝒛}_t,t,⌀,⌀)$.

Optimal Transport Paths for Audio

The OT pairing described in MusicFlow: Cascaded Flow Matching operates within minibatches and provides only an approximation to the true optimal transport map between the noise and data distributions. In this subsection, we develop the theory of rectified flows for audio, which iteratively straighten the learned trajectories toward the true OT map, achieving near-linear paths that can be traversed in very few steps.

The key insight, due to Liu et al. (2023), is that the reflow procedure takes a trained flow model and uses its own generated trajectories as training data for a new model, whose paths are provably straighter.

Acceleration: Consistency and Distillation

Flow matching reduces the number of ODE steps needed for audio generation from dozens to a handful. But even 5 or 10 neural network forward passes may be too many for applications that demand true real-time performance: live music accompaniment, interactive sound design, streaming text-to-speech, or in-game audio synthesis. These applications require generating audio faster than it is consumed, with latency measured in milliseconds rather than seconds.

In this section we study two families of techniques that push audio generation toward the single-step regime: consistency models, which learn to map any point along a diffusion or flow trajectory directly to the trajectory's endpoint, and progressive distillation, which iteratively halves the number of required steps by training a student model to match a teacher's two-step output in a single step.

The Real-Time Imperative

For non-interactive applications (e.g., generating a soundtrack for a video), RTF $\ll 1$ is desirable but not strictly required. For interactive applications, the constraint is more nuanced: the system must produce the next chunk of audio before the current chunk finishes playing. If the chunk size is $L$ samples at sample rate $f_s$, the maximum allowable latency per chunk is $T_{\text{max}}=L/f_s$. For a chunk of 1024 samples at 24,kHz, this is approximately 43,ms.

Decomposing generation latency.

The total generation time for an audio diffusion or flow model can be decomposed as (Latency Decomp)$$\begin{array}{ccc}& T_{\text{compute}}=N_{\text{steps}}\cdot T_{\text{fwd}}+T_{\text{cond}}+T_{\text{decode}},& \text{<span class="tml-eqn"></span>}\end{array}$$ where $N_{\text{steps}}$ is the number of denoising or ODE steps, $T_{\text{fwd}}$ is the time for a single forward pass of the backbone network, $T_{\text{cond}}$ is the time to compute conditioning embeddings (text encoder, CLAP, etc.), and $T_{\text{decode}}$ is the time for the audio codec decoder. The acceleration techniques in this section target $N_{\text{steps}}$; complementary approaches (model pruning, quantisation, efficient attention) target $T_{\text{fwd}}$. In practice, the product $N_{\text{steps}}\cdot T_{\text{fwd}}$ dominates, so reducing $N_{\text{steps}}$ from 50 to 1 yields a near-50$\times$ end-to-end speedup.

AudioLCM: Consistency Distillation for Audio

Consistency models, introduced by Song et al. (2023), learn a function $f_{\theta }({𝒛}_t,t)$ that maps any point ${𝒛}_t$ along the probability flow ODE trajectory to that trajectory's clean endpoint ${𝒛}_{ϵ}\approx {𝒛}_{\text{data}}$. The defining property is self-consistency: for any two points ${𝒛}_s$ and ${𝒛}_t$ on the same trajectory, $f_{\theta }({𝒛}_s,s)=f_{\theta }({𝒛}_t,t)$. AudioLCM adapts this framework to latent audio diffusion models, achieving dramatic speedups.

The boundary condition (i) ensures that the function acts as the identity near $t=ϵ$, while self-consistency (ii) ensures that applying $f_{\theta }$ at any timestep yields the same clean output. In practice, $f_{\theta }$ is parameterised using the pretrained diffusion model's architecture with a skip connection that enforces the boundary condition: (Consistency Param)$$\begin{array}{ccc}& f_{\theta }({𝒛}_t,t)=c_{\text{skip}}(t){𝒛}_t+c_{\text{out}}(t)F_{\theta }({𝒛}_t,t,𝒄),& \text{<span class="tml-eqn"></span>}\end{array}$$ where $c_{\text{skip}}(ϵ)=1$, $c_{\text{out}}(ϵ)=0$, and $F_{\theta }$ is the base network.

Consistency distillation training.

The consistency function is trained by distilling the pretrained diffusion (teacher) model. Given a pair of timesteps $t_{n+1}>t_n$ and a noisy sample ${𝒛}_{t_{n+1}}$, the teacher model performs one ODE step to estimate ${\widehat{𝒛}}_{t_n}$. The consistency distillation loss then enforces self-consistency: (CD LOSS)$$\begin{array}{ccc}& {ℒ}_{\text{CD}}={𝔼}_{n,{𝒛}_{t_{n+1}}}\left[d\right(f_{\theta }({𝒛}_{t_{n+1}},t_{n+1}),f_{{\theta }^{-}}({\widehat{𝒛}}_{t_n},t_n)\left)\right],& \text{<span class="tml-eqn"></span>}\end{array}$$ where $d(\cdot ,\cdot )$ is a distance metric (typically the LPIPS perceptual distance adapted for spectrograms, or a combination of $L^1$ and $\mathrm{mel}$-spectrogram losses), ${\widehat{𝒛}}_{t_n}$ is the teacher's one-step estimate, and ${\theta }^{-}$ denotes the exponential moving average (EMA) of the student's parameters (the “target network”, updated as ${\theta }^{-}\leftarrow \mu {\theta }^{-}+(1-\mu )\theta$).

ConsistencyTTA: Single-Step Text-to-Audio

ConsistencyTTA pushes the consistency model paradigm to its logical extreme: generating audio in a single forward pass. While AudioLCM achieves near-single-step generation through distillation from a pretrained teacher, ConsistencyTTA explores consistency training (CT), where the consistency property is enforced directly during training without a separate teacher model.

Quality-speed trade-offs.

Single-step generation fundamentally changes the quality-speed landscape. In multi-step models, generation quality improves monotonically with the number of steps (up to the teacher's quality ceiling). In single-step models, there is no such knob to turn at inference time; all quality must be baked into the model at training time. This creates a different set of design trade-offs:

1. Model capacity. Single-step models typically require larger networks than their multi-step counterparts to compensate for the loss of iterative refinement. A 1B-parameter single-step model may match the quality of a 400M-parameter 50-step model.

2. Training cost. Consistency training requires careful annealing of the timestep schedule during training and is more sensitive to hyperparameters than standard diffusion training. The total training compute is typically 1.5 to 3$\times$ that of the corresponding multi-step model.

3. Diversity. Single-step models tend to produce less diverse outputs than multi-step models, because the stochastic exploration that occurs during iterative sampling is absent. This manifests as a slight reduction in the range of timbres, rhythmic patterns, and harmonic progressions observed across multiple generations from the same text prompt.

4. Guidance integration. Classifier-free guidance, which requires evaluating the model twice per step (conditional and unconditional), doubles the cost of each step. For single-step models, this overhead is absolute: the model must either forgo guidance or use distilled guidance (where the guidance signal is absorbed into the model weights during training).

Progressive Distillation for Audio

Progressive distillation offers an alternative path to few-step audio generation that is conceptually simpler than consistency training. The idea is to iteratively halve the number of required sampling steps by training a student model to match the output of two teacher steps in a single student step.

Long-Form Audio and Music Generation

The audio and music generation systems studied so far produce segments of fixed, modest duration: typically 5 to 30 seconds. This window is dictated by the model's context length (for autoregressive systems) or the latent tensor size (for diffusion and flow models), both of which are constrained by GPU memory and the quadratic cost of attention. Yet the audio that humans actually want to generate, a complete song, a podcast episode, a film soundtrack, spans minutes or hours. Bridging the gap between a model's native generation window and the durations demanded by real applications is one of the central open challenges in audio generation.

This section analyses the long-form challenge from both theoretical and practical perspectives. We begin by characterising the hierarchical structure of music and audio (The Long-Form Challenge), then examine hierarchical generation approaches (Hierarchical Approaches), autoregressive chunking with overlap (Autoregressive Chunking with Overlap), and song-level structure modelling (Song-Level Structure Modelling). The reader may find it helpful to compare these approaches with the analogous strategies for long video generation discussed in Long Video Generation.

The Long-Form Challenge

Music possesses a natural hierarchical structure spanning multiple timescales. Understanding these timescales is essential for designing systems that can generate coherent long-form content.

Current generation models with a 10-second window operate comfortably at the phrase level but struggle at the section level and fail entirely at the song level. The fundamental difficulty is that coherence at higher levels requires information about events that occurred many seconds or minutes in the past.

Memory and compute scaling.

The computational cost of extending the generation window depends on the model architecture. For a transformer with full self-attention over $L$ tokens, the memory scales as $O(L^2)$ and the compute per step as $O(L^{2}d)$, where $d$ is the model dimension. Generating a 4-minute song at 24,kHz with compression factor 320 requires $L=\mathrm{18,000}$ tokens. Compared to a 10-second window ($L=750$), the sequence is exactly $\mathrm{18,000}/750=24$ times longer, so this represents a ${24}^2=576\times$ increase in attention memory and compute. Even with linear-attention variants or sparse attention patterns, the scaling remains at least $O(L\log L)$, making brute-force extension of the generation window impractical for song-length audio.

Hierarchical Approaches

One strategy for overcoming the context-length bottleneck is to generate music at multiple levels of the timescale hierarchy, using the output of coarser levels to condition finer levels. This mirrors the hierarchical approach used in long video generation (see Long Video Generation) and in cascade-based image generation.

A reference design.

The following is a composite of how hierarchical systems are typically built, not a description of any one published system. The structure model is a small transformer that generates a sequence of “structure tokens” at a rate of approximately one token per bar (roughly 2 seconds of music). Each structure token encodes a low-dimensional embedding of the harmonic, rhythmic, and timbral characteristics of the corresponding bar. The structure tokens are obtained during training by applying a learned quantiser to bar-level summaries extracted from the training data.

The audio model is a larger diffusion transformer that generates the full-resolution audio latent conditioned on the structure tokens via cross-attention. Because the structure tokens span the entire song, the audio model has access to global structural information even when it generates individual chunks.

Autoregressive Chunking with Overlap

The most widely used approach to long-form audio generation is conceptually simple: generate the audio one chunk at a time, conditioning each new chunk on the tail of the previous chunk to ensure continuity at the boundary. This mirrors the autoregressive chunk generation strategy for long video (see Long Video Generation), adapted to the particular demands of audio coherence.

Let ${𝒙}^{(k)}\in {\mathrm{ℝ}}^L$ denote the $k$-th chunk of audio (in the latent space, with $L$ tokens per chunk), and let $\delta \in \{1,\dots ,L-1\}$ be the overlap parameter. The generation factorises as (AR Chunk)$$\begin{array}{ccc}& p({𝒙}^{1:K})=p({𝒙}^{(1)})\prod _{k=2}^{K}p({𝒙}^{(k)}|{𝒙}_{L-\delta :L}^{(k-1)}),& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${𝒙}_{L-\delta :L}^{(k-1)}$ denotes the last $\delta$ tokens of chunk $k-1$, and $K=\lceil (N-\delta )/(L-\delta )\rceil$ is the total number of chunks for an $N$-token output.

Cross-fading.

The overlap region between consecutive chunks is generated twice: once as the tail of chunk $k-1$ and once as the head of chunk $k$. To avoid discontinuities, the two versions are blended using a cross-fade: (Crossfade)$$\begin{array}{ccc}& {\widehat{𝒙}}_n=\{\begin{array}{cc}{𝒙}_n^{(k-1)}& \text{if }n<L-\delta ,\\ (1-{\alpha }_n){𝒙}_n^{(k-1)}+{\alpha }_n{𝒙}_{n-(L-\delta )}^{(k)}& \text{if }L-\delta \le n<L,\\ {𝒙}_{n-(L-\delta )}^{(k)}& \text{if }n\ge L,\end{array}& \text{<span class="tml-eqn"></span>}\end{array}$$ where $n$ indexes the output stream aligned to chunk $k-1$ and ${\alpha }_n=(n-(L-\delta ))/\delta$ is a linear blend weight. Note the index shift: the overlap region occupies positions $L-\delta ,\dots ,L-1$ of chunk $k-1$ but positions $1,\dots ,\delta$ of chunk $k$, since each chunk is generated in its own local coordinates. Writing ${𝒙}_n^{(k)}$ in the blend would compare the tail of one chunk against the tail of the next. More sophisticated cross-fade functions (e.g., raised cosine, equal power) change the trade-off at the seam; see Exercise 17.

Temporal inpainting for chunk boundaries.

An alternative to cross-fading is temporal inpainting: the model generates the overlap region conditioned on both the tail of the previous chunk and the head of the next chunk (or a preliminary version of it). This produces a smoother transition because the model can “see” both sides of the boundary. The inpainting formulation treats the non-overlap regions as observed and the overlap region as latent: (Inpainting)$$\begin{array}{ccc}& p({𝒙}_{\text{overlap}}|{𝒙}_{\text{left}},{𝒙}_{\text{right}})\propto p({𝒙}_{\text{left}},{𝒙}_{\text{overlap}},{𝒙}_{\text{right}}),& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${𝒙}_{\text{left}}$ and ${𝒙}_{\text{right}}$ are the observed segments. In practice, this posterior is approximated using the Replace-then-Denoise strategy: at each denoising step, the known regions are replaced with their noised versions, and only the overlap region is updated by the model.

Song-Level Structure Modelling

Neither hierarchical generation nor autoregressive chunking directly models the large-scale form of a song. Musical form, the arrangement of sections into a coherent whole, is a defining feature of composed music, distinguishing it from aimless improvisation. Common forms include:

• AABA form: Two statements of a theme (A), a contrasting bridge (B), and a return to the theme (A). Common in jazz standards and Tin Pan Alley songs.

• Verse–chorus form: Alternating verses (new lyrics over similar music) and a recurring chorus (fixed lyrics and melody). Dominant in pop and rock music.

• Rondo form: A principal theme alternates with contrasting episodes: ABACADA. Common in classical music.

• Through-composed: No repeated sections; the music evolves continuously. Common in art songs and some film scores.

Self-similarity matrices.

A powerful tool for analysing and imposing musical form is the self-similarity matrix (SSM). Given a sequence of feature vectors ${\{{𝒇}_t\}}_{t=1}^T$ extracted from an audio recording (or generated audio), the SSM is the $T\times T$ matrix $𝐒$ with entries (SSM)$$\begin{array}{ccc}& S_{ij}=\kappa ({𝒇}_i,{𝒇}_j),& \text{<span class="tml-eqn"></span>}\end{array}$$ where $\kappa$ is a similarity kernel (e.g., cosine similarity, RBF kernel). For music with clear sectional structure, the SSM exhibits a block-diagonal pattern: elements within the same section have high similarity, and repeated sections (e.g., two occurrences of the chorus) produce off-diagonal blocks of high similarity.

Multi-Track Generation and Source Separation

Music, in its most natural form, is not a monolithic waveform but a superposition of individual stems: drums, bass, vocals, keyboards, guitars, strings, and other instruments. Professional music production treats each stem separately, adjusting its volume, panning, equalization, and effects before combining the stems into a final mix. This section studies the generative modelling of multi-track music and its dual problem, source separation, from a unified probabilistic perspective.

Multi-Track Generation

Multi-track generation aims to produce a set of $K$ coherent stems $\{{𝒙}_1,\dots ,{𝒙}_K\}$ that sound musically meaningful both individually and when mixed together. The simplest approach is to model the joint distribution $p({𝒙}_1,\dots ,{𝒙}_K|𝒄)$ directly, treating the multi-track audio as a single high-dimensional object. More practical approaches decompose this joint distribution into manageable factors.

Joint generation.

The joint approach concatenates or interleaves the stems along the channel dimension: $𝑿=[{𝒙}_1;{𝒙}_2;\dots ;{𝒙}_K]\in {\mathrm{ℝ}}^{K\times T}$ is treated as a $K$-channel audio signal. A diffusion or flow model then generates $𝑿$ as if it were a single multi-channel audio. The advantage is that the model can capture arbitrary dependencies between stems (e.g., the bass following the kick drum, the vocals interacting with the harmonic progression). The disadvantage is the increased dimensionality: for $K=4$ stems, the latent space is 4 times larger, increasing memory and compute requirements by roughly the same factor.

Sequential generation.

The sequential approach generates stems in a fixed order (e.g., drums $\to$ bass $\to$ harmony $\to$ vocals), conditioning each new stem on the previously generated ones. This reduces the per-step dimensionality to that of a single stem but introduces an order dependence: the first stem receives no musical context from the others, which can lead to suboptimal results if the “most important” stem (often vocals or melody) is not generated first.

Source Separation as Inverse Generation

Source separation, the task of extracting individual stems ${𝒙}_1,\dots ,{𝒙}_K$ from a mixture ${𝒙}_{\text{mix}}={\sum }_{k=1}^K{𝒙}_k$, is the inverse problem dual to multi-track generation. A generative model that can produce plausible stems provides a natural prior for this inverse problem, leading to a Bayesian formulation of source separation.

Diffusion-based source separation.

The posterior in (Separation Posterior) is intractable in general, but it can be approximated using the machinery of diffusion models. The key idea is to run the reverse diffusion process for each source ${𝒙}_k$ while enforcing the mixture constraint at each denoising step. Concretely, at each timestep $t$ of the reverse process:

1. Predict the clean estimate ${\widehat{𝒙}}_k^{(t)}$ for each source $k$ using source-specific diffusion models (or a shared model with source-class conditioning).

2. Project the estimates onto the mixture constraint: replace $\{{\widehat{𝒙}}_k^{(t)}\}$ with the nearest set of signals (in $L^2$ norm) that sum to ${𝒙}_{\text{mix}}$: (Mixture Projection)$$\begin{array}{ccc}& {\widetilde{𝒙}}_k^{(t)}={\widehat{𝒙}}_k^{(t)}+\frac{1}{K}({𝒙}_{\text{mix}}-\sum _{j=1}^K{\widehat{𝒙}}_j^{(t)}).& \text{<span class="tml-eqn"></span>}\end{array}$$ This additive correction distributes the residual equally among all sources.

3. Continue the reverse diffusion from the projected estimates.

Style Transfer and Timbre Control

Beyond generating audio from text, many applications require transforming existing audio: changing the instrumentation of a piano recording to an orchestral arrangement, transferring the “feel” of one drum pattern to another, or converting a speaking voice to a singing voice while preserving the linguistic content. These tasks fall under the broad umbrella of audio style transfer.

Timbre transfer.

Timbre, often described as the “colour” of a sound, is the perceptual quality that distinguishes two notes of the same pitch and loudness played by different instruments. Formally, timbre encompasses all spectral and temporal characteristics not captured by pitch, loudness, and duration. Timbre transfer aims to change the timbre of an audio signal while preserving its pitch, rhythm, and temporal structure.

In the diffusion/flow framework, timbre transfer is implemented as constrained sampling: the model generates audio that matches a target timbre distribution while remaining close to the source audio in a content-preserving feature space.

Genre transfer.

Genre transfer generalises timbre transfer to higher-level musical attributes: converting a jazz piano trio performance into a rock band arrangement, or transforming an electronic dance track into an acoustic folk rendition. This requires changing not only the timbres but also the rhythmic patterns, harmonic voicings, and production style. The constrained sampling framework in (Constrained Style) applies, but the content distance $d_{\text{content}}$ must preserve only the most abstract features (overall harmonic progression, melodic contour, song structure) while allowing the model freedom to modify lower-level details.

Cross-Modal Generation

The previous sections addressed audio and music generation conditioned on text prompts, tags, or melodic hints. In many practical applications, however, the conditioning signal originates from an entirely different modality: a video stream that requires a matching soundtrack, a set of lyrics that must be sung with correct timing, or an existing audio clip that must be extended, inpainted, or super-resolved. Each of these scenarios demands architectures that bridge representational gaps between modalities while preserving the fine-grained temporal structure that characterises audio.

This section formalises three families of cross-modal audio generation: video-to-music (Video-to-Music Generation), lyrics-conditioned singing synthesis (Lyrics-Conditioned Generation), and audio-to-audio tasks such as inpainting and editing (Audio-to-Audio: Inpainting and Editing).

Video-to-Music Generation

A background music track must reflect the visual content, the emotional tone, and the temporal dynamics of its accompanying video. We begin with a precise problem statement.

Visual feature extraction.

The visual encoder ${\varphi }_v:{\mathrm{ℝ}}^{H\times W\times 3}\to {\mathrm{ℝ}}^{d_v}$ is typically a pretrained vision transformer (e.g., CLIP ViT-L/14 or DINOv2). For a video of $F$ frames, we obtain $𝐕=({\varphi }_v({𝐈}_1),\dots ,{\varphi }_v({𝐈}_F))$. Two strategies dominate the literature:

1. Frame-level features. Each frame is encoded independently, yielding $𝐕\in {\mathrm{ℝ}}^{F\times d_v}$. This preserves fine-grained temporal information but ignores inter-frame motion.

2. Clip-level features. A video encoder such as VideoMAE or TimeSformer processes short clips (e.g., 16 frames) and produces a single vector per clip. This captures motion but reduces temporal resolution.

In practice, Diff-BGM [26] and VidMuse [27] combine both: frame-level features provide a dense temporal signal, while a clip-level summary vector captures global scene semantics.

Temporal alignment mechanisms.

The mismatch between video frame rate ($r_v$, typically 24 fps) and audio token rate ($r_a$, often 50 Hz for neural codec tokens) necessitates an alignment module. Let $𝐀\in {\mathrm{ℝ}}^{L_a\times F}$ denote a soft alignment matrix, where $L_a=\lceil T{r}_a/f_s\rceil$ is the number of audio tokens. The aligned visual features are (V2M Align)$$\begin{array}{ccc}& \widetilde{𝐕}=𝐀𝐕\in {\mathrm{ℝ}}^{L_a\times d_v},& \text{<span class="tml-eqn"></span>}\end{array}$$ so that each audio token position receives a weighted combination of visual features. Diff-BGM learns $𝐀$ via cross-attention within a latent diffusion model; VidMuse employs a dedicated temporal alignment network that predicts $𝐀$ from rhythm and motion energy signals.

Diff-BGM.

Diff-BGM [26] uses a two-stage approach:

1. Video analysis. A pretrained scene classifier and motion energy estimator produce a scene label sequence and a continuous arousal curve $a(t)\in [0,1]$.

2. Conditional latent diffusion. A U-Net denoiser operates on mel-spectrogram latents $𝒛\in {\mathrm{ℝ}}^{C\times T^{\prime }}$ (where $T^{\prime }$ is the compressed time dimension), conditioned on both the scene label embedding and the arousal curve via cross-attention and FiLM layers.

The training objective follows the standard latent diffusion loss: (Diffbgm LOSS)$$\begin{array}{ccc}& {ℒ}_{\text{Diff-BGM}}={𝔼}_{t,𝝐,{𝒛}_0}\left[{\Vert 𝝐-{𝝐}_{\theta }({𝒛}_t,t,𝐕)\Vert }_2^2\right],& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${𝒛}_t=\sqrt{{\bar{\alpha }}_t}{𝒛}_0+\sqrt{1-{\bar{\alpha }}_t}𝝐$ is the noised latent and $𝐕$ summarises the visual conditioning.

VidMuse.

VidMuse [27] extends the framework to long-form video by introducing a hierarchical conditioning scheme. A global video embedding captures scene-level semantics (genre, mood), while local frame-level embeddings guide temporal details. The generation model is a transformer-based language model over EnCodec tokens, with the visual features injected via cross-attention at every layer: (Vidmuse AR)$$\begin{array}{ccc}& p_{\theta }(𝒙|𝐕)=\prod _{n=1}^N{p}_{\theta }(x_n|x_{<n},\widetilde{𝐕}),& \text{<span class="tml-eqn"></span>}\end{array}$$ where $x_n$ denotes the $n$-th audio token and $\widetilde{𝐕}$ is the temporally aligned visual feature sequence from (V2M Align).

Lyrics-Conditioned Generation

Generating singing voice that matches given lyrics requires solving two intertwined problems: synthesising intelligible vocal content and producing musically coherent melodic and rhythmic patterns. This subsection analyses the conditioning mechanisms, alignment strategies, and the interplay with large-scale music models such as Jukebox (Case Study: Jukebox and Music Generation).

Character-level vs. word-level conditioning.

The choice of text granularity has significant architectural implications:

• Character-level conditioning. Each character (or phoneme) is treated as a separate token, enabling fine-grained control over pronunciation and duration. The conditioning sequence is long, which increases the computational cost of cross-attention but allows the model to learn sub-word timing patterns (e.g., melisma, where a single syllable spans multiple notes).

• Word-level conditioning. Each word is mapped to a single embedding vector, reducing the conditioning sequence length. This simplifies the alignment problem but sacrifices control over intra-word timing. Word-level models typically require a separate duration model to predict how many audio frames correspond to each word.

The monotonicity constraint reflects the fact that, in normal singing, lyrics are produced in order. The duration vector $𝒅=(d_1,\dots ,d_M)\in {\mathrm{\mathbb{Z}}}_{>0}^M$ with ${\sum }_m{d}_m=N$ is the natural parameterisation of the alignment.

Duration modelling.

Given lyrics $𝒍$ and a target melody contour ${𝒇}_0=(f_{\mathrm{0,1}},\dots ,f_{0,N})$ (fundamental frequency), a duration model predicts the duration of each phoneme: (Duration Model)$$\begin{array}{ccc}& 𝒅=g_{\psi }(𝒍,{𝒇}_0)\in {\mathrm{ℝ}}_{>0}^M,& \text{<span class="tml-eqn"></span>}\end{array}$$ where $g_{\psi }$ is typically a small transformer or convolutional network. The predicted durations are rounded and used to expand the phoneme embeddings to the audio frame rate via repetition: (Expand)$$\begin{array}{ccc}& {\tilde{l}}_n=l_{\alpha (n)},\text{where }\alpha (n)=\min \{m:\sum _{j=1}^m{d}_j\ge n\}.& \text{<span class="tml-eqn"></span>}\end{array}$$

Jukebox and VQ-VAE approaches.

Jukebox [28] demonstrated that lyrics can be incorporated as conditioning for a hierarchical VQ-VAE music model. The lyrics are encoded by a small transformer and injected via addition to the audio token embeddings at each level of the hierarchy. While Jukebox produces impressive results, the alignment between lyrics and audio is largely learned implicitly, without an explicit duration model. This leads to occasional misalignment artefacts (skipped words, repeated phrases) that more structured approaches avoid.

Audio-to-Audio: Inpainting and Editing

Audio-to-audio generation encompasses tasks where both the input and output are audio signals: inpainting missing regions, extending existing clips, performing super-resolution, and making localised edits. These tasks are unified by conditioning the generation on a partially observed audio signal.

Diffusion-based inpainting.

In the diffusion framework, inpainting is implemented by constraining the observed region at each denoising step. Let ${𝒛}_t=\sqrt{{\bar{\alpha }}_t}{𝒛}_0+\sqrt{1-{\bar{\alpha }}_t}𝝐$ be the forward diffusion at time $t$. At each reverse step, after computing the denoised estimate ${\widehat{𝒛}}_0$ from the network prediction, we replace the observed region with the corresponding noised observation: (Inpaint Replace)$$\begin{array}{ccc}& {𝒛}_{t-1}^{(n)}=\{\begin{array}{cc}\sqrt{{\bar{\alpha }}_{t-1}}{𝒛}_0^{\text{obs}}+\sqrt{1-{\bar{\alpha }}_{t-1}}{𝝐}_n& \text{if }n\in {\mathrm{\Omega }}_{\text{obs}},\\ {\widehat{𝒛}}_{t-1}^{(n)}& \text{if }n\in {\mathrm{\Omega }}_{\text{miss}},\end{array}& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${\widehat{𝒛}}_{t-1}$ is the standard reverse-step output and ${𝝐}_n\sim \mathrm{Normal}(0,1)$ is fresh noise. This repaint strategy [4] ensures the observed region remains faithful to the input throughout the denoising process.

Audio super-resolution.

Super-resolution is a special case of audio-to-audio generation where ${\mathrm{\Omega }}_{\text{obs}}$ corresponds to the low-frequency content. Given a band-limited signal ${𝒙}_{\text{low}}\in {\mathrm{ℝ}}^T$ with bandwidth $B_{\text{low}}$, the goal is to synthesise the high-frequency band $[B_{\text{low}},B_{\text{high}}]$ conditioned on ${𝒙}_{\text{low}}$. In the frequency domain, this can be formulated as (Superres)$$\begin{array}{ccc}& \hat{X}(f)=\{\begin{array}{cc}{X}_{\text{low}}(f)& \text{if }\left|f\right|\le B_{\text{low}},\\ G_{\theta }(f|X_{\text{low}})& \text{if }{B}_{\text{low}}<\left|f\right|\le B_{\text{high}},\end{array}& \text{<span class="tml-eqn"></span>}\end{array}$$ where $G_{\theta }$ is a generative model for the missing bandwidth.

Audio extension (outpainting).

Audio extension (also called outpainting or continuation) generates new audio that seamlessly continues an existing clip. Formally, given ${𝒙}_{1:T_0}$, we wish to sample ${𝒙}_{T_0+1:T}\sim p({𝒙}_{T_0+1:T}|{𝒙}_{1:T_0},𝒄)$. Autoregressive models handle this naturally by simply continuing the token generation. For diffusion models, the continuation is treated as a right-boundary inpainting problem: the observed region is ${\mathrm{\Omega }}_{\text{obs}}=\{1,\dots ,T_0\}$ and the missing region is ${\mathrm{\Omega }}_{\text{miss}}=\{T_0+1,\dots ,T\}$.

Symbolic Music Generation

All generation methods discussed so far operate on audio waveforms or their spectral representations. An older but complementary tradition represents music symbolically, as sequences of discrete note events. Symbolic representations capture pitch, timing, duration, and velocity with exact precision, at the cost of discarding timbre, room acoustics, and performance nuances. This section covers the dominant symbolic representation (MIDI), transformer-based generation models, and recent diffusion approaches that operate directly on piano-roll matrices.

MIDI and Symbolic Representations

The Musical Instrument Digital Interface (MIDI) standard, introduced in 1983, remains the most widely used symbolic music representation. A MIDI file encodes a piece of music as a sequence of timestamped events.

Piano roll representation.

For computational purposes, it is often convenient to convert a MIDI event sequence into a piano roll: a binary matrix that indicates which pitches are active at each time step.

Alternative symbolic encodings.

Several alternatives to the raw MIDI event sequence have been proposed to improve the efficiency and learnability of symbolic music models:

• REMI (REvamped MIDI-derived) [5]: Introduces explicit bar and position tokens, encoding each note as a sequence of sub-tokens: [Bar], [Position], [Pitch], [Duration], [Velocity]. This makes the bar structure explicit and improves long-range coherence.

• Compound Word (CP) [6]: Groups the sub-tokens of REMI into a single compound token, where each compound token is a concatenation of embeddings for position, pitch, duration, and velocity. This reduces the sequence length by a factor of 4–5 compared to REMI.

• Octuple [7]: Extends the compound word idea to eight attributes (time, bar, position, instrument, pitch, duration, velocity, tempo), enabling multi-track representation in a single token per note.

Transformer Models for Symbolic Music

The application of transformer architectures to symbolic music generation has been remarkably successful. The key insight is that a MIDI token sequence, after appropriate encoding, is structurally similar to a natural language sentence: it is a sequence of discrete tokens with both local dependencies (notes within a chord) and long-range dependencies (harmonic progressions, melodic repetitions, verse-chorus structures). We refer the reader to 17 for the general treatment of autoregressive models.

Music Transformer.

The Music Transformer [29] adapts the transformer decoder architecture for symbolic music generation, with one crucial modification: relative attention.

Standard absolute positional encodings (sinusoidal or learned) add a fixed position-dependent bias to each token. In music, however, the important relationships are often relative: a perfect fifth interval (7 semitones) has the same harmonic character regardless of its absolute pitch, and a rhythmic pattern at the beginning of a phrase has the same metrical function when it recurs four bars later.

Compound word tokenisation.

As noted in the previous subsection, the Compound Word (CP) encoding [6] represents each musical note as a single compound token, where each compound token consists of multiple attribute embeddings that are summed or concatenated: (Compound)$$\begin{array}{ccc}& {𝒆}_n^{\text{CP}}={𝒆}_n^{\text{pos}}+{𝒆}_n^{\text{pitch}}+{𝒆}_n^{\text{dur}}+{𝒆}_n^{\text{vel}},& \text{<span class="tml-eqn"></span>}\end{array}$$ where each ${𝒆}_n^{(\cdot )}\in {\mathrm{ℝ}}^d$ is a learned embedding for the corresponding attribute. The transformer then operates on the compound sequence $({𝒆}_1^{\text{CP}},\dots ,{𝒆}_N^{\text{CP}})$, with separate prediction heads for each attribute at the output.

Longer context and hierarchical generation.

Standard transformer architectures with context window $W$ can capture dependencies spanning at most $W$ tokens. For a CP-encoded piece, $W=2048$ covers approximately 2 minutes of music. To generate longer pieces, several strategies have been explored:

1. Sliding window with memory. The model generates in overlapping windows, carrying forward a compressed memory (e.g., the hidden states of the last $k$ tokens).

2. Hierarchical generation. A high-level model generates a structural plan (e.g., chord progression, section labels), and a low-level model fills in the notes conditioned on this plan. This decomposes long-range structure from local note-level detail.

3. Sparse attention. Attention patterns such as Longformer or BigBird allow the model to attend to distant tokens at reduced computational cost, extending the effective context to tens of thousands of tokens.

Diffusion for Symbolic Music

While autoregressive transformers have been the dominant paradigm for symbolic music, diffusion models offer an attractive alternative: they can generate all time steps in parallel and naturally handle tasks like inpainting and arrangement. The challenge is that the piano roll $𝐏\in {\{\mathrm{0,1}\}}^{T\times 128}$ is a discrete object, whereas standard diffusion operates on continuous spaces.

Challenges of the discrete nature.

The relaxation from $\{\mathrm{0,1}\}$ to $[\mathrm{0,1}]$ introduces a mismatch between the training data (binary piano rolls) and the diffusion process (continuous). Several issues arise:

1. Boundary effects. The Gaussian noise process can push values outside $[\mathrm{0,1}]$, requiring clipping or a sigmoid transformation.

2. Thresholding artefacts. The binarisation step at generation time is non-differentiable and can produce noisy results near the 0.5 threshold.

3. Sparse activations. A typical piano roll is very sparse: at any given time step, at most 10–15 of the 128 pitches are active (and usually fewer than 6). The diffusion model must learn to produce sharp, sparse outputs from a dense Gaussian process.

Architecture considerations.

The piano roll $𝐏\in {\{\mathrm{0,1}\}}^{T\times 128}$ is a two-dimensional structure that can be treated as a single-channel image. This observation motivates the use of 2D U-Net architectures, where the “height” dimension is pitch and the “width” dimension is time. However, the two axes have very different semantics:

• The pitch axis has harmonic structure: notes separated by octaves (12 semitones) have similar function, and common intervals (thirds, fifths, octaves) have characteristic sizes.

• The time axis has rhythmic structure: events are organised around a metrical grid (bars, beats, subdivisions).

Effective architectures exploit these distinct structures through asymmetric kernel sizes, separate pitch and time attention layers, or hybrid convolutional-attention designs.

Commercial and Industrial Systems

The research advances described in previous sections have been translated into commercial products that can generate full-length songs, complete with vocals, lyrics, instrumentation, and mixing. This section surveys the major commercial and industrial systems as of 2024–2025, analyses their known (or inferred) architectures, and provides a comparative evaluation. We emphasise that many of these systems are closed-source, so the architectural details presented here are based on published papers, patent filings, blog posts, and empirical observation.

Google MusicLM / MusicFX

MusicLM [30] was one of the first systems to demonstrate high-quality, long-form text-to-music generation. Its architecture exemplifies the hierarchical token generation paradigm that has become the dominant approach for audio generation at scale.

Architecture overview.

MusicLM uses a three-stage hierarchical generation process:

1. Semantic tokens. A first transformer maps a text description to a sequence of semantic tokens. These tokens are not MuLan outputs: they are obtained by $k$-means quantising the intermediate representations of a self-supervised speech/audio model, w2v-BERT, at a coarse temporal resolution ($\sim$25,Hz), and they capture high-level musical attributes (genre, mood, instrumentation, melodic contour). MuLan [9] plays a different role: it is a joint text-audio embedding model (analogous to CLIP for text-image), trained on $\sim$44M audio-text pairs, whose quantised embedding supplies the conditioning prefix for this stage. At training time the MuLan audio embedding of the target clip is used; at inference time the MuLan text embedding of the prompt is substituted, which is what removes the need for paired text-music training data.

2. Coarse acoustic tokens. A second transformer maps the semantic tokens to coarse acoustic tokens from a SoundStream [10] neural audio codec. These tokens encode the broad spectral shape and harmonic content at $\sim$50,Hz.

3. Fine acoustic tokens. A third transformer (or the same transformer with a different head) refines the coarse tokens by predicting the residual quantisation levels of the codec, capturing fine spectral detail and high-frequency content.

The full generation pipeline can be written as (Musiclm Hierarchy)$$\begin{array}{ccc}& p(𝒙|{𝒄}_{\text{text}})=\underset{\text{semantic stage}}{\underbrace{p(𝒔|{𝒄}_{\text{text}})}}\cdot \underset{\text{coarse acoustic}}{\underbrace{p({𝒂}_c|𝒔)}}\cdot \underset{\text{fine acoustic}}{\underbrace{p({𝒂}_f|{𝒂}_c,𝒔)}}\cdot \underset{\text{codec decoder}}{\underbrace{p(𝒙|{𝒂}_f,{𝒂}_c)}},& \text{<span class="tml-eqn"></span>}\end{array}$$ where $𝒔$ denotes semantic tokens, ${𝒂}_c$ coarse acoustic tokens, ${𝒂}_f$ fine acoustic tokens, and $𝒙$ the output waveform.

MuLan conditioning.

The MuLan model projects both text and audio into a shared embedding space ${\mathrm{ℝ}}^{d_{\text{mu}}}$, trained with a contrastive loss: (Mulan LOSS)$$\begin{array}{ccc}& {ℒ}_{\text{MuLan}}=-\frac{1}{B}\sum _{i=1}^B\log \frac{\exp ({\varphi }_t({𝒄}_i{)}^{\top }{\varphi }_a({𝒙}_i)/\tau )}{{\sum }_{j=1}^B\exp ({\varphi }_t({𝒄}_i{)}^{\top }{\varphi }_a({𝒙}_j)/\tau )},& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${\varphi }_t$ and ${\varphi }_a$ are the text and audio encoders, $B$ is the batch size, and $\tau$ is a temperature parameter. At generation time, the text is encoded by ${\varphi }_t$, and the resulting embedding conditions the semantic token generator.

MusicFX and Gemini-era updates.

Following MusicLM, Google released MusicFX as a user-facing product integrated with the Gemini family of models. Key improvements include:

• Longer generation. MusicFX supports generation of up to 70 seconds (compared to MusicLM's 30 seconds), achieved through sliding-window generation with overlapping context.

• Improved conditioning. The MuLan embedding is supplemented with T5 text embeddings, providing richer semantic conditioning for complex prompts.

• Quality filtering. A MuLan-based quality classifier filters generated candidates, selecting the output with the highest text-audio alignment score. For a batch of $K$ candidates, the selected output is (Quality Filter)$$\begin{array}{ccc}& {𝒙}^{\ast }={\mathrm{arg\hspace{0.17em}max}}_{{𝒙}_k,k=1,\dots ,K}{\varphi }_t{(𝒄)}^{\top }{\varphi }_a({𝒙}_k).& \text{<span class="tml-eqn"></span>}\end{array}$$

Suno

Suno (versions 3 through 4) is a commercial system that generates complete songs with vocals, lyrics, and full instrumentation. It represents a significant leap in scope compared to earlier systems that focused on instrumental music or short clips.

Inferred architecture.

Based on public information and empirical analysis, Suno appears to employ a two-stage architecture:

1. Structure planning. Given a text prompt (or user-provided lyrics), a language model generates a structural plan that specifies: itemize

2. Song structure (intro, verse, chorus, bridge, outro),

3. Lyrical content for each section (if not user-provided),

4. Approximate duration and tempo,

5. Genre and stylistic tags. itemize This stage can be understood as sampling from a conditional distribution over structural descriptions: (SUNO Structure)$$\begin{array}{ccc}& 𝒔\sim p_{{\theta }_1}(𝒔|{𝒄}_{\text{prompt}}),& \text{<span class="tml-eqn"></span>}\end{array}$$ where $𝒔$ encodes the song structure and ${𝒄}_{\text{prompt}}$ is the user's input.

6. Audio synthesis. A second model generates the audio waveform conditioned on the structural plan: (SUNO Synthesis)$$\begin{array}{ccc}& 𝒙\sim p_{{\theta }_2}(𝒙|𝒔).& \text{<span class="tml-eqn"></span>}\end{array}$$ This stage likely combines a transformer-based token generator (for coarse structure) with a diffusion-based refinement stage (for high-fidelity audio). The vocal track and instrumental track may be generated jointly or in separate streams that are mixed in a final step.

Full-song generation.

A distinguishing feature of Suno is its ability to generate songs of 2–4 minutes in length with coherent large-scale structure. This requires:

• Long-range coherence. The model must maintain harmonic, melodic, and rhythmic consistency across the entire song. The structure planning stage addresses this by providing a high-level blueprint.

• Vocal-instrumental alignment. The generated vocals must be synchronised with the instrumental accompaniment, both in timing (lyrics align with the beat) and in pitch (vocals follow the chord progression).

• Dynamic range. Real songs have dynamics (quiet verses, loud choruses). The model must vary energy, density, and spectral content across sections.

Udio

Udio, developed by former members of Google DeepMind, is a competing commercial system that also generates full songs with vocals and instrumentation.

Architecture overview.

Udio's architecture is believed to centre on a diffusion-based generation model operating in a latent space, conditioned on text embeddings from a large language model. Key design choices (inferred from public demonstrations and interviews) include:

• Latent diffusion. Audio is encoded into a compressed latent space via a neural audio codec, and the diffusion process operates in this latent space (similar to the approach of AudioLDM and Stable Audio).

• LLM-based text understanding. The text prompt is processed by a large language model that extracts structured attributes (genre, mood, tempo, instrumentation) before conditioning the diffusion model.

• Iterative refinement. The generation may involve multiple passes: a first pass generates a draft, and subsequent passes refine specific aspects (vocal clarity, instrumental balance, dynamic range).

Comparison with Suno.

While both systems target the same use case (full-song generation from text prompts or lyrics), empirical evaluations by users and independent reviewers suggest the following qualitative differences:

• Vocal quality. Suno tends to produce more natural-sounding vocals with better pronunciation, while Udio excels at certain vocal styles (e.g., operatic, choral).

• Instrumental variety. Udio shows greater diversity in instrumentation and arrangement, particularly for genres outside mainstream pop.

• Structural coherence. Both systems occasionally produce songs with structural anomalies (e.g., abrupt endings, missing transitions), though the rate of such anomalies has decreased with successive versions.

Comparative Analysis

We summarise the key features of the major commercial and research systems in the following table.

Evaluation Metrics for Audio Generation

Evaluating generated audio is challenging because perceptual quality, fidelity to a conditioning signal, and musical coherence are all multidimensional attributes that resist reduction to a single number. This section provides a rigorous treatment of the major evaluation metrics used in the audio generation literature, including their mathematical definitions, statistical properties, and practical limitations.

Fréchet Audio Distance

The Fréchet Audio Distance (FAD) is the audio analogue of the Fréchet Inception Distance (FID) used in image generation. It measures the distance between the distribution of real audio and the distribution of generated audio in a learned feature space.

The matrix square root ${({𝚺}_r{𝚺}_g)}^{1/2}$ is the unique positive semidefinite matrix $𝐌$ satisfying ${𝐌}^2={𝚺}_r{𝚺}_g$. When ${𝚺}_r$ and ${𝚺}_g$ commute (i.e., share the same eigenvectors), this simplifies to the element-wise square root of their eigenvalue products.

Feature extractors for FAD.

The choice of feature extractor $\psi$ significantly affects the FAD score:

• VGGish [11]: A CNN trained on AudioSet for audio event classification. This is the most common choice but may not capture music-specific features (melody, harmony, rhythm) effectively.

• PANN (PANNs-CNN14): A larger CNN also trained on AudioSet, providing a richer feature space ($d=2048$) compared to VGGish ($d=128$).

• CLAP embeddings: Using the audio encoder of a CLAP model as $\psi$ yields a FAD variant that is sensitive to semantic audio attributes. This variant is sometimes called the “CLAP-FAD.”

• EnCodec features: Using intermediate representations from a neural audio codec captures perceptual quality at a fine-grained level.

Perceptual Quality Metrics

Unlike distributional metrics such as FAD that compare populations of audio clips, perceptual quality metrics evaluate individual audio signals against a reference or in isolation.

CLAP Score

The CLAP (Contrastive Language-Audio Pretraining) score measures the alignment between a generated audio clip and its conditioning text prompt, analogous to the CLIP score in image generation.

The CLAP score is related to the CLAP contrastive loss (CLAP: Contrastive Language–Audio Pretraining) used during CLAP pretraining. A higher CLAP score indicates that the generated audio is more semantically aligned with the text description.

Mean Opinion Score

Despite the development of numerous objective metrics, subjective human evaluation remains the gold standard for assessing audio quality and musicality. The Mean Opinion Score is the most widely used protocol.

Confidence intervals.

Under the assumption that the ratings for stimulus $i$ are i.i.d. draws from a distribution with mean ${\mu }_i$ and variance ${\sigma }_i^2$, the $95\%$ confidence interval for ${\text{MOS}}_i$ is (MOS CI)$$\begin{array}{ccc}& {\text{MOS}}_i\pm 1.96\cdot \frac{{\hat{\sigma }}_i}{\sqrt{K}},& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${\hat{\sigma }}_i=\sqrt{\frac{1}{K-1}{\sum }_{k=1}^K{(s_{k,i}-{\text{MOS}}_i)}^2}$ is the sample standard deviation. The i.i.d. assumption is often violated in practice (raters have different internal scales and biases), which motivates more sophisticated analysis methods.

MUSHRA protocol.

The MUltiple Stimuli with Hidden Reference and Anchor (MUSHRA; ITU-R BS.1534) protocol is a variant designed for evaluating intermediate-quality audio (such as codec outputs or generated music). Key features include:

• A hidden reference (the original high-quality audio) is included among the test stimuli.

• A low-quality anchor (e.g., 3.5,kHz low-pass filtered version) is included to calibrate the scale.

• Raters assign scores on a continuous 0–100 scale rather than a discrete 1–5 scale.

• Raters can switch between stimuli in real time.

The MUSHRA score for stimulus $i$ is (Mushra)$$\begin{array}{ccc}& {\text{MUSHRA}}_i=\frac{1}{K}\sum _{k=1}^K{s}_{k,i},s_{k,i}\in [0,100].& \text{<span class="tml-eqn"></span>}\end{array}$$

Statistical testing.

To determine whether two systems produce significantly different quality, we use the Wilcoxon signed-rank test, which is appropriate for paired ordinal data and does not assume normality.

Evaluation Summary

We summarise the major evaluation metrics and their properties in the following table.

The mechanism is the audio analogue of the precision–recall trade-off in image generation: FAD is a distributional metric that charges a system for missing modes, whereas a per-stimulus rating never sees the modes a system failed to produce. A mode-collapsed generator that always emits one impeccable piano loop can therefore beat a genuinely broad generator in a listening test and lose badly on FAD. Whether real listener panels behave exactly like $\overline{q}$ is an empirical question, not a theorem; what the proposition establishes is that no monotone relation between the two families of metric can exist in principle, so a disagreement between them is not by itself evidence that one of them is misconfigured.

Musicality and Perceptual Quality

The evaluation metrics in Evaluation Metrics for Audio Generation assess generic audio quality and text-audio alignment. For music generation specifically, we also need to evaluate attributes that are central to musicality: rhythmic regularity, harmonic coherence, melodic contour, structural organisation, and stylistic diversity. This section introduces domain-specific metrics for these attributes and analyses the fundamental trade-off between diversity and quality.

Musical Structure Metrics

Musical structure operates at multiple temporal scales: individual notes (milliseconds), beats and bars (seconds), phrases (tens of seconds), and sections (minutes). We discuss metrics at each scale.

Beat tracking accuracy.

A musically coherent piece should have a clear and consistent beat. Given a generated audio signal $𝒙$, a beat tracker extracts a sequence of beat times $\widehat{𝒃}=({\hat{b}}_1,\dots ,{\hat{b}}_B)$. If a reference beat grid $𝒃=(b_1,\dots ,b_B)$ is available, the F-measure of beat tracking is (Fbeat)$$\begin{array}{ccc}& F_{\text{beat}}=\frac{2\cdot P_{\text{beat}}\cdot R_{\text{beat}}}{P_{\text{beat}}+R_{\text{beat}}},& \text{<span class="tml-eqn"></span>}\end{array}$$ where precision $P_{\text{beat}}$ is the fraction of predicted beats within a tolerance window $\delta$ (typically $\delta =70$,ms) of a true beat, and recall $R_{\text{beat}}$ is the fraction of true beats matched by a prediction.

For generated music without a reference, we instead compute beat consistency: the regularity of inter-beat intervals. Let ${\mathrm{\Delta }}_n={\hat{b}}_{n+1}-{\hat{b}}_n$ be the $n$-th inter-beat interval. The normalised tempo deviation is (Tempo DEV)$$\begin{array}{ccc}& {\sigma }_{\text{tempo}}=\frac{\text{std}({\mathrm{\Delta }}_1,\dots ,{\mathrm{\Delta }}_{B-1})}{\text{mean}({\mathrm{\Delta }}_1,\dots ,{\mathrm{\Delta }}_{B-1})},& \text{<span class="tml-eqn"></span>}\end{array}$$ where $\text{std}$ and $\text{mean}$ denote the sample standard deviation and mean, respectively. Lower values indicate more consistent tempo. Professional recordings typically have ${\sigma }_{\text{tempo}}<0.05$ (excluding intentional rubato), while generated music from early systems often exhibits ${\sigma }_{\text{tempo}}>0.15$.

Key and chord estimation.

Harmonic coherence can be assessed by estimating the key and chord progression of the generated music and evaluating their consistency. Let $\hat{k}\in \{C,C♯,D,\dots ,B\}\times \{\text{major},\text{minor}\}$ be the estimated global key and ${\widehat{𝒄}}_{\text{chord}}=({\hat{c}}_1,\dots ,{\hat{c}}_M)$ the estimated chord sequence. Two metrics are relevant:

1. Key consistency. The fraction of estimated chords that are diatonic to (i.e., belong to the scale of) the estimated key: (Keycons)$$\begin{array}{ccc}& \text{KeyCons}=\frac{1}{M}\sum _{m=1}^M\mathrm{𝟏}[{\hat{c}}_m\in \text{Diatonic}(\hat{k})].& \text{<span class="tml-eqn"></span>}\end{array}$$ Professional music typically has $\text{KeyCons}\ge 0.80$ (allowing for secondary dominants, borrowed chords, and modulations).

2. Chord transition smoothness. The average “harmonic distance” between consecutive chords, measured by the number of common tones or the distance in the Tonnetz (a graph-theoretic representation of pitch relationships): (Chordsmooth)$$\begin{array}{ccc}& \text{ChordSmooth}=\frac{1}{M-1}\sum _{m=1}^{M-1}{d}_{\text{Tonnetz}}({\hat{c}}_m,{\hat{c}}_{m+1}).& \text{<span class="tml-eqn"></span>}\end{array}$$ Lower values indicate smoother harmonic progressions.

Self-similarity analysis.

Musical form (verse-chorus structure, ABA form, rondo) creates characteristic patterns of repetition and contrast. These can be quantified using the self-similarity matrix (SSM).

In the SSM of well-structured music, one observes:

• Diagonal blocks. Sections of the piece that are internally consistent (e.g., a verse) appear as bright blocks along the main diagonal.

• Off-diagonal stripes. Repeated sections (e.g., the chorus returning) appear as bright stripes parallel to the main diagonal.

• Checkerboard patterns. Alternating sections (ABAB form) produce a checkerboard-like pattern.

The structural complexity of a piece can be quantified as the entropy of the SVD spectrum of $𝐒$: (Struct Entropy)$$\begin{array}{ccc}& H_{\text{struct}}=-\sum _{i=1}^N{\tilde{\sigma }}_i\log {\tilde{\sigma }}_i,& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${\tilde{\sigma }}_i={\sigma }_i/{\sum }_j{\sigma }_j$ is the normalised $i$-th singular value of $𝐒$. A piece with highly repetitive structure (low complexity) will have a few dominant singular values and low entropy, while a piece with no repetition (high complexity) will have a flat spectrum and high entropy.

Diversity and Coverage

A good music generation system should not only produce high-quality samples but also exhibit diversity: generating different outputs for different prompts (inter-prompt diversity) and varied outputs for the same prompt across different random seeds (intra-prompt diversity).

Intra-class diversity.

For a fixed prompt $𝒄$, generate $K$ samples $\{{𝒙}_1,\dots ,{𝒙}_K\}$ and compute pairwise distances in a feature space: (Intra DIV)$$\begin{array}{ccc}& {\text{Div}}_{\text{intra}}(𝒄)=\frac{2}{K(K-1)}\sum _{i<j}d(\psi ({𝒙}_i),\psi ({𝒙}_j)),& \text{<span class="tml-eqn"></span>}\end{array}$$ where $d$ is a distance function (e.g., cosine distance) and $\psi$ is a feature extractor. Low intra-class diversity indicates mode collapse: the model produces nearly identical outputs regardless of the random seed.

Genre coverage.

For a set of genre labels $𝒢=\{g_1,\dots ,g_G\}$, generate $N$ samples from genre-specific prompts and classify them using a pretrained genre classifier. The genre coverage is the fraction of genres for which the classifier assigns the correct label to at least one generated sample: (Genre Coverage)$$\begin{array}{ccc}& \text{Coverage}=\frac{1}{G}\sum _{g=1}^G\mathrm{𝟏}[\exists i:\hat{g}({𝒙}_i)=g\text{ and prompt}({𝒙}_i)=g].& \text{<span class="tml-eqn"></span>}\end{array}$$

Detection and Watermarking

The generative audio systems developed in the preceding sections can synthesise speech, music, and sound effects of remarkable quality. This very capability raises an urgent question: how can we distinguish AI-generated audio from recordings of real-world events? The stakes are considerable. Synthetic speech can impersonate individuals for fraud; generated music can circumvent copyright protections; fabricated sound effects can manipulate evidence. In this section, we formalise the detection problem, study two families of solutions (passive detection and active watermarking), and analyse their information-theoretic limits.

The Detection Problem

We begin by casting detection as a binary hypothesis test.

The Neyman–Pearson lemma provides the optimal detector when the two distributions are known.

SynthID for Audio

Active watermarking takes a fundamentally different approach from passive detection: rather than trying to identify synthetic audio after the fact, the generator embeds an imperceptible signal in every output, which a dedicated decoder can later extract. Google DeepMind's SynthID [31] for audio operates in the spectrogram domain, embedding a binary payload into the time-frequency representation of the generated waveform.

The training objective balances imperceptibility and recoverability: (Synthid LOSS)$$\begin{array}{ccc}& {ℒ}_{\mathrm{SynthID}}=\underset{\text{imperceptibility}}{\underbrace{{𝔼}_{𝐒,𝒃}\left[{\Vert \mathrm{\Delta }{𝐒}_{\theta }(𝐒,𝒃)\Vert }^2\right]}}+\lambda \underset{\text{recovery under attack}}{\underbrace{{𝔼}_{𝐒,𝒃,𝒜}\left[\sum _{\ell =1}^L\mathrm{BCE}\right([{𝒟}_{\psi }(𝒜(\widetilde{𝐒})){]}_{\ell },b_{\ell })]}},& \text{<span class="tml-eqn"></span>}\end{array}$$ where $\mathrm{BCE}$ denotes the binary cross-entropy loss and $\lambda >0$ controls the trade-off.

AudioSeal

While SynthID operates globally on the spectrogram, Meta's AudioSeal [32] introduces localized watermarking that operates at the sample level, enabling temporal localisation of which portions of an audio signal are synthetic.

The AudioSeal architecture uses a convolutional encoder-decoder structure with a multi-scale discriminator for perceptual quality.

The AudioSeal training loss combines several terms: (Audioseal LOSS)$$\begin{array}{ccc}& \begin{array}{cc}{ℒ}_{\mathrm{AudioSeal}}& =\underset{\text{imperc.}}{\underbrace{{\Vert {𝒘}_{\theta }(𝒙,𝒃)\Vert }^2}}\\ & +{\lambda }_1\underset{\text{quality}}{\underbrace{{ℒ}_{\mathrm{adv}}(G_{\theta },D_{\mathrm{disc}})}}\\ & +{\lambda }_2\underset{\text{detection}}{\underbrace{𝔼[-\sum _{n=1}^N{y}_n\log {[D_{\psi }(\widetilde{𝒙})]}_n+(1-y_n)\log (1-{[D_{\psi }(\widetilde{𝒙})]}_n)]}}\\ & +{\lambda }_3\underset{\text{recovery}}{\underbrace{𝔼\left[\sum _{\ell =1}^L\mathrm{BCE}({[M_{\psi }(\widetilde{𝒙})]}_{\ell },b_{\ell })\right]}},\end{array}& \text{<span class="tml-eqn"></span>}\end{array}$$ where $y_n\in \{\mathrm{0,1}\}$ indicates whether sample $n$ is watermarked, $D_{\mathrm{disc}}$ is a multi-scale waveform discriminator (similar to the HiFi-GAN discriminator), and ${\lambda }_1,{\lambda }_2,{\lambda }_3>0$ are balancing hyperparameters.

Statistical Detection Methods

When watermarking is not available, passive detection must rely on statistical artifacts that distinguish synthetic audio from real recordings. These artifacts arise from the generation process itself: the architecture, the training data, and the sampling procedure all leave traces in the output distribution.

The adversarial robustness of detection methods is a critical concern. An adversary who knows the detection method can attempt to evade it.

The Arms Race

Post-processing attacks represent a significant practical threat to both passive detection and watermarking.

Ethical and Legal Considerations

The technical capabilities developed in this chapter intersect with a web of ethical, legal, and societal concerns. In this section, we examine the principal areas of tension: copyright and intellectual property, voice cloning and deepfakes, and the emerging frameworks for attribution and provenance.

Copyright and Intellectual Property

Generative music models are trained on large corpora of copyrighted recordings. This raises two distinct legal questions: (i) whether training on copyrighted material constitutes fair use, and (ii) whether the generated output infringes the copyright of training examples.

From a mathematical perspective, the question of whether a generated piece is “substantially similar” to a training example can be formalised as a distance computation.

Voice Cloning and Deepfakes

Zero-shot voice cloning systems (as described in earlier sections of this chapter) can synthesise speech in any person's voice from a few seconds of reference audio. While this enables legitimate applications (accessibility, dubbing, voice preservation for medical patients), it also creates serious risks.

1. Identity fraud. Cloned voices can be used to bypass voice authentication systems, impersonate individuals in phone calls, or create fraudulent audio evidence.

2. Non-consensual content. An individual's voice can be used without consent to generate speech they never uttered, creating reputational and psychological harm.

3. Misinformation. Fabricated speech attributed to public figures can spread misinformation, undermine trust in audio evidence, and destabilise democratic processes.

Technical countermeasures include:

1. Speaker verification. Deploying robust speaker verification systems that can detect cloned voices by comparing against enrolled voiceprints.

2. Mandatory watermarking. Requiring all voice synthesis systems to embed watermarks (as in sec:audiogen:detection:synthid,sec:audiogen:detection:audioseal) in their output.

3. Consent mechanisms. Requiring explicit consent from the voice owner before cloning is permitted, enforced through API access controls and voice registration databases.

Attribution and Provenance

The Coalition for Content Provenance and Authenticity (C2PA) standard provides a framework for attaching cryptographically signed metadata to media files, creating an unbroken chain of provenance from creation to consumption.

For audio, the C2PA manifest includes:

1. A claim declaring that the content was generated by a specific AI system.

2. A signature from the generating system's certificate authority, ensuring the claim is authentic.

3. Action records documenting any transformations (editing, mixing, compression) applied after generation.

Watermarking provides a complementary layer of provenance that survives the removal of metadata (which can be stripped by simple file format conversion). The combination of cryptographic provenance (C2PA) and signal-level provenance (watermarking, as discussed in Detection and Watermarking) provides a defence-in-depth strategy for audio attribution.

Connections and Synthesis

Audio generation draws on and contributes to nearly every major topic in this book. In this section, we make these connections explicit, providing insight boxes that link the techniques of this chapter to their theoretical foundations elsewhere. The reader is encouraged to revisit the referenced chapters with the audio perspective in mind; the connections often illuminate both sides.

Variational Inference

Optimal Transport and Flow Matching

Generative Adversarial Networks

Vector Quantisation

Autoregressive Models

Video Diffusion

Diffusion Language Models

Connections Map

fig:audiogen:connection-map visualises the connections between audio generation and the other topics covered in this book.

Open Problems and Future Directions

Despite the remarkable progress documented in this chapter, audio and music generation systems remain far from the ideal creative tool. In this section, we identify five fundamental open problems that define the frontier of the field. For each problem, we provide a formal statement, discuss why it is hard, and outline promising directions of attack.

Real-Time Interactive Generation

Throughput is no longer the obstacle. The systems surveyed in this chapter generate audio comfortably faster than real time on a modern GPU: the codec pipeline of Example 9 runs at RTF $\approx 0.8$, Stable Audio produces 45 seconds of music in 2.5 seconds (Example 18), few-step flow matching reaches RTF $0.0075$ (tab:audiogen:nfe-wallclock), distilled consistency models RTF $0.0016$ (Example 23), and chunked long-form generation RTF $0.017$ over three minutes (Example 26).

What remains hard is latency, which is a different quantity. Real-time factor measures total wall-clock time divided by output duration and can be driven down by batching and by generating a long clip in one shot; interactive use instead requires that the first sample appear within tens of milliseconds of the triggering event, and that every subsequent block arrive before its playout deadline, with no batching over the future and no opportunity to look ahead. A model that generates 30 seconds of audio in 0.5 seconds has an excellent RTF and a 500 ms latency, which is an order of magnitude too slow for a musician playing an instrument. Interactive applications (live performance, game audio, conversational agents) are bounded by the second number, not the first.

The latency budget can be decomposed into three stages, each of which must be optimised.

1. Encoding stage: Computing the conditioning representation (text embedding, control signal processing). This is typically fast ($<10$ ms) and is not the bottleneck.

2. Generation stage: Running the diffusion or autoregressive model to produce the latent representation. For a diffusion model with $T$ denoising steps, each requiring a full forward pass through a large transformer, this is the dominant cost. Consistency distillation (reducing $T$ to 1–4 steps) and model distillation (reducing the model size) are the principal approaches.

3. Decoding stage: Converting the latent representation to a waveform via the decoder/vocoder. For a streaming codec decoder, this can operate in real-time with modest computation, but the latency depends on the codec's frame size.

Song-Length Coherence

Duration alone is no longer the barrier: as tab:audiogen:commercial-comparison records, Suno and Udio already emit four-minute tracks, and research systems reach 95 seconds in a single shot. What is still missing is structure over those durations. Open research models generate 10 to 30 seconds before their context window runs out, and the commercial systems that go further do so by chunking or by planning at a coarse resolution, which buys length but not guaranteed thematic return. Generating a complete song (3 to 5 minutes) whose verses, choruses, bridge, key changes and dynamic arc are genuinely related to one another - rather than locally plausible and globally arbitrary - remains an open problem.

The difficulty lies in the mismatch between the model's context window and the time scales of musical structure. A typical diffusion model's context window covers 10–30 seconds, but musical structure operates at multiple hierarchical levels:

Faithful Instrument and Vocal Rendering

Even the best current systems produce audio that experts can distinguish from real recordings. The gap is most apparent for solo instruments and vocals, where listeners have extremely sensitive expectations.

This conjecture, if true, implies that future progress in audio generation quality requires advances in the generative model itself (better architectures, training procedures, and data) rather than better audio compression.

Unified Audio Generation

Current systems are specialised: separate models for speech synthesis, music generation, and sound effect production. A unified model that handles all audio modalities with a single architecture and a single set of weights would be both more efficient and more capable, enabling cross-modal generation (e.g., generating music with lyrics, or sound effects synchronised to speech).

The principal challenge is the distributional diversity: speech, music, and sound effects have very different statistical properties (spectral envelopes, temporal dynamics, harmonic structures). A single model must learn to represent all of these within a shared latent space without sacrificing quality in any modality.

Verifiable Provenance

As discussed in Detection and Watermarking, current watermarking systems face a fundamental tension between robustness and imperceptibility, bounded by the information-theoretic capacity of the watermarking channel (Proposition 47).

The unforgeability requirement connects to the cryptographic literature on digital signatures and commitment schemes. The challenge is that unlike traditional digital signatures (which protect a specific bitstring), audio watermarks must survive signal-processing transformations that alter the audio content while preserving its perceptual identity.

Exercises