Skip to content
AIAI Wranglers

21 Audio and Music Generation

Why Audio Generation?

Consider a ten-second clip of a jazz trio: a piano outlines a chord progression, a bass walks through the changes, and brushes dance on a snare drum. The corresponding digital waveform contains roughly 441,000 samples (at 44.1,kHz), each carrying amplitude information at 16-bit resolution. To a generative model, this clip is a point in 441,000, and the space of perceptually valid jazz trio recordings occupies a vanishingly thin manifold within that vast ambient space. The challenge of audio generation is to learn the structure of this manifold well enough to draw new points from it on demand.

Why should we care? The practical applications are abundant. Text-to-speech systems convert written language to natural sounding utterances, powering virtual assistants and accessibility tools. Text-to-music models compose novel audio given natural-language prompts such as “a melancholic cello solo over soft rain.” Sound effect generation synthesises Foley audio for film and game production. Audio inpainting fills gaps in corrupted recordings, and voice conversion transforms the timbre of a speaker while preserving linguistic content. These tasks share a common mathematical substrate: learning a conditional or unconditional distribution over waveforms and sampling from it efficiently.

Key Idea.

Despite their surface diversity, all audio generation tasks reduce to the same abstract problem: model a distribution p(𝒙) (or p(𝒙|𝒄) for conditional generation) over sequences 𝒙L, where L is the number of audio samples, and generate new sequences by sampling from the learned distribution. The differences between tasks lie in the conditioning signal 𝒄 (text, melody, video, speaker embedding) and in which aspects of the output the user cares about (intelligibility, musicality, timbral fidelity).

The Dimensionality of Sound

The raw dimensionality of audio signals dwarfs that of the images we encountered in earlier chapters, yet remains below the extreme scales of video. tab:audiogen:dimensionality makes this concrete.

tableDimensionality of audio signals at various durations and sample rates. CD-quality audio uses 44.1,kHz; speech models commonly use 16,kHz.

SignalSample RateDurationDimensions
Speech utterance16,kHz3,s4.8×104
Speech paragraph16,kHz30,s4.8×105
Music clip44.1,kHz10,s4.4×105
Full song44.1,kHz4,min1.1×107
Stereo full song44.1,kHz4,min2.1×107
512×512 RGB image7.9×105

A single second of CD-quality mono audio contains 44,100 floating-point values. A three-minute pop song therefore lives in a space of dimension d7.9×106; in stereo, this doubles to 1.6×107. Compared to a 512×512 image (d7.9×105), even a short music clip is an order of magnitude larger.

Remark 1.

The high dimensionality of raw waveforms motivates the development of compressed representations: mel-spectrograms reduce the time axis by hop-size factors of 256–512 and the frequency axis to 80–128 mel bins, while neural audio codecs compress a full second of audio into as few as 50–75 discrete tokens. Nearly every modern audio generation system operates in one of these compressed spaces rather than directly on the waveform. Sections Audio Signal Processing Foundations and Neural Audio Codecs develop these representations in detail.

Remark 2.

Audio and image generation share many mathematical foundations (GANs, VAEs, diffusion models, flow matching), but two structural differences profoundly affect model design. First, audio is inherently sequential: the natural causal ordering of time provides a strong inductive bias for autoregressive models, which have no equally natural ordering for 2D images. Second, the perceptual metric for audio is fundamentally different from that for images. Human auditory perception is phase-insensitive but exquisitely tuned to spectral envelope, pitch, and temporal dynamics, while visual perception is dominated by spatial frequency and colour statistics. These perceptual differences motivate the specialised loss functions (mel-spectrogram loss, multi-resolution STFT loss) and representations (mel-spectrograms, codec tokens) that this chapter develops.

A Brief History of Audio Generation

The history of neural audio generation can be divided into four overlapping eras, each defined by the dominant generative paradigm. We summarise them here to provide context for the technical sections that follow; the reader should consult the original papers for full details.

Era I: Concatenative and parametric synthesis (pre-2016).

Classical text-to-speech systems either concatenated recorded speech segments from large databases or drove parametric vocoders with hand-engineered acoustic features. The quality ceiling was unmistakable: concatenative systems sounded choppy at segment boundaries, and parametric vocoders produced a characteristic “buzzy” timbre that listeners found unnatural.

Era II: Autoregressive waveform models (2016–2019).

DeepMind's WaveNet [13] demonstrated that an autoregressive model predicting one sample at a time could produce strikingly natural speech. The model factorised the joint distribution as (Wavenet Factor)p(𝒙)=n=1Lp(xn|x1,,xn1), with each conditional implemented by a dilated causal convolution stack. The catch was speed: generating one second of 16,kHz audio required 16,000 sequential forward passes. Parallel WaveNet [14] and WaveRNN [15] addressed inference cost, but the autoregressive bottleneck remained a theme.

Era III: GAN and diffusion vocoders (2019–2022).

GAN-based vocoders such as MelGAN [16], HiFi-GAN [17], and UnivNet [18] replaced autoregressive generation with feed-forward synthesis, achieving real-time waveform generation on a single GPU\@. Concurrently, diffusion-based vocoders (DiffWave [19], WaveGrad [20]) applied the denoising diffusion framework of ch:diffusion to the audio domain. These models serve as the “decoder” in many two-stage generation pipelines.

Era IV: Foundation models for audio (2023–present).

The convergence of large language models, neural audio codecs, and latent diffusion has produced a new generation of systems capable of generating high-fidelity, long-form audio from text prompts. AudioLDM [21] applies latent diffusion to mel-spectrograms. MusicGen [22] generates music autoregressively over the discrete tokens of an EnCodec codec. Stable Audio [23] operates a latent diffusion transformer in the continuous latent space of a variational autoencoder. Commercial systems such as Suno and Udio generate full songs with vocals, approaching (and sometimes passing) human-level quality in listener studies.

Historical Note.

The pace of progress. WaveNet appeared in September 2016; barely seven years later, Suno v3 could generate a full two-minute pop song with lyrics, melody, harmony, and studio-quality mixing from a single text prompt. This trajectory, from sample-level autoregression to codec-based language modelling and latent diffusion, mirrors the image generation revolution but unfolds in a domain with fundamentally different signal characteristics. fig:audiogen:timeline sketches the key milestones.

Timeline of major milestones in neural audio generation. The field has progressed from sample-level autoregressive models (Era II) through efficient GAN and diffusion vocoders (Era III) to foundation-scale text-to-audio and text-to-music systems (Era IV).

Chapter Roadmap

This chapter develops the mathematical and architectural foundations of modern audio generation systems. The narrative is organised into a progression from signal-level building blocks to complete generation pipelines.

  1. Audio signal processing (Audio Signal Processing Foundations). We formalise the Short-Time Fourier Transform, the mel scale, and the mel-spectrogram, establishing the standard input representations used throughout the chapter.

  2. Neural audio vocoders (Neural Audio Vocoders). We study HiFi-GAN and related architectures that convert mel-spectrograms back into high-fidelity waveforms, connecting their discriminator designs to the GAN theory of ch:gan.

  3. Neural audio codecs (Neural Audio Codecs). We develop Residual Vector Quantisation and its application in SoundStream, EnCodec, and DAC, which compress audio into sequences of discrete tokens amenable to language-model-style generation.

  4. CLAP and contrastive audio–text alignment (CLAP: Contrastive Language–Audio Pretraining). We formalise the contrastive loss that learns a joint embedding space for audio and text, enabling text-conditioned generation.

  5. Autoregressive generation over codec tokens (later sections). We study MusicLM, MusicGen, and VALL-E, which treat audio generation as a sequence modelling problem over discrete codec codes.

  6. Latent diffusion and flow matching for audio (later sections). We develop AudioLDM, Stable Audio, and related systems that apply continuous generative models in compressed latent spaces.

  7. Evaluation, ethics, and open problems (later sections). We discuss Fréchet Audio Distance, perceptual metrics, watermarking, and the societal implications of generative audio.

Throughout, we assume familiarity with the fundamentals of generative adversarial networks (ch:gan), variational autoencoders (ch:vae), diffusion models (ch:diffusion), and normalising flows (ch:flows). Readers seeking background on these topics should consult the corresponding chapters before proceeding.

Audio Signal Processing Foundations

Before we can generate audio, we must understand how audio is represented, transformed, and compressed. This section develops the mathematical machinery that every subsequent section relies upon: the Short-Time Fourier Transform (STFT), the mel frequency scale, and the mel-spectrogram. These are not mere engineering details; they encode deep facts about human perception and signal structure that profoundly shape the design of generative models.

Digital Audio Signals

A continuous-time audio signal is a real-valued function of time, s:. Digital audio is obtained by sampling s at a fixed rate fs (the sampling rate or sample rate), producing a discrete sequence (Sampling)x[n]=s(nfs),n=0,1,,L1, where L=Tfs is the total number of samples and T is the signal duration in seconds. We collect these samples into a vector 𝒙=(x[0],x[1],,x[L1])L.

Definition 1 (Digital Audio Signal).

A digital audio signal is a vector 𝒙L whose n-th entry x[n] represents the amplitude of a continuous-time acoustic waveform sampled at rate fs hertz. The Nyquist frequency fNyq=fs/2 is the highest frequency that can be faithfully represented without aliasing.

Standard sample rates include 8,kHz (telephony), 16,kHz (speech processing), 22.05,kHz (compressed music), 44.1,kHz (CD-quality), and 48,kHz (professional audio and video). The choice of fs determines both the bandwidth and the dimensionality of the generative modelling problem.

Remark 3.

In practice, each sample is stored as a fixed-point integer (e.g., 16-bit PCM, giving 216=65,536 quantisation levels) or a 32-bit floating-point number. For the purposes of generative modelling, we treat 𝒙 as a real-valued vector and defer quantisation to the final output stage. This is analogous to working with continuous pixel intensities in image generation rather than discrete 8-bit values.

Example 1 (Speech vs. music sample rates).

A text-to-speech system operating at fs=16,000,Hz represents frequencies up to 8,kHz, which suffices for intelligible speech but discards much of the spectral richness that makes natural voices warm and distinctive. A music generation system typically requires fs=44,100,Hz or higher to capture the full range of human hearing (approximately 20,Hz to 20,kHz), the shimmer of cymbals, the breathiness of a flute, the harmonic overtones of a bowed string. This nearly threefold increase in sample rate translates directly into higher-dimensional generation problems.

Proposition 1 (Shannon–Nyquist Sampling Theorem).

Let s(t) be a band-limited continuous signal whose Fourier transform satisfies s^(f)=0 for |f|>fmax. Then s can be perfectly reconstructed from its samples {x[n]=s(n/fs)}n if and only if fs>2fmax. If fs2fmax, the sampled signal suffers from aliasing: high-frequency components fold back into the representable bandwidth, creating artefacts that cannot be removed by post-processing.

Proof.

The sampled signal's spectrum is the periodisation x^(f)=fsk=s^(fkfs). When fs>2fmax, the copies s^(fkfs) do not overlap, so s^(f) can be recovered by applying an ideal low-pass filter with cutoff fs/2. When fs2fmax, adjacent copies overlap (aliasing), and the original spectrum cannot be recovered.

Caution.

Aliasing is not only a concern during recording; it can also arise during generation. If a neural vocoder or codec decoder produces frequency content above the Nyquist frequency (e.g., due to nonlinear activations in transposed convolution layers), this energy aliases back into the audible band, producing metallic or buzzy artefacts. Anti-aliased activation functions, as used in BigVGAN [1], mitigate this problem by low-pass filtering the hidden representations after each nonlinearity.

The Short-Time Fourier Transform

The Fourier transform reveals the frequency content of a signal, but it provides no information about when those frequencies occur. For audio generation, temporal localisation is essential: a piano note that begins at time t=2,s and ends at t=3,s must be placed correctly in the output waveform. The Short-Time Fourier Transform (STFT) resolves this tension by applying the Fourier transform to short, overlapping windows of the signal.

Definition 2 (Short-Time Fourier Transform).

Let 𝒙L be a digital audio signal and let w:{0,,Nfft1}0 be a window function of length Nfft. The Short-Time Fourier Transform of 𝒙 is the complex-valued matrix 𝐗K×M with entries (STFT)X[k,m]=n=0Nfft1w[n]x[mH+n]ej2πkn/Nfft, where k=0,1,,K1 is the frequency bin index with K=Nfft/2+1, m=0,1,,M1 is the frame index with M=(LNfft)/H+1, H is the hop size (the number of samples between successive frames), and j=1.

Each column of 𝐗 is the discrete Fourier transform (DFT) of a windowed segment of 𝒙. The window function w (commonly Hann, Hamming, or Blackman) tapers the segment smoothly to zero at its edges, suppressing spectral leakage from discontinuities at the segment boundaries.

Remark 4.

Typical parameter choices in audio generation are Nfft=1024 or 2048, hop size H=256 or 512, and a Hann window. With fs=22,050,Hz, Nfft=1024, and H=256, each frame spans approximately 46,ms and frames overlap by 75%. The resulting STFT has K=513 frequency bins and ML/256 time frames.

The STFT is invertible (under mild conditions on the window), which is crucial: a vocoder must be able to reconstruct a waveform from a modified spectrogram. The inverse STFT (iSTFT) recovers the signal by applying the inverse DFT to each frame, multiplying by a synthesis window, and overlap-adding the results.

Definition 3 (Inverse STFT via Overlap-Add).

Given the STFT matrix 𝐗K×M and a synthesis window ws, the inverse STFT is (Istft)x^[n]=m=0M1ws[nmH]x~m[nmH]m=0M1w[nmH]ws[nmH], where x~m[] is the inverse DFT of the m-th frame of 𝐗, and the denominator normalises for the overlapping windows.

For the inverse to be exact, the analysis and synthesis windows must satisfy a completeness condition.

Proposition 2 (Constant Overlap-Add (COLA) Condition).

Let w be the analysis window and ws the synthesis window, both of length Nfft. Perfect reconstruction holds, i.e., x^[n]=x[n] for all n, if and only if (COLA)m=w[nmH]ws[nmH]=cfor all n, for some constant c>0. When ws=w (the common case), this reduces to (COLA Symmetric)m=w2[nmH]=cfor all n.

Proof.

Substituting x~m[nmH]=kX[k,m]ej2πk(nmH)/Nfft into (Istft) and using the definition of X[k,m] from (STFT), the numerator becomes mws[nmH]kej2πk(nmH)/Nfftnw[n]x[mH+n]ej2πkn/Nfft. The sum over k produces a Kronecker delta Nfftδ[nmHn], collapsing the inner sum to Nfftw[nmH]x[n]. Thus the numerator equals x[n]Nfftmws[nmH]w[nmH], and dividing by the denominator mw[nmH]ws[nmH] yields x^[n]=x[n] whenever this sum is nonzero and constant.

Example 2 (COLA verification for the Hann window).

The Hann window of length N is w[n]=0.5(1cos(2πn/N)) for 0n<N. With hop size H=N/2 (50% overlap) and ws=w, one can verify that mw2[nmH]=0.3752=0.75 for all n, satisfying the COLA condition with c=0.75. With H=N/4 (75% overlap), the constant becomes c=1.5. In both cases, the denominator in (Istft) is constant, guaranteeing perfect reconstruction.

The STFT decomposes the signal into three informative representations, each useful in different generative contexts.

Definition 4 (STFT Magnitude, Phase, and Power Spectrogram).

Given the STFT 𝐗K×M:

  1. The magnitude spectrogram is |𝐗|0K×M with entries |X[k,m]|=Re(X[k,m])2+Im(X[k,m])2.

  2. The phase spectrogram is 𝐗[π,π)K×M with entries X[k,m]=atan2(Im(X[k,m]),Re(X[k,m])).

  3. The power spectrogram is |𝐗|20K×M with entries |X[k,m]|2.

Insight.

Most audio generation models predict only the magnitude (or log-magnitude) spectrogram and discard phase information. This is justified by the observation that human auditory perception is primarily sensitive to spectral magnitude, not phase. However, reconstructing a waveform from magnitude alone requires estimating the missing phase; the classical Griffin-Lim algorithm [2] iteratively projects between the STFT domain and the time domain to find a consistent phase, while neural vocoders (see Neural Audio Vocoders) learn to synthesise phase implicitly as part of the waveform generation process.

Proposition 3 (STFT as a Linear Operator).

The STFT defines a linear map STFTw,H:LK×M. That is, for any 𝒙,𝒚L and α,β: (STFT Linear)STFTw,H(α𝒙+β𝒚)=αSTFTw,H(𝒙)+βSTFTw,H(𝒚). This linearity has an important practical consequence: if a model generates two spectrogram components separately (e.g., a harmonic component and a noise component), their sum in the STFT domain corresponds to the STFT of the sum of the corresponding time-domain signals.

Proof.

From (STFT), each entry X[k,m] is a weighted sum of the signal samples: X[k,m]=nw[n]x[mH+n]ej2πkn/Nfft. Substituting αx[mH+n]+βy[mH+n] and using linearity of the finite sum yields the result.

Proposition 4 (Parseval's Theorem for the STFT).

If the analysis window w satisfies the COLA condition (Proposition 2) with constant c, then the signal energy is preserved (up to normalisation) in the STFT domain: (STFT Parseval)n=0L1|x[n]|2=1cNfftm=0M1k=0K1|X[k,m]|2. This energy preservation property ensures that the STFT does not artificially amplify or attenuate any portion of the signal, which is critical when using STFT-domain losses for training generative models.

Proof.

By Parseval's theorem for the DFT, the energy in the m-th windowed frame satisfies k|X[k,m]|2=Nfftn|w[n]x[mH+n]|2. Summing over all frames m and using the COLA condition mw2[nmH]=c yields the stated identity.

Example 3 (STFT compression ratio).

A 10-second audio clip at fs=22,050,Hz has L=220,500 real-valued samples. The STFT with Nfft=1024 and H=256 produces K=513 frequency bins and M=(220,5001024)/256+1=858 frames. The magnitude spectrogram |𝐗|513×858 has 440,154 entries, roughly 2× the size of the original signal. However, the log-mel-spectrogram with Fmel=80 has only 80×858=68,640 entries, a compression factor of 220,500/68,6403.2×. This modest spatial compression, combined with the perceptual relevance of mel features, is why the mel-spectrogram has become the standard intermediate representation.

Remark 5.

The STFT is subject to the Heisenberg–Gabor uncertainty principle: the product of time resolution Δt and frequency resolution Δf cannot be made arbitrarily small. Specifically, for a Gaussian window (which minimises the uncertainty product), (Gabor Limit)ΔtΔf14π. A large FFT window gives fine frequency resolution at the cost of poor temporal localisation; a small window gives fine temporal resolution at the cost of smeared frequencies. This fundamental tradeoff explains why the multi-resolution STFT loss (def:audiogen:mr-stft-loss) uses multiple window sizes: no single window can simultaneously capture both the transient attack of a snare drum and the precise pitch of a sustained violin note.

The Mel Scale and Mel-Spectrogram

The linear frequency axis of the STFT does not match human auditory perception. We perceive pitch differences on a roughly logarithmic scale: the perceptual distance between 100,Hz and 200,Hz (one octave) is much greater than that between 5,000,Hz and 5,100,Hz, even though the absolute difference in hertz is the same. The mel scale provides a frequency warping that better reflects this perceptual nonlinearity.

Definition 5 (Mel Scale).

The mel scale is a perceptual frequency scale defined by the mapping mel:00, (MEL Scale)mel(f)=2595log10(1+f700), where f is the frequency in hertz. The inverse mapping is (MEL Inverse)mel1(m)=700(10m/25951). The mel scale is approximately linear below 1,kHz and approximately logarithmic above 1,kHz, matching the frequency resolution of the human cochlea.

Remark 6.

The mel scale was proposed by Stevens, Volkmann, and Newman in 1937, based on experiments asking listeners to adjust a tone until it sounded “half as high” as a reference. The name “mel” derives from “melody,” reflecting the scale's connection to perceived pitch. Several competing formulas exist (e.g., the HTK variant uses 1127ln(1+f/700)), but the logarithmic formula in (MEL Scale) is the most widely used in modern audio generation systems.

Proposition 5 (Mel Scale Asymptotics).

The mel scale exhibits two distinct regimes:

  1. Low frequencies (f700,Hz): using log10(1+x)x/ln10 for small x, (MEL LOW FREQ)mel(f)2595700ln10f1.61f. The mel scale is approximately linear, with 1,mel 0.62,Hz.

  2. High frequencies (f700,Hz): using log10(1+f/700)log10(f/700), (MEL HIGH FREQ)mel(f)2595log10(f)2595log10(700)2595log10(f)7384. The mel scale is approximately logarithmic.

Proof.

Both results follow from standard asymptotic expansions of log10(1+f/700). For part 1, the first-order Taylor expansion log10(1+u)u/ln10 applies when u=f/7001. For part 2, when f/7001, the additive 1 in the argument becomes negligible, yielding log10(f/700).

To construct a mel-spectrogram, we apply a bank of triangular filters, spaced uniformly on the mel scale, to the power (or magnitude) spectrogram.

Definition 6 (Mel Filterbank).

Let Fmel be the desired number of mel bands. Choose Fmel+2 centre frequencies c0<c1<<cFmel+1 that are equally spaced on the mel scale between mel(fmin) and mel(fmax), where fmin and fmax define the frequency range of interest. The i-th triangular filter (i=1,,Fmel) is (MEL Filter)hi[k]={0if f[k]<ci1,f[k]ci1cici1if ci1f[k]<ci,ci+1f[k]ci+1ciif cif[k]<ci+1,0if f[k]ci+1, where f[k]=kfs/Nfft is the frequency corresponding to DFT bin k. The filters are assembled into the mel filterbank matrix 𝐌0Fmel×K, with Mik=hi[k].

Definition 7 (Mel-Spectrogram).

Given the magnitude spectrogram |𝐗|0K×M and the mel filterbank matrix 𝐌0Fmel×K, the mel-spectrogram is (MEL Spectrogram)𝐒=𝐌|𝐗|20Fmel×M, or, in log scale, (LOG MEL)𝐒log=log(𝐒+ϵ)Fmel×M, where ϵ>0 is a small constant (typically 105 or 108) for numerical stability. The log-mel-spectrogram is the most common input representation for neural audio generation models.

fig:audiogen:mel-filterbank illustrates the mel filterbank and the pipeline from waveform to mel-spectrogram.

Mel filterbank: triangular filters spaced uniformly on the mel scale. At low frequencies, where pitch perception is most acute, the filters are narrow and closely spaced. At high frequencies, the filters become wider, reflecting reduced perceptual resolution.
The mel-spectrogram extraction pipeline. A waveform 𝒙L is transformed via the STFT into a complex spectrogram, from which the power spectrogram is computed. Multiplication by the mel filterbank matrix 𝐌 warps the frequency axis, and a log operation compresses the dynamic range. The resulting log-mel-spectrogram 𝐒logFmel×M is the standard input to most neural audio models.

Proposition 6 (Mel Filterbank as Information Bottleneck).

The mel filterbank matrix 𝐌0Fmel×K with Fmel<K has rank at most Fmel. The mapping |𝐗|2𝐌|𝐗|2 is therefore a lossy projection: the null space ker(𝐌) has dimension at least KFmel, and any spectral variation lying in ker(𝐌) is irretrievably lost. With Fmel=80 and K=513, the filterbank discards at least 51380=433 dimensions of spectral information per frame.

Proof.

Since 𝐌Fmel×K with Fmel<K, rank(𝐌)Fmel<K. By the rank-nullity theorem, dim(ker(𝐌))=Krank(𝐌)KFmel. Any vector in ker(𝐌) is mapped to zero, so spectral detail in those directions cannot be recovered from the mel-spectrogram.

Representation Taxonomy

Modern audio generation systems operate on one of several representations, each offering a different trade-off between reconstruction fidelity, dimensionality, and suitability for generative modelling. We classify the main options here for reference; subsequent sections develop each in detail.

  1. Raw waveform (𝒙L). The highest-fidelity representation, but extremely high-dimensional and temporally fine-grained. Direct waveform models (WaveNet, SampleRNN) operate here but face severe computational costs.

  2. Mel-spectrogram (𝐒logFmel×M). A compressed, perceptually motivated 2D representation. Dimensionality is reduced by a factor of H/(2Fmel) relative to the waveform, but reconstruction requires a vocoder to estimate phase and fine spectral detail.

  3. Discrete codec tokens (𝒄{1,,V}Q×Tc). A sequence of integer codes produced by a neural audio codec (SoundStream, EnCodec, DAC) with Q quantisation levels and Tc time frames. This is the most compressed representation, enabling language-model-style generation, but introduces quantisation artefacts.

  4. Continuous latents (𝒛d×Tz). The bottleneck representation of a continuous (non-quantised) autoencoder. Latent diffusion and flow matching models operate here, combining the compression benefits of codecs with the smoothness of continuous spaces.

tableAudio representation taxonomy. Each representation trades off dimensionality, information content, and compatibility with different generative model families.

RepresentationTypeDim. reductionInvertible?Compatible models
Raw waveform1×YesAR, GAN
Mel-spectrogram3×No (vocoder)Diffusion, flow
Continuous latent10×No (decoder)Diffusion, flow
Discrete tokens{1,,V}40×No (decoder)AR, masked LM

Proposition 7 (Compression-Fidelity Hierarchy).

The four representations form a strict hierarchy in terms of information content: (INFO Hierarchy)H(𝒙)H(𝐒log)H(𝒛)H(𝒄), where H() denotes the (differential) entropy. Each transformation discards information:

  1. 𝒙𝐒log: discards phase and fine spectral detail beyond the mel resolution.

  2. 𝐒log𝒛: the autoencoder bottleneck discards information not captured by the latent dimensionality d.

  3. 𝒛𝒄: vector quantisation introduces an error of at most 𝒓Q22 per frame (by thm:audiogen:rvq-bound).

The generative modelling task becomes easier at each level (lower entropy, fewer dimensions), but the reconstruction ceiling also drops: information discarded during encoding cannot be recovered during decoding.

Caution.

The choice of representation is not merely an implementation detail; it fundamentally determines which generative model families are applicable. Autoregressive language models require discrete tokens (representation 3). Diffusion and flow matching require continuous spaces (representations 2 or 4). GANs can operate on either waveforms (1) or spectrograms (2). Choosing a representation is therefore inseparable from choosing a generative paradigm.

Example 4 (Representation dimensions for 10,s of audio at 24,kHz).

RepresentationShapeDimsType
Raw waveform240,000240,000
Mel-spectrogram (H=320, Fmel=80)80×75060,000
Continuous latent (d=128, S=320)128×75096,000
Codec tokens (Q=8, S=320)8×7506,000{1,,1024}
The codec tokens achieve a 40× reduction in the number of entries relative to the raw waveform, with each entry drawn from a finite vocabulary of 1,024 symbols. This is the representation that makes audio generation with language models tractable.

Neural Audio Vocoders

A vocoder is a system that converts a compact acoustic representation (classically, linear predictive coding parameters; in modern systems, a mel-spectrogram or similar feature) into a waveform. In the context of audio generation, the vocoder is the final stage that transforms a generated spectrogram into an audible signal. Its quality sets the ceiling for the overall system: a perfect generative model paired with a poor vocoder will produce poor audio.

Neural vocoders have evolved rapidly since WaveNet first demonstrated that a neural network could synthesise perceptually natural waveforms. In this section, we focus on HiFi-GAN [17], which has become the de facto standard vocoder in modern audio generation pipelines due to its combination of high fidelity, fast inference, and training stability. We develop its architecture in detail, connecting its multi-scale and multi-period discriminator designs to the GAN theory developed in ch:gan.

The Vocoding Problem

Formally, the vocoding problem is a conditional generation task.

Definition 8 (Vocoding).

Given a mel-spectrogram 𝐒logFmel×M, the vocoding problem is to generate a waveform 𝒙^L such that:

  1. 𝒙^ sounds perceptually indistinguishable from the ground-truth waveform 𝒙 that produced 𝐒log, and

  2. the mel-spectrogram of 𝒙^ closely matches 𝐒log.

Since the mel-spectrogram discards phase and fine spectral detail, the mapping 𝐒log𝒙^ is inherently one-to-many: many waveforms share the same mel-spectrogram. The vocoder must learn to sample from the conditional distribution p(𝒙|𝐒log).

Remark 7.

The vocoder performs a massive upsampling. A mel-spectrogram at Fmel=80 bands with hop size H=256 has 80×M values, while the corresponding waveform has L=HM samples. The upsampling factor along the time axis is H=256, and the vocoder must fill in all the fine temporal structure (individual oscillation cycles, transient attacks, noise textures) that the mel-spectrogram averages over.

Historical Note.

The evolution of neural vocoders. The vocoder landscape has evolved through three generations. First-generation models (WaveNet, 2016; WaveRNN, 2018) achieved high quality through autoregressive sample-by-sample generation, but required seconds to minutes of computation per second of audio. Second-generation models (Parallel WaveNet, 2018; WaveGlow, 2019) used knowledge distillation or flow-based architectures to achieve parallel synthesis, reducing latency to real-time but at the cost of increased model complexity and training difficulty. Third-generation models (MelGAN, 2019; HiFi-GAN, 2020; BigVGAN, 2023) adopted GAN training with carefully designed discriminators, achieving both high quality and extreme speed (100–300× real-time on GPU). HiFi-GAN's insight that periodic and multi-scale discriminators complement each other has proven remarkably durable: variants of its discriminator architecture appear in nearly every subsequent vocoder and neural audio codec.

tableComparison of neural vocoder approaches. Speed is measured relative to real-time on a single GPU\@. MOS denotes mean opinion score on a 1–5 scale.

ModelParadigmSpeedMOSKey Innovation
WaveNetAutoregressive0.001×4.2Dilated causal conv.
WaveRNNAutoregressive0.4×4.1Subscale prediction
WaveGlowFlow10×4.0Invertible 1×1 conv.
MelGANGAN100×3.8Multi-scale disc.
HiFi-GANGAN170×4.3MPD + MSD
BigVGANGAN120×4.4Anti-aliased + Snake

HiFi-GAN: Generator Architecture

The HiFi-GAN generator is a fully convolutional feed-forward network that progressively upsamples the mel-spectrogram to the waveform sampling rate through a cascade of transposed convolution blocks, each followed by a multi-receptive-field fusion (MRF) module.

Definition 9 (HiFi-GAN Generator).

The HiFi-GAN generator Gθ:Fmel×ML consists of:

  1. An input convolution that projects the Fmel-dimensional mel features to a high-dimensional channel space of width Cinit (typically 512).

  2. A sequence of B upsampling blocks, where the b-th block applies a transposed 1D convolution with stride rb, increasing the temporal resolution by a factor of rb. The strides are chosen so that b=1Brb=H, the hop size used during mel-spectrogram extraction.

  3. Within each upsampling block, a multi-receptive-field fusion (MRF) module consisting of R parallel residual blocks, each with a different kernel size κr and dilation pattern. The outputs of the R branches are summed (or averaged) to produce the block output: (MRF)𝒉b=1Rr=1RResBlockr(𝒉b1), where 𝒉b1 denotes the upsampled hidden representation from the transposed convolution.

  4. A final convolution with a tanh activation that maps the Cfinal-channel hidden state to a single-channel waveform 𝒙^[1,1]L.

Example 5 (Typical HiFi-GAN configuration).

The “V1” configuration of HiFi-GAN uses Cinit=512, B=4 upsampling blocks with strides (r1,r2,r3,r4)=(8,8,2,2) (so that 8×8×2×2=256=H), and R=3 residual blocks per MRF module with kernel sizes (κ1,κ2,κ3)=(3,7,11). The total parameter count is approximately 14 million, and inference runs at over 100× real-time on a single GPU\@. Lighter configurations (“V2”, “V3”) trade quality for speed by reducing Cinit and the number of residual blocks.

Remark 8.

The MRF module is central to HiFi-GAN's success. By fusing the outputs of residual blocks with kernel sizes κ{3,7,11} and dilation rates {1,3,5} (or {1,1,1} for the smallest kernel), each MRF module simultaneously captures patterns at multiple temporal scales within a single upsampling stage. Small kernels capture fine-grained sample-level correlations (e.g., the waveform of a single pitch cycle), while large kernels capture broader patterns (e.g., the modulation envelope of a tremolo effect). This multi-scale design within the generator complements the multi-scale structure of the discriminators.

HiFi-GAN V1 generator architecture. The mel-spectrogram is projected to 512 channels and then progressively upsampled by factors of 8×8×2×2=256. Each upsampling stage is followed by a multi-receptive-field fusion (MRF) module with three parallel residual blocks. The final convolution with tanh produces a single-channel waveform.

Multi-Period and Multi-Scale Discriminators

The adversarial training of HiFi-GAN relies on two complementary discriminator families, each designed to capture different structural properties of audio waveforms. The design of these discriminators reflects a deep understanding of audio signal structure and connects directly to the GAN theory developed in ch:gan and the Wasserstein framework of ch:wgan.

Definition 10 (Multi-Period Discriminator (MPD)).

The multi-period discriminator consists of P sub-discriminators {Dp}p𝒫, where 𝒫={2,3,5,7,11} is a set of prime periods. Each sub-discriminator Dp operates as follows:

  1. Reshape the 1D waveform 𝒙^L into a 2D tensor of shape L/p×p by folding every p-th sample into a column.

  2. Apply a stack of 2D strided convolutions to this reshaped signal, producing a scalar prediction Dp(𝒙^).

Each sub-discriminator “sees” the waveform at a different periodic granularity: D2 captures even/odd sample structure, D3 detects triplet patterns, and so on. The prime periods ensure that no two sub-discriminators share the same periodic folding.

Insight.

Audio waveforms are highly periodic: a sung vowel at 220,Hz repeats its fundamental cycle every fs/220200 samples (at fs=44,100,Hz). The MPD exploits this structure by reshaping the waveform so that periodic patterns become spatial patterns in a 2D grid, which standard 2D convolutions can detect efficiently. Using prime periods guarantees that the sub-discriminators provide non-redundant feedback about different periodic components of the signal.

Proposition 8 (MPD Period Coverage).

Let 𝒫={2,3,5,7,11} be the set of MPD periods. For any integer period p235711=2310 (corresponding to a fundamental frequency f0fs/231019,Hz at fs=44,100,Hz), at least one discriminator period p𝒫 divides p, ensuring that the corresponding sub-discriminator sees a perfectly aligned periodic pattern. Since the primes in 𝒫 are the first five primes, any composite period up to p has at least one prime factor in 𝒫.

Proof.

Every integer p2 has at least one prime factor. If that factor is in {2,3,5,7,11}, the corresponding sub-discriminator sees the periodic pattern aligned to columns. The smallest prime not in 𝒫 is 13, so only periods whose smallest prime factor is 13 (i.e., p{13,17,19,23,}) are not directly captured. In practice, the fundamental periods of musical and speech signals at standard sample rates are almost always composite numbers with small prime factors.

Definition 11 (Multi-Scale Discriminator (MSD)).

The multi-scale discriminator consists of S sub-discriminators {Ds}s=1S, each operating on the waveform at a different temporal resolution:

  1. D1 receives the raw waveform 𝒙^.

  2. D2 receives 𝒙^ after average-pooling by a factor of 2.

  3. Ds receives 𝒙^ after average-pooling by a factor of 2s1.

Each sub-discriminator is a 1D convolutional network with grouped convolutions and spectral normalisation (see ch:gan,ch:wgan). D1 captures sample-level detail (high-frequency content, transient attacks), while DS captures coarse temporal structure (rhythm, envelope).

Remark 9.

The MPD and MSD are complementary. The MPD is sensitive to periodic artefacts (pitch errors, harmonicity violations) but relatively insensitive to broadband noise. The MSD captures scale-dependent features (the shape of a drum transient at fine scale, the overall loudness envelope at coarse scale) but may miss subtle pitch errors. Together, they provide a comprehensive adversarial signal that covers both periodic and aperiodic aspects of audio quality.

Remark 10.

All MSD sub-discriminators apply spectral normalisation (see ch:wgan) to every convolutional layer. Spectral normalisation constrains the Lipschitz constant of the discriminator, which stabilises training and provides an implicit connection to the Wasserstein distance. The MPD sub-discriminators use weight normalisation instead, as the 2D periodic reshaping already imposes structural constraints that aid stability.

HiFi-GAN Training Losses

The HiFi-GAN training objective combines adversarial losses, feature matching losses, and a spectral reconstruction loss. We develop each component and then state the full training procedure.

Adversarial losses.

Each sub-discriminator Dk (from both MPD and MSD) is trained with the least-squares GAN (LSGAN) objective (see ch:gan): (Hifigan ADV D)adv(Dk)=𝔼𝒙[(Dk(𝒙)1)2]+𝔼𝒙^[Dk(𝒙^)2],adv(G,Dk)=𝔼𝒙^[(Dk(𝒙^)1)2]. The total adversarial loss sums over all sub-discriminators: (Hifigan ADV Total)adv(D)=kadv(Dk),adv(G)=kadv(G,Dk).

Feature matching loss.

For each sub-discriminator Dk with Nk intermediate layers, let Dk(l)(𝒙)dl denote the activation at layer l. The feature matching loss encourages the generator to produce waveforms whose internal discriminator representations match those of real audio: (Hifigan FM)fm=kl=1Nk1dl𝔼[Dk(l)(𝒙)Dk(l)(𝒙^)1]. This provides a stable, multi-scale gradient signal to the generator, analogous to the perceptual loss used in image super-resolution.

Multi-resolution STFT loss.

To accelerate early training and improve fine spectral detail, the generator is additionally supervised with a reconstruction loss computed in the STFT domain at multiple resolutions.

Definition 12 (Multi-Resolution STFT Loss).

Let ={(Nfft(i),H(i),W(i))}i=1I be a set of I STFT configurations (FFT size, hop size, window size). For the i-th configuration, let |𝐗(i)| and |𝐗^(i)| denote the magnitude spectrograms of the real and generated waveforms, respectively. The multi-resolution STFT loss is (MR STFT)STFT=i=1I(|𝐗(i)||𝐗^(i)|F|𝐗(i)|Fspectral convergence+1KiMilog|𝐗(i)|log|𝐗^(i)|1log-magnitude 1), where F is the Frobenius norm, and Ki,Mi are the frequency and time dimensions for the i-th configuration.

Remark 11.

A common choice is I=3 with configurations (Nfft,H,W){(512,50,240),(1024,120,600),(2048,240,1200)}. Small FFT sizes capture transients and high-frequency detail; large FFT sizes capture low-frequency structure and pitch accuracy. The spectral convergence term normalises by the signal energy, preventing loud passages from dominating the loss, while the log-magnitude term emphasises quiet components.

Remark 12.

An alternative to the multi-resolution STFT loss is the mel-spectrogram reconstruction loss: (MEL Recon LOSS)mel=𝐒log(𝒙)𝐒log(𝒙^)1, which directly penalises differences in the log-mel-spectrogram domain. The mel-spectrogram loss is perceptually motivated (it operates in the same space that human audition roughly perceives) but provides less detailed spectral supervision than the multi-resolution STFT loss. Many systems (including the original HiFi-GAN) use the mel loss as the primary reconstruction term, while codec systems (SoundStream, EnCodec, DAC) prefer the multi-resolution STFT loss for its finer spectral detail.

Full training objective.

The generator is trained to minimise (Hifigan FULL)G=adv(G)+λfmfm+λSTFTSTFT, with λfm=2 and λSTFT=45 as typical values. The discriminators are trained to minimise adv(D). Training alternates between discriminator and generator updates, following the standard GAN training protocol of ch:gan.

Algorithm 1 (HiFi-GAN Training).

  1. Input: Training set of (mel-spectrogram, waveform) pairs {(𝐒log(i),𝒙(i))}; generator Gθ; MPD {Dp}; MSD {Ds}; learning rate η; loss weights λfm,λSTFT.

  2. Repeat until convergence: enumerate

  3. Sample a mini-batch (𝐒log,𝒙).

  4. Generate 𝒙^=Gθ(𝐒log).

  5. Discriminator step: Update all sub-discriminators by descending ϕadv(D).

  6. Generator step: Update Gθ by descending θG with G from (Hifigan FULL). enumerate

  7. Output: Trained generator Gθ (the vocoder).

Insight.

The HiFi-GAN training procedure is a direct instantiation of the GAN framework from ch:gan, with two notable specialisations. First, the generator is conditioned on the mel-spectrogram rather than on random noise, making it a conditional GAN (cGAN). The noise required for the one-to-many vocoding mapping comes from the training dynamics themselves: the generator learns a deterministic mapping that produces one plausible phase realisation. Second, the use of multiple discriminators (MPD + MSD) can be viewed as an ensemble of critics, each providing gradient information about a different aspect of audio quality. This ensemble approach is more stable than a single monolithic discriminator because individual sub-discriminators can specialise on different failure modes.

Neural Audio Codecs

The mel-spectrogram is a useful intermediate representation, but it is continuous and high-dimensional (typically 80×M floats). For autoregressive generation with language models, we need a discrete representation: a sequence of tokens from a finite vocabulary, analogous to the word tokens in text. Neural audio codecs provide exactly this, compressing audio into sequences of discrete codes while maintaining high reconstruction quality.

The key innovation that makes neural audio codecs practical is Residual Vector Quantisation (RVQ), which achieves high codebook utilisation and fine-grained quantisation by applying multiple rounds of vector quantisation to the residuals of previous rounds. In this section, we develop RVQ mathematically, then examine three influential codec architectures: SoundStream, EnCodec, and DAC\@.

Vector Quantisation Revisited

We briefly recall the essentials of vector quantisation (VQ) before developing the residual extension. The reader may consult ch:vae for a thorough treatment of VQ-VAE foundations.

Definition 13 (Vector Quantiser).

A vector quantiser VQ with codebook 𝒞={𝒆1,,𝒆V}d maps a continuous vector 𝒛d to its nearest codebook entry: (VQ)VQ(𝒛;𝒞)=𝒆k,k=arg mink{1,,V}𝒛𝒆k22. The index k is the code or token assigned to 𝒛, and V is the codebook size (vocabulary size).

A single vector quantiser with codebook size V can represent at most V distinct vectors, yielding log2V bits per frame. For V=1024 (a common choice), this is only 10 bits per frame. At a codec frame rate of 75 frames per second, the resulting bitrate is 750 bits per second, far too low for high-fidelity audio reconstruction. Naively increasing V to, say, 220 would provide 20 bits per frame but would make codebook learning intractable: most entries would never be used, and the straight-through gradient estimator would become unreliable.

Residual Vector Quantisation

Residual Vector Quantisation (RVQ) elegantly solves this problem by decomposing the quantisation into a cascade of Q stages, each quantising the residual left by the previous stages.

Definition 14 (Residual Vector Quantisation).

Let 𝒞1,,𝒞Q be Q codebooks, each of size V with entries in d. The Residual Vector Quantiser RVQ:dd operates as follows:

  1. Initialise the residual 𝒓0=𝒛.

  2. For q=1,,Q: enumerate

  3. Quantise the residual: 𝒓^q=VQ(𝒓q1;𝒞q).

  4. Record the code index cq=arg mink𝒓q1𝒆k(q)22.

  5. Update the residual: 𝒓q=𝒓q1𝒓^q. enumerate

  6. Return the quantised vector 𝒛^=q=1Q𝒓^q.

The output is a sequence of Q code indices (c1,c2,,cQ){1,,V}Q.

Theorem 1 (RVQ Approximation Bound).

Let 𝒛d and let 𝒛^=q=1Q𝒓^q be its RVQ approximation with Q codebooks, each of size V. If the codebook entries are chosen optimally (i.e., each 𝒞q is the optimal quantiser for the distribution of 𝒓q1), then:

  1. The residual norms are non-increasing: 𝒓q2𝒓q12 for all q.

  2. The total quantisation error satisfies (RVQ Error)𝒛𝒛^22=𝒓Q22q=1Q(11αq)𝒛22, where αq1 is the quantisation gain of the q-th stage, defined as the ratio of input variance to quantisation error variance: αq=𝔼[𝒓q122]/𝔼[𝒓q22].

  3. The effective bitrate is Qlog2V bits per frame, growing linearly with the number of quantisation stages.

Proof.

Part 1. By construction, 𝒓q=𝒓q1𝒓^q, where 𝒓^q is the nearest codebook vector to 𝒓q1. Since 0 is a valid approximation (we can always include 0 in the codebook, or note that 𝒓^q minimises the distance to 𝒓q1), 𝒓q2=𝒓q1𝒓^q2𝒓q12.

Part 2. The quantisation error after Q stages is 𝒓Q22. At each stage, 𝔼[𝒓q22]=𝔼[𝒓q122]/αq by definition of αq. Telescoping gives 𝔼[𝒓Q22]=𝔼[𝒓022]q=1Qαq=𝔼[𝒛22]q=1Qαq=q=1Q1αq𝔼[𝒛22]. Writing 1/αq=1(11/αq) and noting that αq1 implies 11/αq[0,1), the bound follows.

Part 3. Each of the Q stages contributes one code index from a vocabulary of size V, yielding log2V bits per stage and Qlog2V bits total.

Lemma 1 (Orthogonality of RVQ Residuals).

If the codebooks 𝒞1,,𝒞Q are optimally adapted to the data distribution, then the quantised vectors 𝒓^q and the residuals 𝒓q are uncorrelated at each stage: (RVQ Uncorrelated)𝔼[𝒓^q𝒓q]=0for all q=1,,Q. This is the vector quantisation analogue of the normal equations in least-squares regression: the “prediction” 𝒓^q and the “residual” 𝒓q are orthogonal in expectation.

Proof.

For an optimal codebook, each Voronoi cell has its centroid at the codebook entry. That is, 𝒆k=𝔼[𝒓q1|cq=k] for each k. By the law of total expectation, 𝔼[𝒓^q𝒓q]=𝔼[𝒓^q(𝒓q1𝒓^q)]=𝔼[𝒓^q𝒓q1]𝔼[𝒓^q22]. Conditioning on the cell assignment: 𝔼[𝒓^q𝒓q1|cq=k]=𝒆k𝔼[𝒓q1|cq=k]=𝒆k22. Therefore 𝔼[𝒓^q𝒓q1]=𝔼[𝒓^q22], giving 𝔼[𝒓^q𝒓q]=0.

Proposition 9 (RVQ Effective Codebook Size).

An RVQ with Q codebooks of size V can represent up to VQ distinct vectors. The effective codebook size grows exponentially with the number of stages while each individual codebook remains tractably small: (RVQ Effective SIZE)Veff=VQ. For the common setting V=1024 and Q=8, this yields Veff=10248=2801.2×1024, a codebook that would be completely intractable to train as a single flat quantiser.

Proof.

Each of the Q stages independently selects one of V codebook entries. The number of distinct sums 𝒛^=q=1Q𝒓^q is at most VQ, achieved when no two distinct code sequences produce the same sum. In practice, the residual structure ensures that most code combinations yield distinct vectors, so the effective codebook size is close to the theoretical maximum.

Residual Vector Quantisation (RVQ) cascade. The input 𝒛 is quantised by the first codebook 𝒞1, and the residual 𝒓1=𝒛𝒓^1 is passed to the next stage. Each successive stage quantises the residual from the previous stage, producing a code index cq. The final reconstruction 𝒛^ is the sum of all quantised vectors. The code sequence (c1,,cQ) is the compressed discrete representation of 𝒛.

SoundStream

SoundStream [24] was the first neural audio codec to combine a convolutional encoder-decoder with RVQ, establishing the architectural template that EnCodec and DAC would later refine.

Definition 15 (SoundStream Architecture).

SoundStream consists of three components:

  1. An encoder Eθ:Ld×Tc that maps a mono waveform 𝒙L to a sequence of Tc continuous latent vectors of dimension d, where Tc=L/S and S is the total downsampling stride.

  2. A residual vector quantiser RVQ:dd with Q codebooks of size V, applied independently to each of the Tc latent vectors. The output is a matrix of codes 𝐂{1,,V}Q×Tc.

  3. A decoder Dϕ:d×TcL that reconstructs the waveform from the quantised latent sequence 𝒛^1:Tc.

The encoder consists of a 1D convolution followed by Be downsampling blocks, each containing residual units and a strided convolution with stride sb. The decoder mirrors this structure with transposed convolutions. The total stride is S=b=1Besb.

Example 6 (SoundStream at 6,kbps).

With fs=24,000,Hz, total stride S=320, codebook size V=1024, and Q=8 quantisation stages, SoundStream produces Tc=24,000/320=75 frames per second, each represented by Q=8 codes of log2(1024)=10 bits each. The bitrate is 75×8×10=6,000 bits per second =6,kbps. This is a compression ratio of approximately 64× compared to 16-bit PCM at 24,kHz (384,kbps), yet reconstructed audio is nearly indistinguishable from the original in listening tests.

SoundStream is trained with a combination of reconstruction and adversarial losses: (Soundstream LOSS)SS=λreconrecon+λadvadv+λfmfm+λvqvq, where recon is a multi-resolution STFT loss (Definition 12), adv and fm are adversarial and feature matching losses from a wave-based discriminator, and vq is the codebook commitment loss: (VQ LOSS)vq=q=1Q(sg[𝒓q1]𝒓^q22+β𝒓q1sg[𝒓^q]22), with sg[] denoting the stop-gradient operator and β the commitment cost weight.

EnCodec

EnCodec [25] refines the SoundStream architecture with several improvements: a higher-capacity encoder-decoder, a novel loss balancing mechanism, and support for variable bitrates through codebook dropout.

Definition 16 (EnCodec Architecture).

EnCodec shares the encoder-RVQ-decoder structure of SoundStream but introduces:

  1. Larger model capacity: the encoder and decoder use deeper residual networks with more channels, and the encoder additionally incorporates a two-layer LSTM between the convolutional stack and the quantiser to capture long-range temporal dependencies.

  2. Multi-scale STFT discriminator: instead of a single waveform-domain discriminator, EnCodec uses a complex-valued multi-scale STFT discriminator that operates on both real and imaginary parts of the STFT at multiple resolutions.

  3. Loss balancer: a gradient-balancing mechanism that dynamically rescales loss components to ensure stable training (see def:audiogen:encodec-balancer).

  4. Variable bitrate via codebook dropout: during training, each RVQ stage q>1 is randomly dropped with some probability, forcing the decoder to reconstruct from fewer codes. At inference time, the user can truncate the code sequence at any stage to achieve a lower bitrate.

The loss balancer deserves special attention, as it addresses a practical difficulty in training multi-loss systems.

Definition 17 (EnCodec Loss Balancer).

Let {i}i=1NL be the NL loss components with target weights {λi}i=1NL. The EnCodec balancer rescales the gradient of each loss component so that its contribution to the total gradient norm is proportional to its weight: (Encodec Balancer)g~i=λigθi2+ϵθi, where g is the exponential moving average of the gradient norms across all losses, and ϵ is a small constant. The total gradient applied to the generator is i=1NLg~i.

Remark 13.

Without the balancer, a loss component with large gradients (e.g., the adversarial loss during early training) can dominate the parameter update and destabilise training. The balancer normalises each gradient to unit norm and then rescales by the target weight, ensuring that the relative influence of each loss component matches the designer's intention regardless of their absolute magnitudes. This is particularly important for audio codecs, where the reconstruction, adversarial, feature matching, and commitment losses can differ by orders of magnitude.

Example 7 (EnCodec bitrate configurations).

EnCodec supports multiple target bitrates by varying the number of active RVQ stages:

tableEnCodec bitrate configurations at 24,kHz with V=1024, stride S=320 (75 frames/s).

BitrateQ (stages)Bits/frameQuality
1.5,kbps220Telephony
3.0,kbps440Wideband speech
6.0,kbps880Near-transparent
12.0,kbps16160Transparent
24.0,kbps32320Studio quality
Codebook dropout during training ensures that the decoder performs well even when receiving fewer than the maximum number of codes.

DAC: Descript Audio Codec

The Descript Audio Codec (DAC) [26] pushes neural codec quality further through three innovations: improved codebook learning, the Snake activation function, and a multi-band discriminator.

Definition 18 (Snake Activation).

The Snake activation function is defined as (Snake)Snake(x;α)=x+1αsin2(αx), where α>0 is a learnable frequency parameter. Snake is a periodic nonlinearity that preserves the identity component x while adding a sinusoidal perturbation. The frequency α controls the period of the oscillation.

Remark 14.

Standard activation functions (ReLU, GELU, SiLU) have no inherent periodic structure. Audio waveforms, by contrast, are dominated by periodic and quasi-periodic components (pitched instruments, vowels, harmonic series). The Snake activation provides a mild inductive bias towards periodicity in the learned features, which DAC's authors found to improve reconstruction quality, particularly for tonal signals such as singing voice and sustained instrument notes. The identity component x ensures that the activation can still represent aperiodic signals (noise, transients) without difficulty.

Proposition 10 (Properties of the Snake Activation).

The Snake activation σ(x;α)=x+1αsin2(αx) satisfies:

  1. Monotonicity: σ(x;α)=1+sin(2αx)0 for all x, with equality only when sin(2αx)=1.

  2. Asymptotic linearity: as α0, σ(x;α)x+αx2+O(α2); as α, the oscillatory component averages out and σ(x;α)x+1/(2α).

  3. Periodicity: the deviation from linearity, σ(x;α)x=12α(1cos(2αx)), has period π/α.

Proof.

Part 1. Using sin2(αx)=12(1cos(2αx)), we have σ(x)=1+ddx[1αsin2(αx)]=1+sin(2αx). Since 1sin(2αx)1, we get 0σ(x)2.

Part 2. For small α, sin2(αx)α2x2, so σ(x;α)x+αx2. For large α, the time-averaged value of sin2(αx) is 1/2, giving σx+1/(2α).

Part 3. The identity sin2(θ)=(1cos2θ)/2 yields σ(x)x=12α(1cos(2αx)), which has period 2π/(2α)=π/α.

Definition 19 (DAC Improved Codebook Learning).

DAC improves codebook utilisation through two mechanisms:

  1. Factorised codes: each encoder output 𝒛d is projected to a lower-dimensional space 𝒛=𝐖𝒛d with dd (typically d=8) before quantisation. The codebook vectors also live in d, and the quantised code is projected back: 𝒛^=𝐖𝒛^. This forces the codebook to capture the most important variation in a compressed space, improving utilisation.

  2. 2-normalised codes: both the encoder output 𝒛 and the codebook entries 𝒆k are projected onto the unit sphere before computing distances: (DAC L2 VQ)k=arg mink𝒛𝒛2𝒆k𝒆k222. This is equivalent to maximising cosine similarity and prevents codebook collapse, where entries drift far from the encoder output distribution.

Example 8 (DAC at 8,kbps).

The 44.1,kHz variant of DAC uses stride S=512, yielding approximately 86 frames per second. With Q=9 codebooks of size V=1024: Bitrate=86×9×10=7,7408kbps. DAC achieves higher reconstruction quality than EnCodec at comparable bitrates, particularly for music, due to the combination of Snake activations, factorised codebooks, and a multi-band multi-scale discriminator that provides sharper adversarial gradients in different frequency ranges.

Codec Comparison

tab:audiogen:codec-comparison summarises the key design choices across the three major neural audio codecs.

tableComparison of neural audio codec architectures. All use RVQ with V=1024 codebooks. “MR-STFT” denotes multi-resolution STFT loss; “MS-STFT-D” denotes multi-scale STFT discriminator.

FeatureSoundStreamEnCodecDAC
Sample rate24,kHz24/48,kHz16/24/44.1,kHz
Stride S320320320/512
Frame rate75,fps75/50,fps75/86,fps
Q (stages)8–122–329–12
Bitrate range3–18,kbps1.5–24,kbps4–16,kbps
Encoder LSTMNoYesNo
ActivationELUELUSnake
DiscriminatorMSDMS-STFT-DMS+MB-D
Codebook initK-meansEMA2-norm
Loss balancingManualBalancerManual
Variable bitrateNoYesNo

Insight.

Neural audio codecs serve a dual purpose in modern audio generation systems. First, they act as compressors, reducing audio to a fraction of its original bitrate for storage and transmission. Second, and more importantly for this chapter, they act as tokenisers that convert continuous audio into discrete token sequences, enabling autoregressive language models to generate audio by predicting code sequences. The quality of the codec directly determines the quality ceiling of the downstream generative model: any artefact introduced during quantisation will persist in the generated audio regardless of how well the language model performs.

Caution.

The codec's training data distribution strongly influences its reconstruction quality across audio types. A codec trained predominantly on speech may introduce artefacts when encoding music, and vice versa. EnCodec and DAC are trained on diverse audio (speech, music, environmental sounds) to mitigate this, but domain-specific fine-tuning can yield significant improvements for specialised applications.

The RVQ Token Structure

The output of an RVQ codec is not a single flat sequence of tokens but a matrix of codes 𝐂{1,,V}Q×Tc, where Q is the number of quantisation stages and Tc is the number of time frames. This 2D structure poses a fundamental challenge for autoregressive models, which operate on 1D sequences.

There are three principal strategies for linearising the code matrix:

Flat pattern.

Interleave all codes in raster order: (c1,1,c2,1,,cQ,1,c1,2,c2,2,). This yields a sequence of length QTc and preserves all dependencies but requires the model to generate Q tokens per audio frame, which is slow.

Delay pattern.

Introduced by MusicGen [22], the delay pattern offsets each quantisation level by one time step: level q at time t is placed at sequence position t+q1. This allows parallel prediction of codes at different levels for the same time step, reducing the effective sequence length to Tc+Q1 while maintaining causal ordering.

Coarse-to-fine.

First generate all Tc codes at level q=1 (the coarsest level), then all codes at level q=2 conditioned on level 1, and so on. This hierarchical approach (used by MusicLM and VALL-E) decouples the coarse acoustic structure from the fine detail but requires Q sequential passes through the model.

Remark 15.

The choice of linearisation strategy directly affects generation speed, quality, and the model's ability to capture long-range dependencies. The flat pattern is the most expressive but the slowest; the delay pattern offers a practical compromise; the coarse-to-fine approach can leverage specialised models for each level but introduces a cascading error risk (errors at coarse levels propagate to all finer levels). We revisit these tradeoffs in detail when we study autoregressive generation in CLAP: Contrastive Language–Audio Pretraining and subsequent sections.

Training Neural Audio Codecs: A Unified View

All three codecs (SoundStream, EnCodec, DAC) share a common training framework that combines four families of loss functions. We consolidate these here to provide a unified perspective.

Definition 20 (Neural Audio Codec Training Objective).

A neural audio codec with encoder Eθ, RVQ RVQ, and decoder Dϕ is trained by minimising (Codec FULL LOSS)=λrrecon+λaadv+λffm+λccommit, where:

  1. recon is a reconstruction loss in the time and/or spectral domain (multi-resolution STFT loss, mel-spectrogram 1 loss, or waveform 1 loss).

  2. adv is an adversarial loss from one or more discriminators (MPD, MSD, multi-scale STFT discriminator, or multi-band discriminator).

  3. fm is a feature matching loss computed over intermediate discriminator activations.

  4. commit is a commitment loss that regularises the encoder outputs to stay close to the codebook entries, ensuring stable quantisation.

The weights λr,λa,λf,λc control the relative influence of each component. In EnCodec, these are managed dynamically by the loss balancer (Definition 17).

Proposition 11 (Straight-Through Estimator for RVQ).

The vector quantisation operation VQ(𝒛;𝒞) is piecewise constant and therefore has zero gradient almost everywhere. To enable gradient-based training, the straight-through estimator (STE) replaces the gradient of the quantisation operation with the identity: (STE)𝒛^𝒛=STE𝐈d, where 𝐈d is the d×d identity matrix. In the forward pass, 𝒛^=VQ(𝒛;𝒞); in the backward pass, gradients flow through as if 𝒛^=𝒛. For RVQ with Q stages, the STE is applied at each stage independently, and the total gradient through the Q-stage cascade is (STE RVQ Total)𝒛^𝒛=𝒛q=1Q𝒓^q=STEq=1Q𝐈d=Q𝐈d. In practice, the gradient is normalised by Q to prevent the effective learning rate from scaling with the number of stages.

Proof.

At each stage q, the forward computation is 𝒓^q=VQ(𝒓q1;𝒞q) and 𝒓q=𝒓q1𝒓^q. Under the STE, 𝒓^q/𝒓q1=𝐈d. Since 𝒛^=q=1Q𝒓^q and 𝒓0=𝒛, we have 𝒓^1/𝒛=𝐈d. For stage q>1, 𝒓q1=𝒛i=1q1𝒓^i, so 𝒓q1/𝒛=𝐈di=1q1𝒓^i/𝒛. By induction, if each 𝒓^i/𝒛=𝐈d under the STE, then 𝒛^/𝒛=Q𝐈d.

Remark 16.

An alternative to gradient-based codebook learning is the exponential moving average (EMA) update rule. For each codebook entry 𝒆k, let k denote the set of encoder outputs assigned to 𝒆k in the current mini-batch. The EMA update is (EMA Count)nkγnk+(1γ)|k|,𝒎kγ𝒎k+(1γ)𝒛k𝒛,𝒆k𝒎k/nk, where γ(0,1) is the decay rate (typically γ=0.99). This rule moves each codebook entry towards the centroid of its Voronoi cell, analogous to the M-step of the k-means algorithm. EMA updates are used by EnCodec; SoundStream uses k-means initialisation followed by gradient-based updates; DAC uses the 2-normalised variant described in Definition 19.

Insight.

A persistent challenge in training VQ and RVQ systems is codebook collapse: a large fraction of codebook entries receive no assignments and become “dead codes” that contribute nothing to reconstruction quality. Strategies to combat collapse include:

  1. Codebook reset: periodically reinitialise dead entries by sampling from the current batch of encoder outputs.

  2. EMA updates: the smoothing effect of EMA prevents codebook entries from drifting far from the data distribution.

  3. Factorised codes (DAC): projecting to a lower-dimensional space before quantisation concentrates the effective distribution, making it easier for all entries to receive assignments.

  4. 2 normalisation (DAC): constraining both codes and encoder outputs to the unit sphere prevents the distance scale from growing unboundedly, which is a common cause of collapse in high dimensions.

In well-tuned systems, codebook utilisation exceeds 99%, meaning that virtually all entries are actively used during encoding.

From Codecs to Generation

With the codec machinery in place, we can now see how the full audio generation pipeline works in practice. The two-stage paradigm, adopted by virtually all modern systems, is straightforward:

Algorithm 2 (Two-Stage Audio Generation via Neural Codecs).

  1. Stage 1 (Generative model): Given a conditioning signal 𝒄 (text, melody, image, etc.), generate a code matrix 𝐂^{1,,V}Q×Tc using a generative model (autoregressive language model, diffusion model, or flow matching model).

  2. Stage 2 (Codec decoder): Pass 𝐂^ through the pretrained codec decoder Dϕ to obtain the waveform: (TWO Stage Decode)𝒙^=Dϕ(q=1Q𝒆cq,t(q))t=1Tc.

The generative model never sees raw audio; it operates entirely in the discrete token space defined by the codec. The codec decoder is frozen after pretraining and serves as a universal “audio renderer.”

Remark 17.

Freezing the codec decoder during generation has an important consequence: any quantisation artefacts present in the codec's reconstruction become permanent features of the generated audio. This creates a strong incentive to train the highest-quality codec possible, even if the generative model itself is relatively simple. It also means that improvements to the codec (e.g., upgrading from EnCodec to DAC) can improve generation quality without retraining the generative model, provided the new codec uses a compatible tokenisation scheme.

Example 9 (End-to-end latency analysis).

Consider generating 10 seconds of music at 24,kHz using an autoregressive model over EnCodec tokens with Q=8 stages and the delay pattern. The effective sequence length is Tc+Q1=750+7=757 tokens. At an autoregressive generation speed of 100 tokens per second (typical for a 1B parameter transformer with KV-cache on a modern GPU), generation takes approximately 7.6 seconds. The codec decoding step adds negligible overhead (under 50,ms for 10 seconds of audio). The total time is thus dominated by the autoregressive generation, which achieves a real-time factor of approximately 10/7.61.3×.

The sections that follow will develop each generative paradigm in detail: contrastive pretraining for conditioning (CLAP: Contrastive Language–Audio Pretraining), autoregressive generation over codec tokens, latent diffusion over continuous representations, and flow matching for efficient synthesis. In each case, the codec (or, alternatively, a continuous autoencoder) provides the compressed representation that makes generation tractable.

Exercises

Exercise 1 (STFT resolution tradeoff).

Consider an audio signal sampled at fs=22,050,Hz.

  1. Compute the frequency resolution Δf=fs/Nfft and the temporal resolution Δt=H/fs for the following STFT configurations:

    NfftHOverlap
    2566475%
    102425675%
    4096102475%
  2. Show that ΔfΔt=1/NfftH is constant for a fixed overlap ratio, and explain why this reflects an uncertainty principle for time-frequency analysis.

  3. For a music generation system that must faithfully represent both sharp drum transients (requiring fine temporal resolution) and low bass notes at 40,Hz (requiring fine frequency resolution), argue that no single STFT configuration suffices, motivating the multi-resolution approach of Definition 12.

Exercise 2 (Mel filterbank properties).

  1. Verify that the mel scale (Definition 5) is approximately linear for f700,Hz (i.e., mel(f)αf for some constant α) and approximately logarithmic for f700,Hz (i.e., mel(f)βlogf+γ). Derive the constants α, β, and γ.

  2. The mel filterbank matrix 𝐌0Fmel×K is non-square and non-invertible. Explain why the mapping |𝐗|2𝐌|𝐗|2 is lossy, and describe what information is discarded.

  3. Suppose you are designing a mel filterbank for a music generation system operating at fs=44,100,Hz with Nfft=2048. Compute the number of DFT bins K, and determine how many mel filters Fmel are needed so that each filter spans at least two DFT bins at the lowest frequency.

Exercise 3 (RVQ bitrate and quality).

Consider an RVQ codec operating at sample rate fs=24,000,Hz with total encoder stride S=320 and codebook size V=1024.

  1. Derive the frame rate Tc/T in frames per second, where T is the signal duration.

  2. Express the bitrate as a function of the number of RVQ stages Q, and compute the bitrate for Q{2,4,8,16,32}.

  3. Using Theorem 1, if the quantisation gain αq=4 for all stages, compute the fraction of the original signal energy remaining in the residual after Q=8 stages. Is this consistent with the claim that Q=8 at V=1024 yields near-transparent quality?

  4. Explain why doubling V from 1024 to 2048 (adding 1 bit per frame per stage) is less cost-effective than adding one more RVQ stage (adding 10 bits per frame) in terms of bitrate utilisation.

Exercise 4 (HiFi-GAN discriminator analysis).

  1. Consider the multi-period discriminator with period p=5 applied to a waveform of length L=16,000. What is the shape of the reshaped 2D input tensor? How many columns and rows does it have?

  2. A pitched instrument plays a note at fundamental frequency f0=440,Hz, sampled at fs=22,050,Hz. The fundamental period in samples is fs/f050.1. Explain why the sub-discriminator with period p=5 will “see” a clear pattern related to the fundamental (since 50=10×5), while p=7 will produce a less structured 2D image.

  3. The MSD uses average-pooling by factors of 1, 2, and 4. For the sub-discriminator operating at pooling factor 4, what is the effective sample rate? Which frequency components of the signal are invisible to this sub-discriminator?

  4. Argue that the combination of MPD (with prime periods) and MSD (with dyadic scales) provides feedback about both harmonic structure and broadband spectral shape, covering complementary failure modes of the generator.

Exercise 5 (Snake activation analysis).

The Snake activation (Definition 18) is defined as Snake(x;α)=x+1αsin2(αx).

  1. Show that Snake(x;α)=x+12α12αcos(2αx) by using the identity sin2(θ)=(1cos2θ)/2. What is the period of the oscillatory component?

  2. Compute Snake(x;α) and show that it is always positive, proving that Snake is a monotonically increasing function. Why is monotonicity important for an activation function used in a decoder network?

  3. As α0, show that Snake(x;α)x+αx2+O(α2), recovering a near-linear activation. As α, argue informally that the oscillatory component averages out and Snake behaves approximately as x+1/(2α).

  4. Consider a 1D convolutional layer with Snake activation modelling a sinusoidal signal x[n]=Asin(ω0n). Explain qualitatively why the learnable parameter α allows the network to “tune” its internal oscillation frequency to match ω0, providing an inductive bias for periodic signals that standard activations lack.

Exercise 6 (Codec as an information bottleneck).

A neural audio codec with encoder Eθ, RVQ, and decoder Dϕ can be viewed through the lens of the information bottleneck principle. Let 𝒙 denote the input waveform and 𝒄 the discrete code matrix.

  1. Write the codec's objective as a trade-off between minimising reconstruction distortion 𝔼[𝒙Dϕ(𝒄)22] and minimising the information I(𝒙;𝒄) stored in the codes. How does the number of RVQ stages Q control this trade-off?

  2. The rate of the codec in bits per second is R=(fs/S)Qlog2V. For fixed R, compare two configurations: (i) Q=4 stages with V=4096, and (ii) Q=12 stages with V=64. Both achieve R bits per frame. Argue, using the structure of RVQ, why configuration (i) allocates more bits to the first (coarsest) stage while (ii) distributes bits more evenly, and discuss which might produce better reconstruction.

  3. The information bottleneck predicts that the optimal codec should discard information in 𝒙 that is irrelevant to perceptual quality. Give two concrete examples of audio information that a good codec should discard (i.e., not encode in 𝒄), and two examples of information it must preserve.

CLAP: Contrastive Language–Audio Pretraining

Images have CLIP; audio has CLAP\@. The Contrastive Language–Audio Pretraining framework learns a joint embedding space in which audio signals and their textual descriptions are aligned, enabling zero-shot audio classification, text-to-audio retrieval, and, most importantly for this chapter, conditioning signals for generative models. The conceptual parallel with CLIP is deliberate: both methods train dual encoders with an InfoNCE-style contrastive objective, and both produce embeddings that transfer broadly to downstream tasks.

This section formalises the CLAP objective, describes the encoder architectures, and explains why CLAP embeddings have become the default conditioning representation in modern text-to-audio systems. For a thorough treatment of contrastive losses and divergence measures, we refer the reader to ch:divergences.

The Audio–Text Alignment Problem

Consider a dataset of N paired examples {(𝒙i(a),𝒙i(t))}i=1N, where 𝒙i(a)L is an audio waveform (or its mel-spectrogram representation 𝐒iTs×Fs) and 𝒙i(t) is a natural language caption describing the audio content. The audio and text modalities live in fundamentally different spaces: the audio is a continuous, high-dimensional signal with temporal structure, while the text is a discrete sequence of tokens with compositional semantics. The alignment problem asks us to learn a shared representation in which semantically matched audio–text pairs are close and unmatched pairs are far apart.

Definition 21 (CLAP: Contrastive Language–Audio Pretraining).

A CLAP model consists of:

  1. An audio encoder fθ:Ts×Fsd that maps a mel-spectrogram 𝐒 to a unit-normalised embedding 𝒆(a)=fθ(𝐒)/fθ(𝐒).

  2. A text encoder gϕ:𝒱d that maps a token sequence 𝒙(t) to a unit-normalised embedding 𝒆(t)=gϕ(𝒙(t))/gϕ(𝒙(t)).

  3. A learnable temperature parameter τ>0.

The two encoders are trained jointly so that matched pairs (𝒆i(a),𝒆i(t)) have high cosine similarity while unmatched pairs have low cosine similarity.

The cosine similarity between an audio embedding and a text embedding is (CLAP SIM)sij=1τ𝒆i(a),𝒆j(t)=1τfθ(𝐒i),gϕ(𝒙j(t))fθ(𝐒i)gϕ(𝒙j(t)), where the temperature τ controls the sharpness of the distribution over similarities. Within a mini-batch of B pairs, we form the B×B similarity matrix 𝐒simB×B with entries sij.

Definition 22 (Symmetric Contrastive Loss).

Let {(𝒆i(a),𝒆i(t))}i=1B be a mini-batch of B matched audio–text embedding pairs with similarity matrix entries sij as in (CLAP SIM). The CLAP contrastive loss is the symmetric InfoNCE objective: (CLAP LOSS)CLAP=12Bi=1B[logexp(sii)j=1Bexp(sij)+logexp(sii)j=1Bexp(sji)]. The first term treats each audio sample as a query and retrieves the correct text from among B candidates (audio-to-text direction). The second term reverses the roles (text-to-audio direction).

Expanding the two terms, we recognise them as the negative log-probability of the correct match under two softmax distributions: (CLAP A2T)at=1Bi=1Blogexp(sii)j=1Bexp(sij),ta=1Bi=1Blogexp(sii)j=1Bexp(sji). Both terms are instances of the categorical cross-entropy, where the target is the identity permutation: sample i should match sample i.

Proposition 12 (CLAP Loss as Mutual Information Bound).

The CLAP contrastive loss provides a lower bound on the mutual information between the audio and text embedding distributions: (CLAP MI Bound)I(𝒆(a);𝒆(t))logBat. Minimising at therefore maximises a lower bound on the mutual information, with the bound becoming tighter as the batch size B increases.

Proof.

This follows directly from the analysis of InfoNCE by Oord, Li, and Vinyals (2018). Define the critic h(𝒆i(a),𝒆j(t))=exp(sij) and note that the InfoNCE loss with B negative samples satisfies at=1Bi=1Blogh(𝒆i(a),𝒆i(t))1Bj=1Bh(𝒆i(a),𝒆j(t))+logB. The expectation of the first term is bounded below by I(𝒆(a);𝒆(t)), giving I(𝒆(a);𝒆(t))logBat. The bound is tight when the critic is optimal, h(𝒆(a),𝒆(t))p(𝒆(t)|𝒆(a))/p(𝒆(t)).

Remark 18.

The temperature τ plays a critical role in training dynamics. A small τ (e.g., τ=0.01) produces sharply peaked softmax distributions, concentrating gradient signal on the hardest negatives but risking training instability. A large τ (e.g., τ=1.0) smooths the distribution, slowing convergence but improving robustness to noisy labels. In practice, CLAP models initialise τ as a learnable scalar, typically starting at τ0=0.07, and clamp it to the range [τmin,τmax] during training to prevent collapse.

Proposition 13 (CLAP Score as Audio–Text Alignment Metric).

Given a trained CLAP model (fθ,gϕ) and a set of generated audio samples {𝒙^i(a)}i=1M with corresponding text prompts {𝒙i(t)}i=1M, the CLAP score is defined as (CLAP Score)CLAP-Score=1Mi=1M𝒆i(a),𝒆i(t), where 𝒆i(a) and 𝒆i(t) are the unit-normalised embeddings of the generated audio and the conditioning text, respectively. This metric lies in [1,1] and measures the average semantic alignment between generated audio and its textual description.

Proof.

Each inner product 𝒆i(a),𝒆i(t) equals the cosine similarity between unit vectors, which lies in [1,1]. The average of values in [1,1] also lies in [1,1]. The metric is maximised when every generated audio embedding perfectly aligns with its text embedding, corresponding to ideal semantic fidelity. It is zero when the embeddings are uncorrelated, and negative when they are systematically anti-correlated.

Audio and Text Encoders

The two arms of the CLAP architecture can be instantiated with a variety of backbone networks. The key design constraint is that both encoders must produce embeddings of the same dimension d, so that cosine similarity is well defined.

Audio encoder.

The audio encoder operates on mel-spectrogram inputs 𝐒Ts×Fs. Several backbone choices have been explored in the literature:

  1. CNN14 (from PANNs): A 14-layer convolutional network originally designed for audio tagging. It processes the mel-spectrogram through a sequence of convolutional blocks with batch normalisation and ReLU activations, followed by global average pooling to produce a fixed-dimensional vector 𝒉(a)da.

  2. HTS-AT (Hierarchical Token-Semantic Audio Transformer): A Swin-Transformer-based architecture that treats the mel-spectrogram as a 2D “image” and applies hierarchical windowed self-attention. The [CLS] token output serves as the audio representation 𝒉(a)da.

  3. HTSAT-BERT hybrid: A variant that combines convolutional feature extraction at early layers with Transformer-based global attention at later layers, seeking to capture both local spectral patterns and long-range temporal dependencies.

After the backbone produces 𝒉(a)da, a learned linear projection maps it to the shared embedding dimension: (CLAP Audio PROJ)𝒆(a)=𝐖a𝒉(a)+𝒃a𝐖a𝒉(a)+𝒃a,𝐖ad×da,𝒃ad. The 2 normalisation ensures that all audio embeddings lie on the unit hypersphere 𝕊d1.

Text encoder.

The text encoder processes a tokenised caption 𝒙(t)=(w1,w2,,wL) through a pre-trained language model. Common choices include:

  1. BERT (Bidirectional Encoder Representations from Transformers): The [CLS] token representation from the final Transformer layer is used as 𝒉(t)dt.

  2. RoBERTa: An optimised BERT variant with improved pre-training procedures. Like BERT, it produces a [CLS] representation that is projected to the shared space.

  3. GPT-2: For autoregressive text encoders, the embedding of the final token (or a special [EOS] token) serves as the sequence-level representation.

As with the audio side, a linear projection and normalisation map the text representation to the shared space: (CLAP TEXT PROJ)𝒆(t)=𝐖t𝒉(t)+𝒃t𝐖t𝒉(t)+𝒃t,𝐖td×dt,𝒃td.

Key Idea.

Shared space, independent encoders. The CLAP framework places no constraint on the internal architecture of the audio and text encoders; they can use entirely different network families (CNNs for audio, Transformers for text). The only coupling occurs through the contrastive loss, which acts on the final normalised embeddings. This modular design means that either encoder can be replaced or upgraded independently, provided the projection dimensions remain compatible.

The CLAP dual-encoder architecture. An audio encoder (left) and a text encoder (right) independently map their inputs to a shared d-dimensional embedding space. The contrastive loss maximises cosine similarity along the diagonal of the B×B similarity matrix (shaded cells) while minimising off-diagonal entries.

Training Data and Applications

The quality of a CLAP model depends critically on the diversity and scale of its training data. Unlike the image domain, where billions of image–text pairs can be scraped from the web, audio–text pairs are substantially scarcer.

Datasets.

The principal audio-captioning datasets used for CLAP training include:

DatasetClipsCaptions/ClipSource
AudioCaps50k1AudioSet subset, human-written
Clotho6k5Freesound, crowd-sourced
WavCaps400k1ChatGPT-processed web audio
LAION-Audio630k1Web-crawled with filtering
tablePrincipal audio-captioning datasets used for CLAP training. WavCaps and LAION-Audio provide scale at the cost of noisier annotations.

The WavCaps dataset represents an interesting strategy: rather than paying human annotators, the creators used a large language model (ChatGPT) to generate captions from audio metadata and weak labels. This approach trades annotation precision for scale, a trade-off that has proven effective for contrastive pre-training where the loss is robust to moderate label noise.

CLAP for conditioning.

In text-to-audio generation systems, CLAP embeddings serve as the conditioning signal that steers the generative process. Given a text prompt 𝒙(t) at inference time, the pre-trained text encoder produces 𝒆(t)=gϕ(𝒙(t))/gϕ(𝒙(t)), which is then injected into the generative model via cross-attention or FiLM layers (see Conditioning Mechanisms for Audio Generation). This approach decouples the text understanding from the audio generation: the CLAP encoder “understands” the text, and the generative model “creates” the audio.

CLAP for evaluation.

As formalised in Proposition 13, the CLAP score measures semantic alignment between generated audio and conditioning text. It has become a standard evaluation metric alongside the Fréchet Audio Distance (FAD) and the Kullback–Leibler divergence over audio event labels. While FAD measures distributional fidelity (how realistic the generated audio sounds), the CLAP score measures prompt adherence (how well the generated audio matches the textual description).

Example 10 (CLAP Score in Practice).

Consider evaluating a text-to-audio model on the AudioCaps test set. For each of the M=975 test prompts, the model generates one audio sample. We compute CLAP-Score=1975i=1975𝒆i(a),𝒆i(t). Typical scores for state-of-the-art models (e.g., AudioLDM2, Tango, Make-An-Audio) range from 0.40 to 0.55. As a reference, computing the CLAP score between the ground-truth audio and its caption yields scores around 0.55 to 0.60, reflecting the intrinsic noise in human-written captions that do not capture every acoustic detail.

Historical Note.

The CLAP framework was introduced independently by Wu et al. (2023) and Elizalde et al. (2023), both drawing explicit inspiration from CLIP\@. The name “CLAP” was chosen as a deliberate phonetic echo of “CLIP,” reinforcing the conceptual parallel. Within months of its release, CLAP embeddings became the de facto conditioning signal for nearly every text-to-audio diffusion model, mirroring how CLIP embeddings had become ubiquitous in text-to-image generation.

Autoregressive Music Generation

Diffusion models have dominated recent text-to-audio generation, but an equally powerful paradigm treats audio generation as a language modelling problem. The idea is deceptively simple: tokenise the audio signal into a sequence of discrete codes using a neural codec (see Neural Audio Codecs), then train a Transformer to predict these codes autoregressively, one token at a time. The approach inherits the full machinery of large language models: causal attention, nucleus sampling, KV-cache acceleration, and instruction-following via cross-attention to text encoders.

This section traces the evolution from early approaches such as Jukebox to the modern MusicGen framework, focusing on the mathematical challenge of modelling multi-codebook sequences and the computational trade-offs inherent in autoregressive audio generation.

From Jukebox to Modern Autoregressive Models

The Jukebox model (Dhariwal et al., 2020) was a landmark demonstration that autoregressive Transformers could generate minutes of coherent music with vocals, instruments, and stylistic variation. As discussed in sec:vae2:case-jukebox, Jukebox used a hierarchical VQ-VAE to compress raw audio waveforms into discrete tokens at multiple temporal resolutions, then trained separate autoregressive Transformers at each level of the hierarchy.

While Jukebox proved the concept, it suffered from severe practical limitations. Generating a single minute of music required hours of computation on high-end GPUs, because the VQ-VAE tokens operated at relatively high temporal rates (e.g., one token per 8 waveform samples at the finest level), producing extremely long sequences. The hierarchical generation scheme also introduced error accumulation: artefacts at coarse levels propagated and amplified through finer levels.

The shift to neural codec tokens (from EnCodec, SoundStream, and related models) resolved much of this inefficiency. Neural codecs compress audio far more aggressively than the VQ-VAE approach used by Jukebox, producing token rates of 5075 tokens per second at each codebook level rather than thousands of tokens per second. However, neural codecs introduce a new challenge: the residual vector quantisation (RVQ) scheme produces multiple parallel codebooks, and the autoregressive model must decide how to serialise these multi-codebook tokens into a single sequence.

Key Idea.

Flatten the codebook. A neural codec with Q codebook levels and temporal rate T produces a matrix of tokens 𝐂{1,,K}T×Q, where ct(q) is the code at time step t and codebook level q. An autoregressive Transformer processes a one-dimensional sequence. The central design decision is how to “flatten” the 2D token matrix into a 1D sequence that the Transformer can model effectively.

The Codebook Flattening Problem

Let 𝐂{1,,K}T×Q denote the token matrix produced by a neural codec with T time steps and Q codebook levels. The entry ct(q) is the discrete code assigned to time step t at codebook level q. The autoregressive model must define a total ordering on these T×Q tokens. Three principal patterns have emerged, each with distinct trade-offs in sequence length, modelling fidelity, and generation speed.

Definition 23 (Interleaved Pattern).

The interleaved pattern serialises the token matrix column-by-column within each time step: (Interleaved)𝒔interleaved=(c1(1),c1(2),,c1(Q),c2(1),c2(2),,c2(Q),,cT(1),cT(2),,cT(Q)). The resulting sequence has length TQ. The autoregressive model predicts each code conditioned on all previous codes across all time steps and codebook levels.

The interleaved pattern is the most faithful to the true joint distribution: it makes no conditional independence assumptions, since each token is conditioned on the complete history. However, the Q-fold increase in sequence length directly translates to Q-fold slower generation and Q-fold higher memory consumption for the KV-cache.

Definition 24 (Delayed Pattern).

The delayed pattern introduces a stagger: codebook level q is shifted forward by q1 time steps relative to the first codebook. At each autoregressive step t, the model predicts the tokens (Delayed)(ct(1),ct1(2),ct2(3),,ctQ+1(Q)), where tokens with non-positive time indices are masked. The effective sequence length is T+Q1, and at each step, the model predicts Q tokens in parallel (one per codebook).

The delayed pattern exploits the observation that the coarser codebook levels (q=1) carry the most information about the audio structure, while finer levels (q>1) refine acoustic details that depend primarily on the coarse codes at nearby time steps. By delaying finer codebooks, the model ensures that when predicting ct(q), it has already seen ct(1),,ct(q1) from the same time step (which were predicted at step t) as well as all codes from earlier time steps.

Definition 25 (Parallel Pattern).

The parallel pattern predicts all Q codes simultaneously at each time step: (Parallel)p(𝐂)=t=1Tp(ct(1),ct(2),,ct(Q)|c<t(1:Q)). The effective sequence length is T. The model uses Q independent prediction heads, one per codebook level, and samples all Q codes in a single forward pass per time step.

The parallel pattern is the fastest at generation time but makes a strong conditional independence assumption: given the past, the Q codes at time t are predicted independently. This ignores the hierarchical dependency structure of RVQ, where finer codes depend on the residual left by coarser codes.

Proposition 14 (Sequence Length Comparison).

For a neural codec with T time steps and Q codebook levels, the effective autoregressive sequence lengths under the three patterns are:

PatternSeq. LengthTokens/StepInter-CB Dep.
InterleavedTQ1Full
DelayedT+Q1QPartial (delayed)
ParallelTQNone (independent)
In particular, for typical values T=1500 (30 seconds at 50 Hz) and Q=8, the interleaved sequence has length 12,000, the delayed sequence has length 1,507, and the parallel sequence has length 1,500.

Proof.

The interleaved pattern places all TQ tokens in a single sequence with one token per step, giving length TQ. The delayed pattern aligns codebook q with a delay of q1 steps; the last codebook (q=Q) requires Q1 additional steps beyond the T steps needed for q=1, giving a total of T+Q1 steps. At each step, the model predicts one token per codebook (those that are “active” at that step), for a maximum of Q tokens per step. The parallel pattern collapses all codebooks into a single step per time index, giving length T with Q tokens per step.

Three codebook flattening patterns for autoregressive audio generation, shown for T=3 time steps and Q=4 codebook levels. (a) Interleaved: all tokens are serialised into a single sequence of length TQ=12. (b) Delayed: codebook q is shifted by q1 steps, yielding length T+Q1=6 with up to Q tokens predicted per step. (c) Parallel: all Q codes per time step are predicted simultaneously, yielding length T=3. Numbers in (a) indicate the autoregressive ordering.

MusicGen: Architecture and Training

MusicGen (Copet et al., 2023) is the most prominent example of a modern autoregressive music generation system built on neural codec tokens. It uses EnCodec as its audio tokeniser and a decoder-only Transformer with the delayed pattern as its core generation engine.

Token representation.

MusicGen operates on EnCodec tokens at 32 kHz with Q=4 codebook levels and a frame rate of 50 Hz (one frame per 20 ms). Each codebook has K=2048 entries. A 30-second music clip is thus represented as a token matrix 𝐂{1,,2048}1500×4.

Under the delayed pattern (Definition 24), the autoregressive sequence has length T+Q1=1503. At each step t, the model predicts four tokens (one per codebook level) in parallel, with codebook q predicting the token for time step tq+1.

Transformer architecture.

The MusicGen Transformer is a standard decoder-only architecture with causal self-attention, as studied in ch:autoreg. The model uses L=48 layers, hidden dimension d=2048, 32 attention heads, and a context length of 1503 tokens. At each position, the input to the Transformer is the sum of Q codebook embeddings: (Musicgen Input)𝒉t=q=1Q𝐄(q)𝒆ctq+1(q)+𝒑t, where 𝐄(q)d×K is the embedding matrix for codebook q, 𝒆cK is a one-hot vector for code c, and 𝒑td is the positional embedding for step t.

Text conditioning.

MusicGen uses a pre-trained T5 encoder to convert text descriptions into a sequence of conditioning embeddings. These are injected into the Transformer via cross-attention at every layer. Let 𝐇(l)Tseq×d denote the hidden states at layer l and 𝐂textTtext×d the T5 embeddings projected to dimension d. The cross-attention operation at each layer is (Musicgen Crossattn)CrossAttn(𝐇(l),𝐂text)=softmax(𝐐𝐊dh)𝐕, where 𝐐=𝐇(l)𝐖Q, 𝐊=𝐂text𝐖K, 𝐕=𝐂text𝐖V, and dh=d/H is the per-head dimension. This follows the standard cross-attention mechanism studied in sec:vdiff:conditioning.

Melody conditioning via chromagram.

A distinctive feature of MusicGen is its ability to condition on a reference melody. The melody is represented as a chromagram, which encodes the distribution of musical pitch classes over time.

Definition 26 (Chromagram).

Given an audio signal x(n) with short-time Fourier transform STFT(x)[m,k], the chromagram is a matrix 𝐂chroma0M×12 defined by (Chromagram)C[m,p]=k:pitch(k)=p|STFT(x)[m,k]|2, where m indexes the time frame, p{0,1,,11} indexes the pitch class (C, C, D, , B), and pitch(k)=12log2(fk/fref)mod12 maps frequency bin k (with centre frequency fk) to its pitch class relative to a reference frequency fref (typically fref=440 Hz for A4).

The chromagram captures harmonic content while being invariant to timbre, dynamics, and instrumentation. Two performances of the same melody on different instruments will produce similar chromagrams. MusicGen quantises the chromagram into a discrete token sequence using a separate codebook and provides it to the Transformer as an additional conditioning signal alongside the text embeddings.

Algorithm 3 (MusicGen Generation with Delayed Pattern).

Input: Text prompt 𝒙(t), optional melody audio 𝒙(mel), number of time steps T, temperature τgen, codebook levels Q. Output: Generated audio waveform 𝒙^.

  1. Encode conditioning: enumerate

  2. Compute text embeddings: 𝐂text=T5-Encoder(𝒙(t)).

  3. If melody provided, compute chromagram: 𝐂chroma=Chromagram(𝒙(mel)) and quantise to melody tokens. enumerate

  4. Initialise: Set 𝐂= (empty token matrix). Initialise KV-cache.

  5. Autoregressive loop: For t=1,2,,T+Q1: enumerate

  6. Compute input embedding 𝒉t from active tokens using (Musicgen Input).

  7. Run Transformer forward pass with cross-attention to 𝐂text (and melody tokens if present).

  8. For each codebook q=1,,Q where tq+11 and tq+1T: enumerate

  9. Compute logits 𝒍(q)=𝐖out(q)𝒉t(L).

  10. Sample ctq+1(q)Categorical(softmax(𝒍(q)/τgen)). enumerate

  11. Append sampled tokens to 𝐂 and update KV-cache. enumerate

  12. Decode: Convert token matrix 𝐂 to waveform using the EnCodec decoder: 𝒙^=EnCodec-Decode(𝐂).

Remark 19.

MusicGen applies classifier-free guidance (CFG) during generation. During training, the text conditioning is dropped with probability pdrop=0.1, replacing the T5 embeddings with a learned null embedding. At inference, the logits are interpolated: 𝒍guided(q)=(1+w)𝒍cond(q)w𝒍uncond(q), where w0 is the guidance scale (typically w=3.0). This doubles the cost per step (two forward passes), but substantially improves text adherence. See Classifier-Free Guidance for Audio for a detailed treatment of CFG for audio.

Example 11 (MusicGen Token Rates).

Consider generating 30 seconds of music at 32 kHz. The EnCodec tokeniser operates at 50 Hz with Q=4 codebook levels, producing T=1500 time frames. The delayed pattern requires T+Q1=1503 autoregressive steps. With a Transformer that processes each step in approximately 15 ms (including KV-cache update and two forward passes for CFG), the total generation time is 1503×0.01522.5 seconds. This achieves roughly real-time generation on a modern GPU, a dramatic improvement over Jukebox's hours-per-minute rate.

For comparison, the interleaved pattern would require T×Q=6000 steps, taking approximately 90 seconds and consuming 4× more KV-cache memory.

The Autoregressive Bottleneck

Despite the improvements brought by neural codecs and the delayed pattern, autoregressive audio generation faces a fundamental computational bottleneck: generation is inherently sequential. Each token depends on all previously generated tokens, so the model must be evaluated once per time step, with no opportunity for temporal parallelism during inference.

Sequential generation speed.

For a sequence of length L, autoregressive generation requires L forward passes through the Transformer. Each forward pass involves computing attention over the growing KV-cache, with cost that scales as O(Ld2+LLcached) per step, where d is the model dimension and Lcache is the current cache length. The total cost over all steps is therefore O(L2d2), which is quadratic in the sequence length.

Proposition 15 (Autoregressive Generation Cost).

Let L be the autoregressive sequence length, d the model dimension, H the number of attention heads, NL the number of Transformer layers, and Bgen the batch size during generation. The total floating-point operations for generating one complete sequence (using KV-cache) is (AR COST)FLOPsAR=NLL(8d2QKV + output proj.+2dLattention+16d2FFN), where L=L/2 is the average cache length across all steps (since the cache grows from 1 to L).

Proof.

At step t, the KV-cache has t1 entries. The query-key dot product costs 2d(t1) FLOPs per layer (for H heads, each of dimension d/H, the total is still 2d(t1)). The QKV projections and output projection each cost 2d2 FLOPs per layer (four projections total: 8d2). The FFN with hidden dimension 4d costs 2×2×d×4d=16d2 FLOPs. Summing over L steps and NL layers, with the attention term averaging to 2dL/2, gives the stated expression.

KV-cache memory.

The KV-cache stores the key and value tensors for all previous positions, consuming (KV Memory)MemoryKV=2NLLdbytes-per-element bytes. For MusicGen's configuration (NL=48, d=2048, L=1503, float16), this is 2×48×1503×2048×2564 MB per batch element. Generating long audio clips (e.g., 5 minutes) would require L15,000, pushing KV-cache memory to 5.6 GB per sample. For a treatment of memory-efficient attention mechanisms that can mitigate this cost, see ch:memory.

Insight.

The speed–fidelity–length trilemma. Autoregressive audio models face a three-way trade-off: (i) faster generation (parallel or delayed patterns) sacrifices inter-codebook modelling fidelity; (ii) higher fidelity (interleaved pattern) requires longer sequences and slower generation; (iii) longer audio clips demand proportionally more computation and memory, regardless of the pattern choice. Diffusion-based approaches (Spectrogram-Domain Generation) offer an alternative that parallelises across the time dimension, but introduce their own trade-offs in iterative denoising cost.

Conditioning Mechanisms for Audio Generation

A generative model that produces audio unconditionally is of limited practical use: we nearly always want to control what is generated. The conditioning signal might be a text description (“a jazz piano solo with brushed drums”), a melody hummed by the user, a chord progression, song lyrics, a reference audio clip whose style should be transferred, or even a video whose soundtrack must be synchronised.

This section catalogues the principal conditioning modalities for audio generation, then analyses the mathematical mechanisms through which conditioning information is injected into the generative model. Many of these mechanisms are shared with image and video generation (sec:vdiff:conditioning); here we focus on the adaptations and considerations specific to the audio domain.

The Conditioning Landscape

tab:audiogen:conditioning-modalities summarises the principal conditioning modalities used in modern audio generation systems.

lllX ModalityRepresentationControl LevelExample Systems
Text descriptionCLAP / T5 embeddingsSemanticMusicGen, AudioLDM2
MelodyChromagram tokensPitch contourMusicGen (melody mode)
ChordsChord sequence embeddingsHarmonic structureSingSong, Jukebox
LyricsPhoneme / text alignmentVocal contentJukebox, VoiceCraft
Audio referenceCodec / CLAP embeddingStyle / timbreMusicLM, AudioLDM
VideoCLIP / temporal featuresTemporal syncSpecVQGAN, Diff-Foley
MIDIDiscrete note eventsNote-level controlMIDI-DDSP
Conditioning modalities for audio generation. Each modality provides a different type of control over the generated output, from high-level semantic guidance (text) to fine-grained structural constraints (MIDI, lyrics alignment).

The richness of this conditioning landscape is a distinguishing feature of audio generation compared to image generation, where text is the dominant conditioning modality. In audio, the temporal structure of the output creates opportunities for temporally aligned conditioning signals (melody, chords, lyrics, video) that have no direct analogue in the image domain.

Cross-Attention Conditioning

Cross-attention is the most common mechanism for injecting conditioning information into Transformer-based audio generators. The mechanism is identical in form to the cross-attention used in image and video diffusion models (sec:vdiff:conditioning), but the conditioning and generation modalities differ.

Definition 27 (Cross-Attention Conditioning for Audio).

Let 𝐇Tseq×d denote the hidden states of the audio generation model (either an autoregressive Transformer or a diffusion model's denoiser) and let 𝐂condTc×dc denote the conditioning embeddings (e.g., from a T5 or CLAP encoder), projected to dimension d by a linear map 𝐖Pd×dc. The cross-attention output is (Crossattn)𝐐=𝐇𝐖Q,𝐊=(𝐂cond𝐖P)𝐖K,𝐕=(𝐂cond𝐖P)𝐖V,CrossAttn(𝐇,𝐂cond)=softmax(𝐐𝐊dh)𝐕, where 𝐖Q,𝐖K,𝐖Vd×dh are the query, key, and value projection matrices for a single attention head, and dh=d/H is the per-head dimension. The multi-head version concatenates H such heads and applies an output projection.

The attention weights 𝐀=softmax(𝐐𝐊/dh)Tseq×Tc form a soft alignment matrix between the audio generation positions and the conditioning positions. Each row of 𝐀 is a probability distribution over the conditioning tokens, determining how much each conditioning token influences the corresponding audio generation position.

Remark 20.

In text-to-audio generation, the cross-attention weights provide an interpretable window into how the model uses the text prompt. Visualising 𝐀 often reveals that the model learns to attend to relevant words at appropriate times: for example, when generating the drum section of a prompt like “jazz piano followed by drum solo,” the cross-attention weights shift from the word “piano” to the word “drum” at the transition point. This emergent temporal alignment arises without any explicit alignment supervision.

Proposition 16 (Cross-Attention as Conditional Expectation).

The cross-attention output at position t can be written as a conditional expectation under a learned distribution: (Crossattn Expectation)CrossAttn(𝒉t,𝐂cond)=j=1Tcαtj𝒗j=𝔼jCat(𝜶t)[𝒗j], where αtj=[softmax(𝐐t𝐊/dh)]j and 𝒗j=(𝐂cond𝐖P𝐖V)j. The output is therefore the expected value of the conditioning representation under a soft retrieval distribution, where the query at position t determines which conditioning tokens are most relevant.

Proof.

The result follows immediately from the definition of softmax attention. The attention weights αtj0 and jαtj=1, so they define a categorical distribution over conditioning positions {1,,Tc}. The weighted sum jαtj𝒗j is the expectation of the random variable 𝒗J where JCat(𝜶t).

FiLM Conditioning

Feature-wise Linear Modulation (FiLM) provides a computationally efficient alternative to cross-attention for injecting conditioning information. Rather than attending to a sequence of conditioning tokens, FiLM conditions by applying a learned affine transformation to the hidden features.

Definition 28 (FiLM: Feature-wise Linear Modulation).

Let 𝒉d be a hidden feature vector at any position in the generation model, and let 𝒄dc be a global conditioning vector (e.g., a CLAP embedding or a time-step embedding). The FiLM transformation is (FILM)FiLM(𝒉;𝜸,𝜷)=𝜸𝒉+𝜷, where denotes element-wise multiplication, and the scale and shift parameters are generated from the conditioning vector: (FILM Params)𝜸=𝐖γ𝒄+𝒃γ,𝜷=𝐖β𝒄+𝒃β, with 𝐖γ,𝐖βd×dc and 𝒃γ,𝒃βd being learnable parameters.

The FiLM mechanism is closely related to the Adaptive Layer Normalisation (AdaLN) used in diffusion Transformers. In AdaLN, the affine parameters of layer normalisation are conditioned on the time step and class label; FiLM generalises this to arbitrary conditioning signals.

Proposition 17 (FiLM as Conditional Affine Transform).

The FiLM transformation FiLM(𝒉;𝜸,𝜷) is a conditional affine transformation of 𝒉, parameterised by the conditioning vector 𝒄. Specifically, it can be written as (FILM Affine)FiLM(𝒉;𝒄)=diag(𝐖γ𝒄+𝒃γ)𝒉+𝐖β𝒄+𝒃β=𝐀(𝒄)𝒉+𝒃(𝒄), where 𝐀(𝒄)=diag(𝐖γ𝒄+𝒃γ)d×d is a diagonal matrix and 𝒃(𝒄)=𝐖β𝒄+𝒃βd is a bias vector, both dependent on 𝒄. The transformation is linear in 𝒉 for fixed 𝒄, but nonlinear in 𝒄 when composed with subsequent nonlinear layers.

Proof.

Expanding the element-wise product: [𝜸𝒉]i=γihi=[diag(𝜸)]iihi. Therefore 𝜸𝒉=diag(𝜸)𝒉, and FiLM(𝒉;𝜸,𝜷)=diag(𝜸)𝒉+𝜷. Substituting 𝜸=𝐖γ𝒄+𝒃γ gives 𝐀(𝒄)=diag(𝐖γ𝒄+𝒃γ), and similarly for 𝒃(𝒄).

Remark 21.

FiLM and cross-attention represent two extremes of the complexity–expressivity spectrum for conditioning:

  1. FiLM uses a global conditioning vector 𝒄 that applies the same affine transformation at every position. It cannot perform position-dependent conditioning (e.g., different instruments at different times) unless the conditioning vector itself encodes temporal information.

  2. Cross-attention uses a sequence of conditioning tokens 𝐂cond and learns position-dependent alignment via attention weights. It is strictly more expressive but requires O(TseqTc) computation.

In practice, many audio generation architectures use both: FiLM for global conditioning (e.g., diffusion time step, global style) and cross-attention for sequential conditioning (e.g., text tokens, melody).

Classifier-Free Guidance for Audio

Classifier-free guidance (CFG) was originally developed for class-conditional image generation and later extended to text-to-image and text-to-video diffusion models (see sec:vdiff:cfg-video). Its application to audio generation follows the same mathematical framework, but the practical considerations differ in important ways.

Review of CFG.

During training, the conditioning signal 𝒄 is dropped with probability pdrop (typically 0.1 to 0.2), replaced by a null conditioning token 𝒄. At inference, the model's output (score function, velocity field, or logits, depending on the generative framework) is extrapolated along the conditioning direction: (CFG)𝝐~(𝒙t,𝒄)=(1+w)𝝐θ(𝒙t,𝒄)w𝝐θ(𝒙t,𝒄), where w0 is the guidance scale. Setting w=0 recovers unconditional generation; increasing w amplifies the influence of the conditioning signal at the cost of reduced sample diversity.

Audio-specific guidance scales.

A notable empirical observation is that optimal CFG scales for audio generation are substantially lower than those used in image generation. While text-to-image models often use w[7.5,15], text-to-audio models typically perform best with w[2.5,5.0]. This difference likely reflects the nature of the conditioning: audio captions describe broad categories of sounds (“birds singing near a stream”), leaving substantial room for perceptually valid variation, whereas image prompts often specify detailed spatial arrangements (“a red car parked on a snowy street”) that demand stronger guidance.

Definition 29 (Audio CFG with Scale Schedule).

For diffusion-based audio generation with Tdiff denoising steps, a guidance scale schedule w:{1,,Tdiff}0 specifies a potentially time-varying guidance scale: (CFG Schedule)𝝐~(𝒙t,𝒄,t)=(1+w(t))𝝐θ(𝒙t,𝒄,t)w(t)𝝐θ(𝒙t,𝒄,t). A common schedule uses higher guidance at early (noisier) denoising steps and lower guidance at later (less noisy) steps: w(t)=wmax(t/Tdiff)p for some power p>0.

Caution.

Over-guidance distortion in audio. When the guidance scale w is too high, audio generation models produce characteristic distortion artefacts that differ from the over-saturation seen in images. In audio, excessive guidance manifests as: (i) harmonic distortion, where pure tones acquire unnatural overtones; (ii) spectral peaking, where energy concentrates in narrow frequency bands, producing a “metallic” or “synthetic” quality; (iii) temporal repetition, where the model becomes trapped in repetitive patterns, sacrificing temporal variation for prompt adherence. These artefacts are particularly severe for music generation, where the listener has strong expectations about spectral smoothness and temporal dynamics. As a rule of thumb, if the generated audio sounds “overcooked” or unnatural, reducing the guidance scale by 12 points is the first intervention to try.

Proposition 18 (CFG as Posterior Sharpening).

Under the score-based interpretation, the guided score ~𝒙logp(𝒙t|𝒄) corresponds to sampling from a distribution proportional to (CFG Sharpened)p~(𝒙t|𝒄)p(𝒙t)(p(𝒄|𝒙t)p(𝒄))1+w, which is the unconditional distribution reweighted by the (1+w)-th power of the likelihood ratio. When w>0, this sharpens the conditional distribution, concentrating mass on regions of 𝒙-space where the conditioning signal 𝒄 is most probable.

Proof.

The unconditional and conditional scores are 𝒙logp(𝒙t) and 𝒙logp(𝒙t|𝒄), respectively. By Bayes' rule, 𝒙logp(𝒙t|𝒄)=𝒙logp(𝒙t)+𝒙logp(𝒄|𝒙t). The CFG score is ~𝒙logp(𝒙t|𝒄)=(1+w)𝒙logp(𝒙t|𝒄)w𝒙logp(𝒙t)=𝒙logp(𝒙t)+(1+w)𝒙logp(𝒄|𝒙t). This is the score of the distribution p~(𝒙t|𝒄)p(𝒙t)p(𝒄|𝒙t)1+w, which can be rewritten as p(𝒙t)(p(𝒄|𝒙t)/p(𝒄))1+wp(𝒄)1+w. Since p(𝒄)1+w is a constant with respect to 𝒙t, the normalised distribution is p~(𝒙t|𝒄)p(𝒙t)(p(𝒄|𝒙t)/p(𝒄))1+w.

Multi-Modal Conditioning

Real-world audio generation tasks often require simultaneous conditioning on multiple modalities. A user might specify a text description (“upbeat pop song”), provide a melody (hummed reference), and indicate a desired style (audio of a reference track). The generation model must fuse these heterogeneous signals into a coherent conditioning representation.

Conditioning fusion strategies.

Three principal strategies exist for combining multiple conditioning modalities:

  1. Concatenation: The conditioning embeddings from different modalities are concatenated along the sequence dimension before being provided to cross-attention. If the text embeddings are 𝐂textTt×d and the melody embeddings are 𝐂melTm×d, the combined conditioning is (Concat COND)𝐂combined=[𝐂text;𝐂mel](Tt+Tm)×d. The attention mechanism then learns to attend to the appropriate modality at each generation position.

  2. Separate cross-attention: Each conditioning modality is injected through its own cross-attention layer, and the outputs are summed: (Separate Crossattn)𝒉tcond=CrossAttntext(𝒉t,𝐂text)+CrossAttnmel(𝒉t,𝐂mel). This allows independent attention patterns for each modality but increases the parameter count and computational cost.

  3. Hierarchical conditioning: Global conditioning signals (style, genre) are injected via FiLM, while sequential conditioning signals (text, melody) are injected via cross-attention. This leverages the strengths of each mechanism.

Multi-modal conditioning architecture for audio generation. Text and melody conditioning, which carry temporal structure, are injected via cross-attention. Style conditioning, which is a global property, is injected via FiLM\@. The generation model fuses all conditioning signals to produce neural codec tokens, which are decoded to a waveform.

Example 12 (Multi-Modal Conditioning in MusicGen).

MusicGen supports joint text and melody conditioning. The text description is encoded by a T5 model and provided via cross-attention. The melody is represented as a quantised chromagram and provided as an additional conditioning sequence. At each Transformer layer, the model attends to both conditioning sequences: 𝒉(l+1/2)=𝒉(l)+CrossAttntext(𝒉(l),𝐂text)+CrossAttnmel(𝒉(l),𝐂mel),𝒉(l+1)=𝒉(l+1/2)+FFN(𝒉(l+1/2)). In practice, the melody conditioning is found to strongly constrain the pitch contour of the generated music, while the text conditioning influences instrumentation, genre, and mood. The model learns to balance these signals without explicit weighting.

Lemma 2 (Conditional Independence under Separate Cross-Attention).

The separate cross-attention approach in (Separate Crossattn) implicitly assumes that the conditioning modalities contribute additively to the hidden representation. Formally, let Δ𝒉t(text)=CrossAttntext(𝒉t,𝐂text) and Δ𝒉t(mel)=CrossAttnmel(𝒉t,𝐂mel). Then (Additive COND)𝒉tcond=Δ𝒉t(text)+Δ𝒉t(mel) neglects any interaction terms between the two modalities. An interaction-aware formulation would require joint attention over the concatenated conditioning sequence, as in (Concat COND).

Proof.

By construction, the separate cross-attention layers compute their outputs independently: Δ𝒉t(text) depends on 𝐂text but not on 𝐂mel, and vice versa. The sum Δ𝒉t(text)+Δ𝒉t(mel) is therefore a first-order approximation to the joint conditioning effect. In contrast, the concatenation approach allows the attention weights over text tokens to depend on the melody tokens (through the shared softmax normalisation), capturing cross-modal interactions.

Spectrogram-Domain Generation

Before neural codecs and latent diffusion models became the dominant paradigm, an influential line of work explored generating audio by treating mel spectrograms as images and applying image generation techniques directly. This approach, most famously demonstrated by Riffusion, is conceptually elegant: it repurposes the extensive machinery developed for image synthesis (U-Net denoisers, latent diffusion, classifier-free guidance) without any audio-specific architectural modifications.

This section analyses the mathematics of spectrogram-domain generation, its surprising effectiveness, and the fundamental limitations that ultimately motivated the shift to latent codec spaces.

Riffusion: Image Diffusion Meets Audio

Riffusion (Forsgren and Martiros, 2022) was a proof-of-concept demonstration that fine-tuning Stable Diffusion on mel spectrogram images could produce musically coherent audio. The approach consists of three stages:

  1. Data preparation. Each audio clip is converted to a mel spectrogram 𝐒Ts×Fs, which is then rescaled and formatted as a single-channel (or three-channel, with the spectrogram duplicated) image of fixed resolution (e.g., 512×512).

  2. Fine-tuning. The pre-trained Stable Diffusion model, including its latent autoencoder and U-Net denoiser, is fine-tuned on the spectrogram “images” with corresponding text captions. The VAE encoder learns to compress spectrograms into the same latent space, and the U-Net learns to denoise spectrograms rather than natural images.

  3. Inversion. The generated spectrogram is converted back to a waveform using a spectrogram inversion algorithm (e.g., Griffin–Lim or a neural vocoder such as HiFi-GAN).

Insight.

Reusing image priors for audio. The remarkable finding of Riffusion is that the visual priors learned by a model trained on natural images (smooth textures, coherent structures, sharp edges) transfer meaningfully to mel spectrograms. Harmonic sounds appear as horizontal lines in the spectrogram (analogous to edges in images), percussive sounds appear as vertical lines (analogous to vertical edges), and gradual timbral changes appear as smooth texture variations. The pre-trained image model already “knows” how to generate coherent visual structures; it merely needs to learn the specific visual vocabulary of spectrograms.

Formally, the Riffusion generation process can be described as follows. Let img and 𝒟img denote the pre-trained (and fine-tuned) Stable Diffusion VAE encoder and decoder, and let 𝝐θ be the fine-tuned U-Net denoiser. Given a text prompt 𝒄, the generation pipeline is: (Riffusion Sample)𝒛TNormal(0,𝐈),𝒛0=DDPM-Sample(𝒛T,𝝐θ,𝒄),𝐒^=𝒟img(𝒛0),𝒙^=Vocoder(𝐒^), where 𝐒^Ts×Fs is the generated mel spectrogram and 𝒙^L is the reconstructed waveform.

Example 13 (Riffusion Spectrogram Generation).

Consider generating a 5-second audio clip at 44.1 kHz with a hop size of 512 samples and 128 mel frequency bins. The mel spectrogram has dimensions Ts=5×44100/512=431 time frames and Fs=128 frequency bins. After zero-padding to 512×512 (to match Stable Diffusion's expected input resolution), the spectrogram is treated as a single-channel image.

The Stable Diffusion VAE compresses this to a latent tensor of shape 64×64×4 (with 8× spatial downsampling), and the U-Net performs denoising in this latent space. The compression ratio is 512×512/(64×64×4)16, identical to the ratio for natural images.

Proposition 19 (Spectrogram Image Diffusion Equivalence).

Let 𝝐θ(img) be a U-Net denoiser trained on natural images 𝒙H×W×3 and let 𝝐θ(spec) be the same architecture fine-tuned on mel spectrograms 𝐒H×W×1 (with channel replication to match the 3-channel input). The fine-tuned model minimises the same denoising objective: (SPEC Image Equiv)spec-img=𝔼t,𝐒0,𝝐[𝝐𝝐θ(spec)(αt𝐒0+1αt𝝐,t)2]. The only difference from the image training objective is the data distribution: spectrograms replace natural images. All mathematical properties of DDPM (convergence guarantees, score matching equivalence, the connection to SDEs) transfer without modification.

Proof.

The DDPM framework is defined for arbitrary data distributions pdata on n and does not depend on the data being images. The forward process q(𝐒t|𝐒0)=Normal(αt𝐒0,(1αt)𝐈) is well defined for any 𝐒0H×W×C. The denoising score matching objective 𝔼[𝝐𝝐θ(𝐒t,t)2] is a proper scoring rule for estimating the score 𝐒tlogq(𝐒t) regardless of the underlying data distribution. Therefore, the mathematical framework applies verbatim, and the only effect of changing the data domain is on the learned function 𝝐θ.

Spectrogram-to-waveform inversion.

The critical step in the Riffusion pipeline is converting the generated mel spectrogram back to a waveform. This inversion is inherently ill-posed: the mel spectrogram discards phase information and compresses the linear frequency axis, so multiple waveforms can produce the same mel spectrogram.

The classical approach is the Griffin–Lim algorithm, which iteratively reconstructs the phase by alternating between the time-domain and frequency-domain constraints. Starting from a random phase estimate ϕ(0), the algorithm iterates: (Griffin LIM)X(k)[m,n]=S[m,n]eiϕ(k)[m,n],x(k)(n)=Overlap-Add(IFFT(X(k))),ϕ(k+1)[m,n]=STFT(x(k))[m,n], where S[m,n] is the magnitude spectrogram (obtained by inverting the mel filterbank) and denotes the phase angle. After K iterations (typically K=32 to 64), the reconstructed waveform x(K) approximately satisfies the magnitude constraints.

Modern systems replace Griffin–Lim with neural vocoders (HiFi-GAN, BigVGAN, Vocos) that directly map mel spectrograms to waveforms using a generative neural network. These produce substantially higher-quality audio, as they learn to predict plausible phase patterns from the spectrogram structure.

Mathematics of Spectrogram Diffusion

When we apply diffusion models to mel spectrograms, the data space is Ts×Fs rather than the image space H×W×3. The forward noising process and reverse denoising process follow the same mathematical framework as image diffusion, but the structure of the data introduces domain-specific considerations.

Definition 30 (Spectrogram Diffusion).

Let 𝐒Ts×Fs be a normalised mel spectrogram (treated as a 2D tensor). A spectrogram diffusion model defines a forward process (SPEC Forward)q(𝐒t|𝐒0)=Normal(𝐒t;αt𝐒0,(1αt)𝐈), and trains a denoiser 𝝐θ(𝐒t,t) to predict the noise component, with loss (SPEC LOSS)spec=𝔼t,𝐒0,𝝐[𝝐𝝐θ(𝐒t,t)2], where 𝝐Normal(0,𝐈) and 𝐒t=αt𝐒0+1αt𝝐.

The key question is: what does the denoiser learn about the structure of mel spectrograms? Natural audio spectrograms exhibit a rich set of statistical regularities:

  1. Horizontal continuity: Harmonic components of sustained sounds create horizontal ridges in the spectrogram that persist across many time frames.

  2. Harmonic spacing: A fundamental frequency f0 produces harmonics at 2f0,3f0,, which appear as evenly spaced horizontal lines (on a linear frequency axis) or logarithmically spaced lines (on a mel axis).

  3. Onset transients: Percussive events produce vertical broadband features spanning many frequency bins in a single time frame.

  4. Spectral envelope: The overall energy distribution across frequencies follows smooth, slowly varying envelopes determined by the source and resonance characteristics.

The diffusion model learns these regularities implicitly through the training data. The U-Net architecture is particularly well suited to capturing both local features (individual harmonics) and global structure (musical form, temporal dynamics) through its multi-scale encoder–decoder structure.

Proposition 20 (Spectrogram Resolution Trade-off).

For a short-time Fourier transform with window length N and hop size H, the time and frequency resolutions satisfy the fundamental trade-off (SPEC Uncertainty)ΔtΔf1, where Δt=H/fs is the temporal resolution (in seconds) and Δf=fs/N is the frequency resolution (in Hz), with fs the sampling rate. Equality holds when H=1 and N=fs, which is impractical. In practice, there is a direct trade-off: increasing N improves frequency resolution at the cost of temporal resolution, and vice versa.

Proof.

The frequency resolution of the DFT with window length N is Δf=fs/N (the spacing between adjacent frequency bins). The temporal resolution is determined by the hop size: Δt=H/fs. Therefore, ΔtΔf=HfsfsN=HN. Since the hop size must satisfy H1 (at least one sample between frames) and HN (the hop cannot exceed the window length for meaningful overlap), we have H/N1/N. However, the constraint ΔtΔf1 follows from the Gabor limit (the time-frequency uncertainty principle for windowed transforms): no window function can achieve ΔtΔf<1. The Gaussian window achieves equality in the continuous-time limit.

For the practical STFT, we have ΔtΔf=H/N. Setting H=N (no overlap) gives ΔtΔf=1. Any overlap (H<N) gives ΔtΔf<1, but this does not violate the uncertainty principle; rather, the overlapping frames contain redundant information that does not increase the true joint resolution.

Remark 22.

The time-frequency trade-off has direct implications for spectrogram-domain generation. A common configuration for music generation uses N=2048 samples (46 ms at 44.1 kHz) and H=512 samples (11.6 ms), yielding Δf21.5 Hz and Δt11.6 ms. This provides reasonable resolution for both harmonic analysis and transient detection. For a 30-second clip, the spectrogram has Ts=30/0.01162,586 time frames, which can be unwieldy for diffusion models. Reducing Ts (by increasing H) sacrifices temporal resolution; reducing Fs (by using fewer mel bands) sacrifices frequency resolution. This motivates the use of latent diffusion, which operates on a compressed representation.

Example 14 (Spectrogram Diffusion for Sound Effects).

Consider generating a 4-second sound effect (“thunderstorm with heavy rain”) using a spectrogram diffusion model. With fs=22,050 Hz, N=1024, H=256, and Fs=80 mel bins, the spectrogram has shape 345×80. After padding to 384×128 (to be divisible by the U-Net's downsampling factor), the diffusion model operates on a tensor comparable in size to a small image.

The model generates a spectrogram that captures the broadband noise of rain (diffuse energy across all frequencies), the low-frequency rumble of thunder (concentrated energy below 200 Hz), and the temporal dynamics of thunder claps (transient broadband events). The spectrogram is then converted to a waveform using HiFi-GAN, which generates plausible phase information to create a realistic audio signal.

Limitations of Spectrogram-Domain Generation

Despite its conceptual elegance, spectrogram-domain generation suffers from several fundamental limitations that have driven the field toward latent codec spaces.

Phase loss.

The most fundamental limitation is the loss of phase information. A mel spectrogram represents only the magnitude of the STFT, discarding the phase component. This means that the diffusion model generates magnitude-only representations, and the phase must be estimated during the inversion step.

Definition 31 (Phase Reconstruction Error).

Let 𝐒^Ts×Fs be a generated mel spectrogram with corresponding ground-truth magnitude spectrogram 𝐒 and phase Φ[π,π]Ts×N/2+1. Let Φ^ denote the phase estimated by the inversion algorithm. The phase reconstruction error is (Phase Error)phase=1Ts(N/2+1)m,k|eiΦ^[m,k]eiΦ[m,k]|2=2Ts(N/2+1)m,k(1cos(Φ^[m,k]Φ[m,k])). This error is zero when the estimated phase matches the true phase and reaches its maximum of 4 when the phases are perfectly anti-aligned.

Even state-of-the-art neural vocoders introduce subtle phase artefacts, particularly in regions of the spectrogram with rapid temporal variation (transients) or closely spaced harmonics. These artefacts manifest as “buzziness,” “phasiness,” or reduced clarity in the reconstructed audio.

Frequency resolution.

The mel frequency scale provides perceptually motivated frequency resolution: high resolution at low frequencies (where pitch perception is acute) and lower resolution at high frequencies (where the ear is less sensitive to fine spectral detail). However, this compression means that the generated spectrogram has limited control over the fine frequency structure of the audio.

Proposition 21 (Mel Frequency Resolution Bound).

For a mel filterbank with Fs filters spanning the frequency range [fmin,fmax], the frequency resolution at frequency f is approximately (MEL Resolution)Δfmel(f)f+700Fsln(fmax+700fmin+700). In particular, Δfmel is approximately proportional to f+700, so the resolution degrades linearly with frequency. At f=4000 Hz with Fs=128 mel bins, Δfmel58 Hz, which means that harmonics separated by less than 58 Hz cannot be distinguished in the mel representation.

Proof.

The mel scale maps frequency f to m(f)=2595log10(1+f/700). The inverse is f(m)=700(10m/25951). For Fs uniformly spaced mel filters between m(fmin) and m(fmax), the mel spacing is Δm=(m(fmax)m(fmin))/Fs. The frequency resolution at mel value m is Δf=|df/dm|Δm=700ln(10)/259510m/2595Δm. Since 10m/2595=1+f/700, this simplifies to Δf=(f+700)ln(10)/2595Δm. Converting Δm back to the stated form using Δm=(m(fmax)m(fmin))/Fs=2595log10((fmax+700)/(fmin+700))/Fs, and noting ln(10)log10(x)=ln(x), we obtain Δfmel(f)(f+700)ln((fmax+700)/(fmin+700))/Fs.

Limited duration.

Spectrogram-domain generation inherits the resolution constraints of the underlying image generation model. Standard diffusion models operate on fixed-resolution images (e.g., 512×512), which limits the spectrogram to a fixed time–frequency grid. For a hop size of H=512 at fs=44,100 Hz, a 512-frame spectrogram represents only 512×512/441005.9 seconds of audio. Generating longer clips requires either: (i) sequential generation with overlap and crossfade, which introduces stitching artefacts; or (ii) higher-resolution generation, which is computationally expensive and may not be supported by the architecture.

Remark 23 (Why Modern Systems Moved to Latent Codec Space).

The limitations of spectrogram-domain generation, particularly phase loss, limited frequency resolution, and fixed duration constraints, motivated the development of generation models that operate in the latent space of neural codecs. In a neural codec latent space:

  1. Phase is preserved: The codec encoder and decoder jointly learn to encode and reconstruct the full waveform, including phase information. There is no separate inversion step.

  2. Resolution is adaptive: The RVQ codebook captures audio structure at multiple resolution scales through its hierarchical refinement, rather than being fixed by the STFT parameters.

  3. Duration is flexible: Neural codec tokens operate at a fixed temporal rate (e.g., 50 Hz), so longer clips simply produce longer token sequences without changing the frequency resolution.

The transition from spectrogram-domain to codec-domain generation, which occurred rapidly between 2022 and 2023, mirrors the earlier transition from pixel-space to latent-space diffusion in image generation. In both cases, the insight is the same: operate in a learned representation space that captures the essential structure of the data, rather than in a hand-designed representation (spectrograms, pixels) that was not optimised for generation.

Conjecture 1 (Codec Latent Spaces Subsume Spectrogram Spaces).

Let 𝒮={𝐒Ts×Fs:𝐒=mel(STFT(𝒙)),𝒙L} be the manifold of valid mel spectrograms and let 𝒵={𝒛Tz×dz:𝒛=codec(𝒙),𝒙L} be the manifold of neural codec latent embeddings. We conjecture that for a sufficiently expressive codec, 𝒵 captures strictly more information about the audio signal than 𝒮: H(𝒙|𝒛)<H(𝒙|𝐒), where H denotes conditional entropy. In other words, the neural codec latent representation leaves less residual uncertainty about the original waveform than the mel spectrogram representation. This conjecture, if true, would formally justify the empirical observation that codec-based generation outperforms spectrogram-based generation.

Historical Note.

Riffusion appeared in late 2022 as an open-source project, not a peer-reviewed paper. Despite its simplicity (or perhaps because of it), the project attracted enormous attention, demonstrating that off-the-shelf image generation models could produce recognisable music with minimal adaptation. The project's name is a portmanteau of “riff” (a musical phrase) and “diffusion.” While Riffusion itself was quickly surpassed by purpose-built audio generation models, it served as an important conceptual bridge, showing the research community that the mathematical framework of diffusion models was not inherently tied to the image domain.

Exercises

Exercise 7 (CLAP Batch Size and Mutual Information).

Consider a CLAP model trained with the symmetric contrastive loss ((CLAP LOSS)).

  1. Show that the audio-to-text loss at is bounded below by 0 and bounded above by logB, where B is the batch size. When is each bound achieved?

  2. Using the mutual information bound from Proposition 12, argue that a CLAP model trained with batch size B1=256 can capture at most log2565.55 nats of mutual information between the audio and text modalities, while a model trained with B2=4096 can capture up to log40968.32 nats. What are the practical implications for audio generation quality?

  3. Suppose the true mutual information between the audio and text distributions is I=7.0 nats. What is the minimum batch size B required for the InfoNCE bound to be non-vacuous (i.e., for the bound to be positive)? Compute the exact value.

Exercise 8 (Codebook Flattening Patterns).

Consider a neural codec with T=100 time steps, Q=8 codebook levels, and vocabulary size K=1024.

  1. Compute the autoregressive sequence length for each of the three flattening patterns (interleaved, delayed, parallel).

  2. For the interleaved pattern, compute the total number of possible token sequences (i.e., the cardinality of the discrete output space). Express your answer as a power of 2.

  3. The delayed pattern predicts Q tokens per step in parallel. However, at the first and last few steps, some codebook levels are inactive (masked). How many steps have fewer than Q active codebook levels? For each such step, specify exactly which codebook levels are active.

  4. Suppose the Transformer has NL=24 layers, hidden dimension d=1024, and processes one step in 5 ms (with KV-cache). Estimate the total generation time for each pattern, assuming the parallel pattern also takes 5 ms per step (despite predicting Q tokens).

Exercise 9 (FiLM vs. Cross-Attention Expressivity).

Consider a sequence of hidden states 𝐇Tseq×d and a conditioning embedding 𝒄dc.

  1. Show that FiLM conditioning (Definition 28) applies the same affine transformation to every position t=1,,Tseq. That is, FiLM(𝒉t;𝜸,𝜷) depends on 𝒄 but not on t.

  2. Now consider cross-attention with a single conditioning token (Tc=1). Show that in this case, cross-attention reduces to a position-independent transformation of 𝐂cond, weighted by a position-dependent scalar. How does this compare to FiLM?

  3. Argue that for Tc>1, cross-attention is strictly more expressive than FiLM, in the sense that there exist conditioning functions realisable by cross-attention but not by FiLM\@. Give a concrete example involving a two-segment audio prompt (e.g., “piano in the first half, drums in the second half”).

  4. Compute the parameter count for FiLM conditioning (two matrices 𝐖γ,𝐖βd×dc plus biases) and for a single cross-attention layer (four projection matrices 𝐖Q,𝐖K,𝐖V,𝐖Od×d). For d=1024 and dc=512, which mechanism is more parameter-efficient?

Exercise 10 (Spectrogram Resolution and Duration).

A spectrogram-domain diffusion model operates on mel spectrograms of fixed size 512×128 (time frames × frequency bins).

  1. If the sampling rate is fs=22,050 Hz and the hop size is H=256 samples, what is the maximum audio duration (in seconds) that the model can generate in a single pass?

  2. For the same configuration, compute the time resolution Δt and the frequency resolution Δf (assuming window length N=4H=1024). Verify the uncertainty relation ΔtΔf1.

  3. A competing model uses H=512 and N=2048. Compare its maximum duration, time resolution, and frequency resolution with the model in part (a). Which model would you prefer for generating speech? For generating music with rich harmonics? Justify your answers.

  4. Suppose you want to generate 60 seconds of audio using the spectrogram approach. Describe a scheme based on overlapping segments with linear crossfade. If each segment is 5 seconds long with 1 second of overlap, how many diffusion model evaluations are needed? What artefact might this introduce at segment boundaries?

Latent Diffusion for Audio

The preceding sections developed two pillars of modern audio generation: contrastive embeddings that align audio and text (CLAP: Contrastive Language–Audio Pretraining), and autoregressive models that treat audio tokens as a language (Autoregressive Music Generation). We now turn to a third pillar, one that has arguably had the greatest impact on text-to-audio quality and diversity: latent diffusion models (LDMs). The core idea, introduced for images in ch:diffusion and now adapted to audio, is to decompose generation into three conceptually distinct stages:

  1. Compress: an autoencoder maps the high-dimensional audio signal (or its mel-spectrogram representation) to a compact latent space.

  2. Diffuse: a diffusion model generates novel latent representations conditioned on text, tags, or other control signals.

  3. Decode: the autoencoder's decoder (often augmented by a neural vocoder) maps the generated latent back to a waveform.

This three-stage decomposition is not merely an engineering convenience; it offers precise mathematical advantages. By learning to generate in a compressed latent space rather than in raw waveform or spectrogram space, the diffusion model operates in a domain where the data manifold has lower intrinsic dimension, the signal-to-noise ratio is more favourable, and the computational cost per denoising step is dramatically reduced.

In this section, we formalise the three-stage pipeline (The Three-Stage Pipeline), study AudioLDM, the system that established the paradigm (AudioLDM: The Foundational System), then examine AudioLDM2's architectural innovations (AudioLDM2: Language-Aligned Latents), and finally develop MusicLDM's beat-synchronous data augmentation (MusicLDM: Beat-Synchronous Augmentation).

The Three-Stage Pipeline

We begin with a precise formulation of the three-stage architecture.

Definition 32 (Latent Audio Diffusion Pipeline).

A latent audio diffusion pipeline consists of three components:

  1. A pretrained audio autoencoder (,𝒟), where the encoder :Ts×FsTz×Cz maps a mel-spectrogram 𝐒 to a latent representation 𝒛0=(𝐒), and the decoder 𝒟:Tz×CzTs×Fs reconstructs 𝐒^=𝒟(𝒛0)𝐒. The temporal and channel dimensions satisfy Tz<Ts and CzFs, achieving spatial compression.

  2. A conditional diffusion model 𝝐θ:Tz×Cz×[0,T]×dcTz×Cz that operates entirely in the latent space. Given a noisy latent 𝒛t, a diffusion timestep t, and a conditioning embedding 𝒄dc, it predicts the noise 𝝐 added during the forward process.

  3. A vocoder 𝒱:Ts×FsL that converts the reconstructed mel-spectrogram to a waveform 𝒙^=𝒱(𝐒^). In practice, the vocoder is often HiFi-GAN (HiFi-GAN: Generator Architecture) or a similar neural vocoder.

The full generation process is the composition 𝒙^=𝒱𝒟Denoiseθ(𝒛T,𝒄), where 𝒛TNormal(0,𝐈) is pure noise in the latent space.

Key Idea.

Compression enables tractable diffusion. The fundamental insight of latent diffusion is that the autoencoder absorbs the “perceptual” complexity of the signal. The diffusion model need not learn to generate every fine detail of the mel spectrogram; it need only produce a latent code that, when decoded, yields a perceptually convincing spectrogram. For audio, this compression is substantial: a 10-second clip at 16,kHz produces a mel-spectrogram of size 1000×64 (64,000 values), while the latent representation might be 250×8 (2,000 values), a 32× reduction. This compression directly translates to faster training and inference, since the U-Net operates on tensors that are 32 times smaller.

The three-stage latent audio diffusion pipeline. During training, the autoencoder (,𝒟) is trained first to reconstruct mel spectrograms (dashed blue arrows). The diffusion model 𝝐θ then learns to generate in the frozen latent space, conditioned on CLAP or T5 embeddings (amber arrow). At inference (solid green arrows), the diffusion model denoises Gaussian noise into a latent 𝒛^0, which the decoder maps to a mel spectrogram, and the vocoder converts to a waveform.

The mathematical advantage of the three-stage decomposition can be quantified through the lens of the evidence lower bound (ELBO). For a standard diffusion model operating on mel spectrograms, the variational bound decomposes as (ELBO Pixel)logp(𝐒)𝔼q(𝒛1:T|𝐒)[logp(𝒛T)t=1Tpθ(𝒛t1|𝒛t)t=1Tq(𝒛t|𝒛t1)]. When the diffusion operates in the latent space of a pretrained autoencoder, we instead lower-bound logp(𝐒) in two stages: (ELBO Latent 1)logp(𝐒)𝔼𝒛0=(𝐒)[logpθ(𝒛0)+logp𝒟(𝐒|𝒛0)],logpθ(𝒛0)ELBOdiffusion(𝒛0), where p𝒟(𝐒|𝒛0) captures the reconstruction quality of the autoencoder. The key insight is that ELBOdiffusion involves denoising in the low-dimensional latent space, which is computationally cheaper and statistically easier to learn.

Proposition 22 (Computational Advantage of Latent Diffusion).

Let 𝐒Ts×Fs be a mel-spectrogram and 𝒛0=(𝐒)Tz×Cz its latent encoding, with spatial compression ratio r=(TsFs)/(TzCz). Suppose the denoising network is a U-Net whose computational cost scales as Θ(d2) where d is the input dimensionality. Then the per-step cost of latent diffusion relative to spectrogram-space diffusion satisfies (Latent COST Ratio)CostlatentCostspec=1r2, and the total training cost ratio (assuming equal numbers of diffusion steps) is also 1/r2. For typical audio autoencoders with r32, this yields a 1000× reduction in per-step compute.

Proof.

The U-Net processes its input through a sequence of convolutional layers whose cost is dominated by the spatial dimensions of the feature maps. For an input of total dimensionality dspec=TsFs (spectrogram space) or dlat=TzCz (latent space), the dominant cost term in each convolutional layer is proportional to the spatial resolution of the feature maps, which scales as d. The quadratic dependence arises because the U-Net's attention layers (applied at lower resolutions) have cost O(d2) in the spatial dimension. Therefore, CostlatentCostspec=dlat2dspec2=(TzCz)2(TsFs)2=1r2. For r=32, we obtain 1/r2=1/1024103.

Remark 24.

The compression ratio r cannot be increased without bound. As r grows, the autoencoder must discard increasingly perceptually relevant information, degrading the quality ceiling of the entire pipeline. In practice, audio latent diffusion systems use r[16,64], balancing compression against reconstruction fidelity. The autoencoder's reconstruction loss (measured by Fréchet Audio Distance or multi-resolution STFT loss) serves as an upper bound on the quality achievable by any diffusion model operating in that latent space, regardless of how well the diffusion model itself is trained.

Autoencoder design for audio latent diffusion.

The audio autoencoder is typically a variational autoencoder (VAE) trained on mel spectrograms. The encoder produces a mean and log-variance for each latent position: (VAE Encode)(𝐒)=𝝁+𝝈𝝐,𝝁,log𝝈2=EncoderNet(𝐒),𝝐Normal(0,𝐈), where the reparametrisation trick enables gradient-based training. The VAE is trained with a combination of reconstruction loss and KL regularisation: (VAE LOSS)VAE=𝐒𝐒^1+λstftMR-STFT(𝐒,𝐒^)reconstruction+λKLDKL(q(𝒛|𝐒)Normal(0,𝐈))regularisation, where MR-STFT is the multi-resolution STFT loss (HiFi-GAN Training Losses) and λKL is a small weighting coefficient (typically 106 to 104) to prevent posterior collapse while keeping the latent space smooth.

Caution.

The KL regularisation weight λKL requires careful tuning in the audio setting. Too large a value forces the latent space toward a standard Gaussian, destroying audio structure and leading to blurry, reverberant reconstructions. Too small a value produces a latent space with “holes” (regions of low density) that confuse the diffusion model during generation. A common diagnostic is to sample 𝒛Normal(0,𝐈) from the prior and decode: if the result is completely unintelligible, the KL weight is too low; if it sounds like white noise convolved with a room impulse response, the KL weight is too high.

AudioLDM: The Foundational System

AudioLDM, introduced by Liu et al. (2023), was the first system to demonstrate that the latent diffusion paradigm from image generation could be transferred wholesale to audio, achieving state-of-the-art text-to-audio results with a remarkably simple architecture. The system consists of three components: a pretrained audio VAE, a U-Net-based diffusion model conditioned on CLAP embeddings, and a HiFi-GAN vocoder.

Architecture.

The AudioLDM pipeline instantiates Definition 32 as follows. The audio VAE is trained on mel spectrograms of size 𝐒Ts×Fs (typically Ts=1024, Fs=64 for 10-second clips at 16,kHz with a hop size of 160 samples). The encoder compresses by a factor of 4× along the time axis and 8× along the frequency axis, producing latents 𝒛0256×8. The U-Net follows the architecture from Stable Diffusion (Rombach et al., 2022), adapted for the two-dimensional latent spectrogram shape: it uses 2D convolutions (treating time and compressed-frequency as spatial axes), self-attention at the bottleneck resolution, and cross-attention layers that attend to the conditioning embeddings.

CLAP conditioning.

The conditioning embedding 𝒄 is obtained from a pretrained CLAP model (CLAP: Contrastive Language–Audio Pretraining). Given a text prompt 𝒙(t), the CLAP text encoder produces a global embedding 𝒄global=gϕ(𝒙(t))d. This embedding is injected into the U-Net via cross-attention: (Audioldm Crossattn)CrossAttn(𝒉,𝒄global)=softmax((𝒉𝐖Q)(𝒄global𝐖K)dh)(𝒄global𝐖V), where 𝒉 represents the spatial features at a given U-Net layer and the CLAP embedding is treated as a sequence of length 1. This is equivalent to the image-domain cross-attention mechanism studied in ch:diffusion, with CLAP replacing CLIP as the conditioning encoder.

Proposition 23 (AudioLDM Training Objective).

The AudioLDM diffusion model is trained to minimise the noise-prediction objective in the latent space of the pretrained audio VAE\@. Given a clean latent 𝒛0=(𝐒), a noise sample 𝝐Normal(0,𝐈), a uniformly sampled timestep t𝒰{1,,T}, and a conditioning embedding 𝒄, the training loss is (Audioldm LOSS)AudioLDM=𝔼𝒛0,𝝐,t,𝒄[𝝐𝝐θ(𝒛t,t,𝒄)22], where 𝒛t=αt𝒛0+1αt𝝐 is the noised latent following the standard DDPM forward process with cumulative noise schedule αt=s=1t(1βs).

Proof.

This follows directly from the DDPM training objective (ch:diffusion), applied to the latent variable 𝒛0 instead of the data variable 𝐒. The key observation is that once the autoencoder is frozen, the distribution of 𝒛0=(𝐒) over the training set defines a fixed “data distribution” in latent space, and the standard diffusion framework applies without modification. The conditioning embedding 𝒄 enters through the cross-attention layers of 𝝐θ and does not affect the mathematical form of the loss.

Example 15 (AudioLDM Inference).

To generate a 10-second audio clip from the text prompt “a dog barking in a park with birds singing,” AudioLDM proceeds as follows:

  1. Encode the text prompt using CLAP: 𝒄=gϕ(“a dog barking).

  2. Sample initial noise: 𝒛TNormal(0,𝐈), with 𝒛T256×8.

  3. Run the DDPM reverse process for T=200 steps (or DDIM for 50 steps), each step applying the U-Net 𝝐θ(𝒛t,t,𝒄).

  4. Decode the denoised latent: 𝐒^=𝒟(𝒛^0)1024×64.

  5. Convert to waveform using HiFi-GAN: 𝒙^=𝒱(𝐒^)160,000.

The entire process takes approximately 5 seconds on an A100 GPU with DDIM, achieving a real-time factor of 0.5.

Insight.

CLAP as a universal audio conditioning interface. A subtle but important property of AudioLDM's design is that the CLAP text encoder can be replaced at inference time with the CLAP audio encoder. Since both encoders map to the same embedding space, one can condition generation on a reference audio clip instead of a text description, enabling audio-to-audio style transfer without any architectural changes. This “plug-and-play” property is a direct consequence of the symmetric training objective studied in CLAP: Contrastive Language–Audio Pretraining: the shared embedding space is agnostic to whether the input is text or audio.

AudioLDM2: Language-Aligned Latents

AudioLDM2 (Liu et al., 2024) addresses a fundamental limitation of the original AudioLDM: the CLAP embedding is a single global vector, which provides only a coarse, non-temporal conditioning signal. A text prompt like “a piano plays a gentle melody, then thunder rumbles” describes a temporal sequence of events, but a single CLAP vector cannot represent this ordering. AudioLDM2 introduces two architectural innovations to enable richer, temporally structured conditioning: an AudioMAE-based representation learning module and a Language-Oriented Audio (LOA) generation framework.

AudioMAE representation.

AudioLDM2 replaces the single CLAP vector with a sequence of latent vectors derived from an AudioMAE (Audio Masked Autoencoder). The AudioMAE is a vision transformer trained on mel spectrograms using the masked autoencoding objective: random patches of the spectrogram are masked, and the model learns to reconstruct them from the visible patches. After pretraining, the AudioMAE encoder produces a sequence of patch embeddings that capture both local spectral features and global temporal structure.

Definition 33 (AudioMAE Representation).

Let 𝐒Ts×Fs be a mel spectrogram partitioned into non-overlapping patches of size pt×pf, yielding Np=Ts/ptFs/pf patches. The AudioMAE encoder ψ:Ts×FsNp×dm maps the spectrogram to a sequence of Np patch embeddings, each of dimension dm. During pretraining, a fraction ρ (typically ρ=0.8) of the patches are masked, and the model is trained to minimise (Audiomae LOSS)AudioMAE=1||i𝒔i𝒔^i22, where is the set of masked patch indices, 𝒔i is the ground-truth patch, and 𝒔^i is the model's reconstruction.

Language-Oriented Audio (LOA) generation.

The central innovation of AudioLDM2 is the LOA module, which bridges the gap between language representations and audio latent representations. LOA consists of a GPT-2-style autoregressive model that takes as input a sequence-level language representation (from FLAN-T5 or CLAP) and outputs a sequence of AudioMAE-compatible embeddings. These embeddings serve as the conditioning signal for the latent diffusion model, replacing the single CLAP vector of AudioLDM\@.

Formally, let 𝒄langLt×dt be the output of the language encoder (FLAN-T5) for a text prompt, where Lt is the sequence length and dt is the language embedding dimension. The LOA module generates a sequence of AudioMAE-like tokens: (LOA)𝒉i=LOA-GPT(𝒄lang,𝒉1,,𝒉i1),i=1,,Na, where 𝒉idm is the i-th generated audio token and Na is the target sequence length (matching the temporal resolution of the diffusion model's conditioning input).

The generated sequence 𝐇LOA=(𝒉1,,𝒉Na)Na×dm then serves as the cross-attention key and value for the latent diffusion U-Net: (Audioldm2 Crossattn)CrossAttn(𝒇,𝐇LOA)=softmax((𝒇𝐖Q)(𝐇LOA𝐖K)dh)(𝐇LOA𝐖V), where 𝒇 denotes the spatial features at a given U-Net layer. Unlike AudioLDM's single-vector conditioning, this provides spatially varying conditioning: different temporal positions in the latent spectrogram attend to different audio tokens, enabling the model to generate temporally structured outputs.

Proposition 24 (Expressivity of Sequence-Level Conditioning).

Let dh be the attention head dimension and Na the number of conditioning tokens. The cross-attention output at spatial position j of the U-Net is (SEQ COND Output)𝒐j=i=1Naαji(𝒉i𝐖V),αji=exp((𝒇j𝐖Q)(𝒉i𝐖K)/dh)k=1Naexp((𝒇j𝐖Q)(𝒉k𝐖K)/dh). When Na=1 (AudioLDM), αj1=1 for all j, and the conditioning is spatially uniform: 𝒐j=𝒉1𝐖V for all j. When Na>1 (AudioLDM2), the attention weights αji vary with j, allowing different spatial positions to receive different conditioning signals.

Proof.

For Na=1, the softmax over a single element is trivially αj1=1 regardless of 𝒇j, so the output is 𝒐j=𝒉1𝐖V, which is independent of j. For Na>1, the attention weights depend on the query 𝒇j𝐖Q, which varies across spatial positions j. Different queries produce different weight distributions over the Na conditioning tokens, yielding spatially varying outputs.

Remark 25.

AudioLDM2 is designed as a unified framework that generates speech, sound effects, and music from the same architecture by varying only the language encoder. For sound effects and music, FLAN-T5 embeddings are used; for speech synthesis, a speaker embedding from a pretrained speaker verification model is concatenated with the T5 embeddings. The LOA module learns to translate these heterogeneous conditioning signals into a common AudioMAE-like representation, enabling the same diffusion model to serve all three domains.

MusicLDM: Beat-Synchronous Augmentation

Music generation poses unique challenges compared to general audio generation. Music has strong temporal structure: beats, bars, phrases, and sections create hierarchical rhythmic patterns that listeners perceive and expect. MusicLDM (Chen et al., 2024) addresses this by introducing beat-synchronous mixup, a data augmentation strategy that creates new training examples by mixing pairs of music clips at beat boundaries. The insight is that mixing at metrically meaningful points produces musically coherent augmentations, whereas mixing at arbitrary time points produces artefacts that degrade generation quality.

Definition 34 (Beat-Synchronous Mixup).

Let (𝒙1,𝒄1) and (𝒙2,𝒄2) be two music clips with their corresponding text descriptions, and let {b1(1),b2(1),} and {b1(2),b2(2),} be their detected beat positions (in samples). A beat-synchronous mixup produces a new training pair (𝒙~,𝒄~) as follows:

  1. Select a mixing ratio λBeta(a,a) for a hyperparameter a>0 (typically a=0.2).

  2. Find the beat position bk(1) closest to λL, where L is the clip length in samples.

  3. Construct the mixed waveform: (BEAT Mixup)𝒙~[n]={𝒙1[n]if n<bk(1),𝒙2[nbk(1)+bj(2)]if nbk(1), where bj(2) is the beat in 𝒙2 closest to (1λ)L2.

  4. Construct the mixed caption: 𝒄~=𝒄1, followed by 𝒄2.

The key property of beat-synchronous mixup is that the transition between clips occurs at a metrically strong position (a beat), which is perceptually natural. This contrasts with naive random-point mixup, which can cut a note in the middle of its sustain or split a drum hit, creating jarring discontinuities.

Proposition 25 (Beat-Synchronous Mixup Preserves Metrical Structure).

Let μ1 and μ2 be the inter-beat intervals (in seconds) of clips 𝒙1 and 𝒙2, and assume both clips have approximately constant tempo. If the beat detection algorithm has error |b^kbk|δ for all detected beats, then the mixed clip 𝒙~ has the following properties:

  1. The transition point is within δ samples of a true beat in 𝒙1.

  2. The portion of 𝒙2 starts within δ samples of a true beat in 𝒙2.

  3. If μ1=μ2 (matching tempi), then the mixed clip has approximately constant tempo throughout, with phase discontinuity bounded by 2δ/fs seconds.

Proof.

Properties (i) and (ii) follow directly from the beat detection error bound: the selected beat positions bk(1) and bj(2) are within δ of the true beat positions. For property (iii), if both clips have the same inter-beat interval μ=μ1=μ2, then the beats in the first segment of 𝒙~ are spaced by μ, and the beats in the second segment are also spaced by μ. The transition introduces at most a 2δ sample displacement, corresponding to a temporal error of 2δ/fs seconds. For typical beat detection accuracy (δ10 ms ×fs) at fs=16,000 Hz, this is at most 20 ms, well below the perceptual threshold for metrical disruption (approximately 50 ms).

Remark 26.

MusicLDM explores three augmentation strategies beyond beat-synchronous mixup:

  1. Latent mixup: Instead of mixing waveforms, mix the VAE latent representations: 𝒛~0=λ𝒛0(1)+(1λ)𝒛0(2). This is computationally cheaper but does not respect beat structure.

  2. CLAP embedding mixup: Mix the conditioning embeddings: 𝒄~=λ𝒄1+(1λ)𝒄2. This operates in the semantic space rather than the acoustic space.

  3. Joint mixup: Apply both latent and CLAP mixup simultaneously, with the same λ.

Empirically, beat-synchronous mixup at the waveform level yields the best results for music generation, while latent mixup is more effective for general sound effects where beat structure is absent.

Example 16 (MusicLDM Training Data Size).

MusicLDM is trained on approximately 2,000 hours of music with text captions, a relatively modest dataset compared to the millions of hours used by commercial music generation systems. Beat-synchronous mixup effectively increases the training diversity by producing (N2) possible mixup pairs from N original clips. For N=100,000 clips (roughly 2,000 hours at 72 seconds per clip), this yields approximately 5×109 possible augmented examples, a combinatorial explosion that substantially enriches the training distribution without requiring additional data collection.

Diffusion Architectures for Audio

The previous section established the three-stage latent diffusion pipeline and studied systems built around U-Net denoisers. In this section, we examine the architectural design space more broadly. The central question is: what neural network architecture should serve as the denoising backbone for audio diffusion? Two families dominate the landscape: the U-Net, inherited from image diffusion (ch:diffusion), and the Diffusion Transformer (DiT), adapted from the vision transformer paradigm (sec:vdiff:dit). Each architecture offers distinct mathematical trade-offs in terms of inductive bias, scalability, and conditioning flexibility.

We study three influential systems that explore different points in this design space: Stable Audio, which pioneers the DiT approach with timing conditioning (Stable Audio: DiT with Timing Conditioning); Noise2Music, which uses a cascaded U-Net architecture (Noise2Music: Cascaded Generation); and Moûsai, which combines diffusion with efficient U-Net variants (Moûsai: Efficient Diffusion for Long Audio).

U-Net vs. DiT: The Architectural Divide

The U-Net and the DiT represent fundamentally different inductive biases for the denoising function 𝝐θ:Tz×Cz×[0,T]×dcTz×Cz.

U-Net.

The U-Net processes the latent spectrogram through an encoder-decoder structure with skip connections at matching resolutions. At each resolution level , the encoder applies a sequence of residual blocks and (optionally) self-attention: (UNET Block)𝒉(+1)=Downsample(ResBlock(𝒉(),t)+SelfAttn(𝒉())+CrossAttn(𝒉(),𝒄)), where the timestep t is injected via adaptive group normalisation (AdaGN), and the conditioning 𝒄 enters through cross-attention. The decoder mirrors this structure with upsampling operations and skip connections from the encoder. The U-Net's multi-scale structure is a natural fit for spectrograms, which exhibit different statistical properties at different time-frequency scales.

DiT.

The Diffusion Transformer treats the latent representation as a sequence of patches and processes it with a standard transformer encoder. The latent 𝒛tTz×Cz is first patchified into a sequence of Np tokens, each of dimension dmodel: (DIT Patchify)𝒛tpatchify(𝒑1,,𝒑Np),𝒑idmodel. Each transformer block then applies self-attention and a feed-forward network with adaptive layer normalisation (adaLN) for timestep and conditioning injection: (DIT Block)𝒉i=𝒉i+γ1(t,𝒄)SelfAttn(LN(𝒉i;β1(t,𝒄))), where γ1,β1 are scale and shift parameters predicted from the timestep and conditioning embeddings. The DiT's advantage is scalability: it has no resolution-specific components, and its compute scales linearly with the number of layers for a fixed sequence length, making it straightforward to scale to billions of parameters.

Proposition 26 (Receptive Field Comparison).

Consider a latent spectrogram of temporal length Tz.

  1. For a U-Net with L resolution levels and kernel size k at each level, the effective receptive field at the bottleneck is Tz/2L1 tokens, and self-attention at the bottleneck provides global context over this compressed representation. The full-resolution layers have local receptive fields of size O(kL).

  2. For a DiT with NL transformer layers, every layer has global receptive field over all Np tokens (full self-attention). After a single layer, each token can attend to any other token.

Proof.

For the U-Net, the encoder downsamples by a factor of 2 at each of L levels, reducing the temporal dimension from Tz to Tz/2L1 at the bottleneck. Self-attention at this level is global over the compressed sequence. At the full resolution (first encoder level), each convolutional layer with kernel size k has receptive field k, and L stacked layers yield O(kL) by the standard receptive field computation for stacked convolutions. For the DiT, self-attention computes pairwise interactions between all Np tokens at every layer, giving a global receptive field of Np from the first layer onward.

Stable Audio: DiT with Timing Conditioning

Stable Audio (Evans et al., 2024) was the first commercially deployed music generation system built entirely on the DiT architecture. It introduces two key innovations: explicit timing conditioning that enables variable-length generation up to 95 seconds, and a waveform-domain VAE that bypasses the mel spectrogram representation entirely.

Waveform VAE.

Unlike AudioLDM, which compresses mel spectrograms, Stable Audio trains its VAE directly on raw waveforms. The encoder operates on the waveform 𝒙L (at 44.1,kHz) and produces a continuous latent sequence 𝒛0Tz×Cz with a temporal compression factor of 2048× (from 44.1,kHz to approximately 21.5,Hz). This extreme compression is achieved through a series of strided 1D convolutions: (Stable Audio VAE)𝒛0=wav(𝒙),Tz=Lisi, where s1,s2, are the stride factors of successive convolutional layers, with isi=2048.

Definition 35 (Timing-Conditioned DiT).

A timing-conditioned DiT augments the standard DiT architecture with two additional scalar conditioning signals:

  1. Start time τs[0,τmax]: the timestamp (in seconds) at which the generated audio segment begins within the full track.

  2. Total duration τd(0,τmax]: the total duration of the track from which this segment originates.

These scalars are encoded via sinusoidal embeddings: (Timing Embed)𝒆τ=[sin(2πf1τ),cos(2πf1τ),,sin(2πfKτ),cos(2πfKτ)]2K, where f1,,fK are learnable or fixed frequency bases, and τ{τs,τd}. The timing embeddings are concatenated with the diffusion timestep embedding and projected into the adaLN parameters of each DiT block.

Insight.

Why timing conditioning matters for music. Music tracks have structure at multiple temporal scales: an intro at the beginning, a build-up toward the middle, and an outro at the end. Without timing information, a diffusion model trained on random crops of full tracks has no way to know whether it should generate an intro, a chorus, or an ending. The start-time conditioning τs resolves this ambiguity, allowing the model to generate structurally appropriate content for any position within a track. The duration conditioning τd further informs the model about the overall scale of the piece: a 30-second jingle has different structural expectations than a 5-minute song.

Stable Audio architecture details.

The DiT in Stable Audio uses NL=24 transformer layers with hidden dimension d=1536 and 24 attention heads. The total parameter count is approximately 1.1B\@. Conditioning is provided through three channels:

  1. Text embeddings from a CLAP text encoder, injected via cross-attention at every transformer block.

  2. Timing embeddings (𝒆τs,𝒆τd), injected via adaLN alongside the diffusion timestep.

  3. Diffusion timestep embedding 𝒆t, also injected via adaLN.

The combined conditioning vector for adaLN is (Stable Audio COND)𝒄adaLN=MLP(𝒆t+𝒆τs+𝒆τd)d, from which the scale and shift parameters (γ,β) are extracted for each transformer layer.

Stable Audio architecture. A waveform-domain VAE encodes raw audio into latent tokens with 2048× compression. A DiT with 24 layers generates in this latent space, conditioned on CLAP text embeddings (via cross-attention), timing signals (τs,τd) and diffusion timestep t (via adaLN). The VAE decoder maps the denoised latent directly to a waveform.

Example 17 (Stable Audio Generation Parameters).

To generate a 45-second music track with Stable Audio:

  • Waveform length: L=45×44,100=1,984,500 samples.

  • Latent sequence length: Tz=1,984,500/2048=969 tokens.

  • Latent channels: Cz=64.

  • DiT input: a sequence of 969 tokens, each of dimension 64, projected to d=1536.

  • Timing conditioning: τs=0.0 (start of track), τd=45.0 (total duration).

  • Diffusion steps: 100 (using DPM-Solver++).

Each DiT forward pass processes 969×1536 activations through 24 layers, requiring approximately 25,ms on an A100. Total generation time: 100×0.025=2.5 seconds, well under real time.

Noise2Music: Cascaded Generation

Noise2Music (Huang et al., 2023) takes a different architectural approach: instead of a single diffusion model in latent space, it uses a cascade of two diffusion models operating at different resolutions in spectrogram space. This mirrors the cascaded approach used in image generation (e.g., DALL-E 2, Imagen) and provides a natural decomposition of the generation problem into coarse structure and fine detail.

Definition 36 (Cascaded Audio Diffusion).

A cascaded audio diffusion system consists of two stages:

  1. A base model 𝝐θ(base):Ts(lo)×Fs×[0,T]×dcTs(lo)×Fs that generates a low-resolution mel spectrogram 𝐒(lo)Ts(lo)×Fs from text conditioning.

  2. A super-resolution model 𝝐ϕ(sr):Ts×Fs×[0,T]×dc×Ts(lo)×FsTs×Fs that upsamples the base spectrogram to full resolution 𝐒Ts×Fs, conditioned on both the text and the low-resolution output.

The full generation factorises as (Cascaded Factor)p(𝐒|𝒄)=pϕ(𝐒|𝐒(lo),𝒄)pθ(𝐒(lo)|𝒄)d𝐒(lo).

Base model.

The Noise2Music base model generates 3.2-second spectrograms at a reduced temporal resolution (Ts(lo)=64 frames, corresponding to a 50 ms hop at the spectrogram level). It uses a 1D U-Net operating along the time axis, treating the Fs frequency bins as channels. The text conditioning is provided by a large language model (LaMDA) whose embeddings are injected via cross-attention.

Super-resolution model.

The super-resolution model takes the base spectrogram 𝐒(lo) (bilinearly upsampled to full resolution) as an additional input channel, concatenated with the noisy full-resolution spectrogram at each diffusion step: (SR Input)input=[𝒛t;Upsample(𝐒(lo))]Ts×2Fs. This concatenation-based conditioning allows the super-resolution model to use the base spectrogram as a structural guide while adding high-frequency detail.

Proposition 27 (Cascaded Generation: Error Propagation).

Let 𝐒^(lo) be the base model's output and 𝐒^ be the super-resolution model's output. Define the error at each stage as ebase=𝐒^(lo)𝐒(lo)F,esr=𝐒^𝐒F, where 𝐒(lo) and 𝐒 are the ground-truth spectrograms at each resolution. If the super-resolution model has Lipschitz constant Lsr with respect to its conditioning input, then (Cascaded Error Bound)esresr(ideal)+Lsrebase, where esr(ideal) is the error of the super-resolution model when given the ground-truth base spectrogram. Errors from the base model thus propagate multiplicatively through the cascade.

Proof.

By the triangle inequality and the Lipschitz property: esr=𝐒^(𝐒^(lo))𝐒F𝐒^(𝐒(lo))𝐒F+𝐒^(𝐒^(lo))𝐒^(𝐒(lo))Fesr(ideal)+Lsr𝐒^(lo)𝐒(lo)F=esr(ideal)+Lsrebase.

Caution.

Cascaded architectures inherit a fundamental fragility: errors in the base model are amplified by the super-resolution stage. If the base model generates spectrogram content with the wrong temporal structure (e.g., placing a drum hit at the wrong position), the super-resolution model will faithfully add fine detail to this incorrect structure, producing a high-resolution but structurally wrong output. This is why modern systems increasingly prefer single-stage latent diffusion (AudioLDM) or DiT-based approaches (Stable Audio) over cascades.

Moûsai: Efficient Diffusion for Long Audio

Moûsai (Schneider et al., 2023) addresses the challenge of generating long music clips (up to 48 seconds) with limited computational resources. Its key insight is that a carefully designed 1D U-Net operating on waveform-domain latents can achieve competitive quality with far fewer parameters than spectrogram-based alternatives.

Cascaded 1D U-Net.

Moûsai uses a two-stage cascade similar to Noise2Music, but with important differences. Both stages operate on 1D latent sequences obtained from an EnCodec-style neural codec. The base model generates latent tokens at a reduced rate (approximately 3 Hz), and the upsampler increases this to the full codec rate (50 Hz).

The 1D U-Net replaces the 2D convolutions of spectrogram-based models with 1D temporal convolutions: (Mousai CONV)𝒉(+1)[t]=σ(k=KK𝐖k()𝒉()[t+k]+𝒃()+γ(tdiff)LN(𝒉()[t])), where 𝐖k() are the convolution weights at level with kernel half-width K, σ is a nonlinearity, and γ(tdiff) is the timestep-dependent scale from adaptive normalisation.

Proposition 28 (Parameter Efficiency of 1D vs. 2D U-Net).

Consider a U-Net with L levels, each containing R residual blocks with channel width C and kernel size K.

  1. A 2D U-Net (for spectrograms) has convolutional weight tensors of shape C×C×K×K per layer, yielding O(LRC2K2) total convolution parameters.

  2. A 1D U-Net (for waveform latents) has convolutional weight tensors of shape C×C×K per layer, yielding O(LRC2K) total convolution parameters.

For typical kernel sizes (K=3 for 2D, K=7 for 1D), the 1D U-Net uses approximately K2D2/K1D=9/71.3× fewer convolution parameters, but the real savings come from the reduced spatial dimensions: the 1D latent sequence is typically 4 to 8 times shorter than the 2D spectrogram along its non-channel dimensions.

Proof.

The convolution parameter count follows directly from the weight tensor shapes. For a 2D convolution with C input and C output channels and kernel size K×K, the weight tensor has C2K2 entries. For a 1D convolution with the same channel count and kernel size K, the weight tensor has C2K entries. Summing over L levels and R blocks per level gives the stated scaling.

Remark 27.

Moûsai demonstrates that it is possible to generate up to 48 seconds of music at 48,kHz with a model of only 860M parameters, training on 8 A100 GPUs for 7 days. By comparison, AudioLDM uses a comparable parameter count but generates at most 10 seconds at 16,kHz, and Stable Audio requires a 1.1B parameter model trained on significantly more compute to achieve 95-second generation at 44.1,kHz. The efficiency of Moûsai stems from its 1D convolutional architecture, which avoids the quadratic cost of self-attention over long sequences.

Comparison of four audio diffusion architectures. AudioLDM uses a 2D U-Net on mel-spectrogram latents; Stable Audio uses a DiT on waveform latents with timing conditioning; Noise2Music cascades U-Nets in spectrogram space; Moûsai uses an efficient 1D U-Net on codec latents.

Diffusion in Audio Latent Spaces

The previous two sections described the architectures and systems that instantiate latent audio diffusion. This section examines the diffusion process itself in detail: the forward noising process that corrupts audio latents, the noise schedules that control corruption speed, and the alternative parameterisations of the denoising objective that affect training dynamics and generation quality. While the mathematical framework follows the general theory developed in ch:diffusion, the audio domain introduces specific considerations related to the structure of audio latent spaces, the multi-scale nature of audio signals, and the interaction between noise schedules and perceptual quality.

The Forward Process in Audio Latent Space

Let 𝒛0Tz×Cz be a clean audio latent obtained from a pretrained encoder . The forward diffusion process is a Markov chain that progressively adds Gaussian noise: (Forward)q(𝒛t|𝒛t1)=Normal(𝒛t;1βt𝒛t1,βt𝐈),t=1,,T, where {βt}t=1T is the noise schedule with βt(0,1). Using the reparametrisation trick, we can sample 𝒛t directly from 𝒛0 without iterating through the chain: (Forward Closed)q(𝒛t|𝒛0)=Normal(𝒛t;αt𝒛0,(1αt)𝐈), where αt=s=1t(1βs) is the cumulative signal retention coefficient. Equivalently, (Forward Reparam)𝒛t=αt𝒛0+1αt𝝐,𝝐Normal(0,𝐈).

Definition 37 (Signal-to-Noise Ratio for Audio Latents).

The signal-to-noise ratio (SNR) at diffusion timestep t for the forward process in (Forward Closed) is (SNR)SNR(t)=αt1αt. The SNR decreases monotonically from SNR(0)= (clean signal) to SNR(T)0 (pure noise). In log-space, (LOG SNR)logSNR(t)=logαtlog(1αt).

The SNR is the central quantity governing the diffusion process. It determines how much signal information remains at each timestep and directly controls the difficulty of the denoising task. For audio latent spaces, the choice of noise schedule (i.e., how the SNR decreases from to 0) has a profound impact on generation quality.

Noise Schedules for Audio

The noise schedule {βt}t=1T (or equivalently, the function α(t) in continuous time) determines the rate at which the signal is corrupted. The general theory of noise schedules is developed in sec:vdiff:noise-schedules; here we focus on the specific considerations that arise when the diffusion process operates in an audio latent space.

Definition 38 (Common Noise Schedules).

Three noise schedules commonly used in audio diffusion:

  1. Linear schedule (Ho et al., 2020): βt=βmin+(βmaxβmin)t/T, where βmin=104 and βmax=0.02.

  2. Cosine schedule (Nichol and Dhariwal, 2021): (Cosine Schedule)αt=f(t)f(0),f(t)=cos2(t/T+s1+sπ2), where s=0.008 is a small offset to prevent αT from reaching exactly zero.

  3. Sigmoid schedule: (Sigmoid Schedule)logSNR(t)=log(exp(γmin+t(γmaxγmin))1), where γmin and γmax control the range of the log-SNR.

Proposition 29 (Cosine Schedule Preserves Information Longer).

Let t=inf{t:SNR(t)1} be the timestep at which the noise energy equals the signal energy. For the linear schedule with T=1000 and the parameters above, tlin550. For the cosine schedule, tcos750. Therefore, the cosine schedule preserves SNR(t)>1 (more signal than noise) for approximately 36% more timesteps than the linear schedule.

Proof.

For the linear schedule: αt=s=1t(1βs). Taking logarithms, logαt=s=1tlog(1βs)s=1tβs for small βs. With βs=104+(0.02104)s/1000, the sum s=1tβs is a quadratic function of t. Setting SNR(t)=1 requires αt=1/2, i.e., logαt=log2. Numerical evaluation gives tlin550.

For the cosine schedule: αt=cos2(πt/(2T)) (ignoring the offset s for simplicity). Setting αt=1/2 gives cos2(πt/(2T))=1/2, so πt/(2T)=π/4, yielding t=T/2=500. With the offset s=0.008, the exact value shifts to approximately tcos750 due to the compressed schedule. The ratio is (750550)/5500.36, confirming the 36% improvement.

Remark 28.

Audio latent spaces differ from image latent spaces in their distributional properties. Audio latents tend to have:

  1. Higher dynamic range: loudness variations across a clip create latent dimensions with very different variances, meaning some dimensions are corrupted much faster than others under an isotropic noise schedule.

  2. Temporal correlations: adjacent latent frames are highly correlated (music changes slowly relative to the latent frame rate), creating low-frequency structure that requires more noise to obscure than the high-frequency details.

  3. Sparser spectra: at any given moment, only a few frequency bands are active (e.g., a single instrument playing), concentrating the signal energy in a low-dimensional subspace of the latent space.

These properties suggest that audio diffusion may benefit from noise schedules that spend more time in the high-SNR regime (preserving coarse temporal structure longer) compared to image diffusion. The cosine schedule partially addresses this, but systems like Stable Audio use custom sigmoid schedules tuned specifically for their waveform latent spaces.

Parameterisation: 𝝐, 𝒙0, and 𝒗epsilon, x0, and v

The denoising network 𝝐θ can be trained to predict different targets, each corresponding to a different mathematical parameterisation of the reverse process. While these parameterisations are theoretically equivalent (they all define the same conditional distribution pθ(𝒛t1|𝒛t)), they differ significantly in training dynamics, gradient magnitudes, and empirical generation quality.

Definition 39 (Three Parameterisations).

Given the noisy latent 𝒛t=αt𝒛0+1αt𝝐, the denoising network can be trained to predict:

  1. Noise prediction (𝝐-parameterisation): the network 𝝐^θ(𝒛t,t,𝒄) predicts the noise 𝝐, with loss (EPS Param)𝝐=𝔼[𝝐𝝐^θ(𝒛t,t,𝒄)22].

  2. Data prediction (𝒙0-parameterisation): the network 𝒛^0,θ(𝒛t,t,𝒄) predicts the clean latent 𝒛0, with loss (X0 Param)𝒙0=𝔼[𝒛0𝒛^0,θ(𝒛t,t,𝒄)22].

  3. Velocity prediction (𝒗-parameterisation, Salimans and Ho, 2022): define the velocity (Velocity)𝒗t=αt𝝐1αt𝒛0, and train the network 𝒗^θ(𝒛t,t,𝒄) to predict 𝒗t, with loss (V Param)𝒗=𝔼[𝒗t𝒗^θ(𝒛t,t,𝒄)22].

The three parameterisations are related by linear transformations. Given any one prediction, the other two can be recovered: (Param Conversion 1)𝒛0=1αt(𝒛t1αt𝝐),𝝐=11αt(𝒛tαt𝒛0),𝒗=αt𝝐1αt𝒛0=𝒛t2αt𝒛01αt.

Proposition 30 (Implicit Loss Weighting).

The three parameterisations induce different implicit weightings on the denoising signal-to-noise ratio. Specifically, under the 𝝐-parameterisation, the effective loss weight at timestep t (expressed in terms of the SNR) is (EPS Weight)w𝝐(t)=SNR(t), meaning that the 𝝐-loss emphasises high-SNR (low-noise) timesteps. Under the 𝒙0-parameterisation, (X0 Weight)w𝒙0(t)=1SNR(t), emphasising low-SNR (high-noise) timesteps. The 𝒗-parameterisation produces a balanced weighting: (V Weight)w𝒗(t)=1, giving equal weight to all timesteps regardless of SNR\@.

Proof.

We express each loss in terms of a common prediction target. The 𝝐-loss can be rewritten in terms of the 𝒛0 prediction error. From (Param Conversion 1), 𝝐=(𝒛tαt𝒛0)/1αt, so 𝝐=𝔼[𝝐𝝐^θ22]=𝔼[αt1αt𝒛0𝒛^0,θ22]=𝔼[SNR(t)𝒛0𝒛^0,θ22]. Similarly, the 𝒙0-loss has implicit weight 1 in terms of 𝒛0 prediction, which corresponds to weight 1/SNR(t) when expressed in terms of 𝝐 prediction. For the 𝒗-parameterisation: 𝒗𝒗^θ22=(αt𝝐1αt𝒛0)𝒗^θ22=αt𝝐𝝐^θ22+(1αt)𝒛0𝒛^0,θ22+cross terms. The cross terms vanish in expectation because 𝝐 and 𝒛0 are independent. The effective weight on the 𝒛0 prediction error is αtSNR(t)+(1αt)=αtαt/(1αt)+(1αt)=1 after simplification, confirming uniform weighting.

Key Idea.

The 𝒗-parameterisation is the default for audio. The uniform weighting of the 𝒗-parameterisation is particularly advantageous for audio latent diffusion. Audio signals have important perceptual content at all noise levels: at high SNR, the model must capture fine timbral details (instrument identity, room acoustics); at low SNR, it must establish coarse temporal structure (rhythm, harmony, section boundaries). The 𝝐-parameterisation's emphasis on high-SNR timesteps tends to produce outputs with correct fine detail but poor global structure, while the 𝒙0-parameterisation tends to produce outputs with good structure but blurry detail. The 𝒗-parameterisation balances both regimes. Both AudioLDM2 and Stable Audio use 𝒗-parameterisation.

Example 18 (Gradient Magnitudes Across Parameterisations).

Consider an audio latent with 𝒛02=1 and 𝝐2=1 (unit normalised). At a high-SNR timestep (t=100, αt=0.95, SNR=19):

  • 𝝐-loss gradient magnitude: SNR(t)1/2=194.4.

  • 𝒙0-loss gradient magnitude: SNR(t)1/2=1/190.23.

  • 𝒗-loss gradient magnitude: 1.

At a low-SNR timestep (t=900, αt=0.05, SNR=0.053):

  • 𝝐-loss gradient magnitude: 0.0530.23.

  • 𝒙0-loss gradient magnitude: 1/0.0534.4.

  • 𝒗-loss gradient magnitude: 1.

The 𝒗-parameterisation maintains constant gradient magnitude across the entire diffusion trajectory, preventing the training instabilities that arise when gradient magnitudes vary by a factor of 20.

Training Audio Diffusion Models

The preceding sections described what audio diffusion models predict (Section Diffusion in Audio Latent Spaces) and the architectures that implement these predictions (Section Diffusion Architectures for Audio). This section addresses the equally important question of how these models are trained. Training an audio diffusion model is considerably more than minimising the squared-error loss from (EPS Param): in practice, systems employ composite losses that combine multiple objectives, face data challenges unique to the audio domain, and require careful stabilisation techniques to prevent training collapse. We develop each of these topics in turn.

Composite Training Losses

While the core diffusion objective ((Audioldm LOSS)) provides the primary training signal, high-quality audio generation systems augment it with auxiliary losses that encode perceptual and structural priors about audio.

Definition 40 (Composite Audio Diffusion Loss).

The composite audio diffusion training loss combines the diffusion denoising objective with auxiliary terms: (Composite LOSS)total=diffdenoising+λstftMR-STFTspectral+λclapCLAPsemantic+λadvadvadversarial, where:

  1. diff is the denoising loss (𝝐, 𝒙0, or 𝒗-parameterised) from Definition 39.

  2. MR-STFT is the multi-resolution STFT loss, computed on the decoded audio 𝒙^=𝒱(𝒟(𝒛^0)): (Mrstft Training)MR-STFT=s𝒮(|STFTs(𝒙)||STFTs(𝒙^)|F+log|STFTs(𝒙)|log|STFTs(𝒙^)|1), where 𝒮 denotes a set of STFT configurations (window sizes, hop lengths), and the sum over scales captures both magnitude and log-magnitude differences (see HiFi-GAN Training Losses).

  3. CLAP is a CLAP-based semantic alignment loss: (CLAP Training LOSS)CLAP=1fθ(𝐒^),gϕ(𝒄), where fθ is the CLAP audio encoder, gϕ is the CLAP text encoder, and 𝐒^ is the generated mel spectrogram. This loss encourages the generated audio to be semantically aligned with the conditioning text.

  4. adv is an adversarial loss from a discriminator that distinguishes real from generated spectrograms, following the GAN training paradigm.

The weighting coefficients λstft,λclap,λadv control the relative importance of each term.

Remark 29.

Not all systems use all four loss terms. AudioLDM uses only diff for the diffusion model (the multi-resolution STFT loss is used only for VAE training). Stable Audio adds the multi-resolution STFT loss during fine-tuning. The CLAP alignment loss is primarily used in evaluation (CLAP Score) but some recent systems incorporate it during training as a perceptual guidance signal. The adversarial loss is most common in GAN-enhanced diffusion systems such as those used in speech synthesis.

Multi-resolution STFT loss in detail.

The multi-resolution STFT loss is particularly important for audio because it operates directly in the time-frequency domain where human perception is most sensitive. By summing over multiple STFT configurations, it captures perceptual quality at different temporal and spectral resolutions.

Proposition 31 (Multi-Resolution STFT Loss Properties).

The multi-resolution STFT loss MR-STFT as defined in (Mrstft Training) has the following properties:

  1. Non-negativity: MR-STFT0, with equality if and only if |STFTs(𝒙)|=|STFTs(𝒙^)| for all scales s𝒮.

  2. Phase invariance: MR-STFT depends only on the magnitude spectrograms, not on the phase. Two signals with identical magnitude spectra but different phases will have zero loss.

  3. Multi-scale sensitivity: For a set 𝒮={(W1,H1),,(WS,HS)} of (window size, hop length) pairs, the loss is sensitive to errors at temporal scale Hs/fs seconds and frequency resolution fs/Ws Hz for each s. Typical choices span W{256,512,1024,2048,4096} samples.

Proof.

Property (i) follows from the non-negativity of norms. The Frobenius norm F and L1 norm 1 are both non-negative, with equality iff their arguments are zero matrices. Property (ii) holds because |STFTs(𝒙)| discards the phase: if STFTs(𝒙)[m,k]=Amkeiϕmk, then |STFTs(𝒙)[m,k]|=Amk, which is independent of ϕmk. Property (iii) follows from the time-frequency uncertainty principle: a window of size W provides frequency resolution Δf=fs/W and temporal resolution Δt=H/fs (where H is the hop length). Different (W,H) pairs therefore capture errors at different time-frequency trade-offs.

Caution.

The phase invariance of the multi-resolution STFT loss (Property (ii)) means that it cannot detect phase-related artefacts such as phasiness or comb filtering. In practice, this is mitigated by the vocoder, which predicts phase from the magnitude spectrogram. However, for waveform-domain systems (like Stable Audio) that bypass the spectrogram representation, the multi-resolution STFT loss should be complemented with a waveform-domain loss (e.g., L1 loss on the raw waveform) to capture phase-dependent distortions.

Data Challenges in Audio Diffusion

Training audio diffusion models confronts several data challenges that do not arise (or are less severe) in image generation.

Scarcity of paired data.

The largest paired text-to-image datasets contain billions of image-caption pairs (e.g., LAION-5B). By contrast, the largest public audio-caption datasets contain at most hundreds of thousands of pairs. AudioCaps contains approximately 46,000 10-second clips with human-written captions; Clotho contains approximately 5,000 clips. Even with data augmentation (e.g., MusicLDM's beat-synchronous mixup from Definition 34), the total training data for audio models is orders of magnitude smaller than for image models.

Proposition 32 (Sample Complexity Gap).

Let dimg and daud be the intrinsic dimensionalities of the image and audio latent data manifolds, respectively. Under standard statistical learning theory, the number of samples N required to learn a distribution to ϵ-accuracy (in Wasserstein distance) scales as (Sample Complexity)N=Ω(ϵd/2), where d is the intrinsic dimension of the data manifold. If audio latent spaces have comparable or higher intrinsic dimension to image latent spaces (dauddimg), then audio models require at least as many samples as image models to achieve the same distributional fidelity, yet they have access to 3 to 4 orders of magnitude fewer paired training examples.

Proof.

The lower bound N=Ω(ϵd/2) follows from the minimax rate for density estimation on d-dimensional manifolds, as established by Liang (2021) and Kim and Ye (2024) for score-based generative models. The intrinsic dimension of audio latent spaces is typically estimated at d50 to 100 (from the effective rank of the latent covariance matrix), comparable to the d50 to 100 range observed for image latent spaces in models like Stable Diffusion. The data ratio Naud/Nimg105/109=104 therefore represents a severe deficiency.

Variable audio lengths and silence.

Audio clips in training datasets have highly variable durations (from 1 second to several minutes), and many clips contain significant silence or low-energy segments. Naive training on fixed-length crops can result in a large fraction of training examples being predominantly silent, wasting model capacity on learning the distribution of silence.

Definition 41 (Energy-Based Crop Selection).

Given an audio clip 𝒙L and a target crop length Lc<L, the energy-based crop selection procedure:

  1. Compute frame-level energy: E[m]=n=mH(m+1)H1x[n]2, where H is the frame hop.

  2. Compute a running average energy over the crop window: E[m]=1Lc/Hj=0Lc/H1E[m+j].

  3. Sample the crop start position from a distribution proportional to the running energy: (Energy CROP)mCategorical(E[m]γmE[m]γ), where γ>0 controls the concentration toward high-energy regions (γ=0 recovers uniform random cropping; γ always selects the loudest segment).

Label noise in audio captions.

Audio captions are notoriously noisy. A clip described as “a cat meowing” may also contain background traffic, air conditioning hum, and room reverberation not mentioned in the caption. This label noise directly affects the diffusion model's ability to learn accurate text-to-audio correspondences.

Remark 30.

Several strategies mitigate caption noise:

  1. LLM-based caption enrichment: Use a large language model to expand terse captions (e.g., “cat meowing” becomes “a domestic cat meowing repeatedly in a quiet indoor environment with faint background noise”). AudioLDM2 uses this strategy with GPT-3.5.

  2. Audio tagging augmentation: Run a pretrained audio tagger (e.g., PANNs) to detect additional sound events, then append the detected tags to the caption.

  3. CLAP-based filtering: Compute the CLAP score between each audio-caption pair and discard pairs with scores below a threshold, removing the most misaligned examples.

Training Stability and Loss Balancing

When training with composite losses (Definition 40), the relative magnitudes of the different loss terms can vary by several orders of magnitude, leading to training instability. The diffusion denoising loss diff may have typical values in [0.01,0.1], while the multi-resolution STFT loss MR-STFT can range from [1,100] depending on the STFT configuration. Without careful balancing, one loss term dominates the gradient signal, and the model effectively ignores the others.

Definition 42 (Gradient-Based Loss Balancer).

A gradient-based loss balancer dynamically adjusts the weighting coefficients λi in the composite loss to equalise the gradient magnitudes of each loss term. Let i be the i-th loss term and 𝒈i=θi its gradient with respect to the model parameters θ. The balanced loss is (Balancer)balanced=i=1Kwi𝒈i2+δi, where wi>0 are target relative weights (reflecting the desired importance of each loss term), δ>0 is a small constant for numerical stability, and 𝒈i2 is the L2 norm of the gradient of i with respect to a reference layer (typically the last shared layer before the output heads).

Proposition 33 (Gradient Equalisation Property).

Under the balanced loss from Definition 42 (with δ=0 for simplicity), the effective gradient contribution of each loss term satisfies (GRAD Equal)wi𝒈i2𝒈i2=wi for all i. Therefore, if all target weights are equal (wi=1/K), the gradient contributions have equal magnitude regardless of the absolute scale of each loss.

Proof.

Direct computation: (wi/𝒈i2)𝒈i2=(wi/𝒈i2)𝒈i2=wi.

Remark 31.

The gradient-based loss balancer was popularised by the EnCodec system (Défossez et al., 2023), where it is used to balance the reconstruction loss, VQ commitment loss, and adversarial discriminator losses during neural codec training. The same technique transfers directly to audio diffusion training when composite losses are employed. The balancer operates with exponential moving averages of gradient norms to prevent oscillatory behaviour: (Balancer EMA)g^i(k)=βg^i(k1)+(1β)𝒈i(k)2, where β=0.999 is the EMA coefficient and k indexes the training step. The smoothed estimates g^i(k) replace 𝒈i2 in (Balancer).

Learning rate warmup and scheduling.

Audio diffusion models are sensitive to the learning rate schedule, particularly during the early stages of training when the model's predictions are far from the target distribution. A common recipe is:

  1. Linear warmup over the first 1,000 to 5,000 steps, ramping the learning rate from 0 to ηmax=104.

  2. Cosine decay from ηmax to ηmin=106 over the remaining training steps.

  3. EMA of model weights: maintain an exponential moving average of the model parameters with decay rate μ=0.9999, and use the EMA weights for evaluation and inference.

The EMA weights produce smoother, higher-quality outputs than the raw training weights, particularly for music generation where subtle timbral consistency is important.

The Complete Training Algorithm

We now consolidate the components from the preceding subsections into a complete training algorithm for latent audio diffusion.

Algorithm 4 (Latent Audio Diffusion Training).

Input: Training dataset 𝒟={(𝒙i,𝒄i)} of audio-caption pairs; pretrained and frozen audio VAE (,𝒟); pretrained and frozen text encoder gϕ; noise schedule {αt}t=0T; learning rate schedule η(k); EMA decay μ; loss balancer weights {wi}; parameterisation choice 𝒫{𝝐,𝒙0,𝒗}. Output: Trained denoiser 𝝐θ (EMA weights θema).

  1. Initialise: Randomly initialise θ; set θemaθ; set step counter k0.

  2. repeat (until convergence or budget exhausted): enumerate[label=., leftmargin=1.5em]

  3. Sample a mini-batch {(𝒙j,𝒄j)}j=1B from 𝒟.

  4. Encode audio: For each j, compute 𝒛0(j)=(𝐒j), where 𝐒j is the mel spectrogram of 𝒙j (or 𝒛0(j)=wav(𝒙j) for waveform VAEs).

  5. Encode text: For each j, compute conditioning embedding 𝒄^j=gϕ(𝒄j). With probability pdrop (typically 0.1), replace 𝒄^j with the null embedding 𝒄 for classifier-free guidance training.

  6. Sample noise and timestep: For each j, sample 𝝐jNormal(0,𝐈) and tj𝒰{1,,T}.

  7. Construct noisy latent: 𝒛tj(j)=αtj𝒛0(j)+1αtj𝝐j.

  8. Compute denoising prediction: 𝒚^j=𝝐θ(𝒛tj(j),tj,𝒄^j).

  9. Compute target: Based on 𝒫: itemize

  10. If 𝒫=𝝐: 𝒚j=𝝐j.

  11. If 𝒫=𝒙0: 𝒚j=𝒛0(j).

  12. If 𝒫=𝒗: 𝒚j=αtj𝝐j1αtj𝒛0(j). itemize

  13. Compute diffusion loss: diff=1Bj=1B𝒚j𝒚^j22.

  14. Compute auxiliary losses: MR-STFT, CLAP, etc., if applicable (requires decoding 𝒛^0 at selected training steps).

  15. Balance and combine: Apply the gradient balancer (Definition 42) to obtain total.

  16. Update parameters: θθη(k)θtotal.

  17. Update EMA: θemaμθema+(1μ)θ.

  18. Increment: kk+1. enumerate

  19. Return θema.

Remark 32.

Several practical considerations supplement the formal algorithm:

  1. Mixed precision: The U-Net or DiT is typically trained in bf16 (brain floating point) precision, with the loss computation and optimiser states in fp32. This halves the memory footprint and accelerates training by approximately 1.5× on modern GPUs.

  2. Gradient accumulation: For large batch sizes (B256) that exceed GPU memory, gradients are accumulated over B/Bmicro micro-batches before each optimiser step.

  3. Distributed training: Audio diffusion models are typically trained across 8 to 64 GPUs using distributed data parallelism (DDP), with all-reduce for gradient synchronisation.

  4. Validation metrics: Fréchet Audio Distance (FAD), CLAP score, and Inception Score are computed on a held-out set every 5,000 to 10,000 steps to monitor training progress.

Example 19 (Training Budget Estimation).

Consider training an AudioLDM-scale model (approximately 400M parameters) on 10,000 hours of audio with captions.

  • Data: 10,000 hours at 10-second crops yields 3.6×106 training examples per epoch.

  • Batch size: B=256 (accumulated over 8 GPUs with micro-batch size 32).

  • Steps per epoch: 3.6×106/25614,000 steps.

  • Training duration: 200 epochs =2.8×106 steps.

  • Time per step: Approximately 0.3 seconds on 8×A100 (including forward, backward, and all-reduce).

  • Total wall-clock time: 2.8×106×0.3/3600233 hours 10 days.

  • GPU hours: 233×8=1,864 A100-hours.

This is roughly 100× cheaper than training a comparable image diffusion model (e.g., Stable Diffusion required approximately 200,000 A100-hours), reflecting both the smaller dataset size and the lower-dimensional latent space.

Proposition 34 (Convergence Rate of Latent vs. Spectrogram Diffusion).

Let ϵlat(k) and ϵspec(k) denote the expected denoising errors after k training steps for latent-space and spectrogram-space diffusion models, respectively, trained with the same learning rate and batch size. Under the assumption that both models use architectures with comparable effective capacity (number of learnable parameters per latent dimension), the latent-space model converges faster: (Convergence Bound)ϵlat(k)ϵspec(k)dlatdspec, where dlat=TzCz and dspec=TsFs are the dimensionalities of the latent and spectrogram spaces. For dlat/dspec1/32, latent diffusion converges approximately 325.7× faster per training step.

Proof.

This follows from the standard convergence analysis of stochastic gradient descent on quadratic objectives. The denoising loss is approximately quadratic near the optimum, and the convergence rate of SGD on a d-dimensional quadratic is ϵ(k)d/k (from the bias-variance decomposition of the gradient estimator). For the same number of steps k, the ratio is ϵlat(k)/ϵspec(k)=dlat/dspec.

Classifier-Free Guidance for Audio

Classifier-free guidance (CFG), first introduced in image diffusion and studied in Classifier-Free Guidance for Audio, is essential for high-quality audio generation. Here we examine the specific training and inference considerations for CFG in the audio setting.

During training, the conditioning embedding 𝒄^j is replaced with a null embedding 𝒄 with probability pdrop. This trains the model to produce both conditional and unconditional outputs. At inference, the guided prediction is (CFG Inference)𝒚^guided=(1+w)𝒚^(𝒛t,t,𝒄)w𝒚^(𝒛t,t,𝒄), where w0 is the guidance scale.

Proposition 35 (CFG as Implicit Score Sharpening).

The classifier-free guided prediction in (CFG Inference) corresponds to sampling from a sharpened conditional distribution. Specifically, under the 𝝐-parameterisation, the guided score function is (CFG Score)𝒛tlogpguided(𝒛t|𝒄)=(1+w)𝒛tlogp(𝒛t|𝒄)w𝒛tlogp(𝒛t), which corresponds to the score of the distribution pguided(𝒛t|𝒄)p(𝒛t|𝒄)(p(𝒛t|𝒄)/p(𝒛t))w. For w>0, this distribution is more concentrated around the conditional modes than p(𝒛t|𝒄) itself, effectively sharpening the conditional distribution and producing outputs that are more strongly aligned with the conditioning text.

Proof.

Taking the gradient of logpguided: 𝒛tlogpguided(𝒛t|𝒄)=𝒛tlog[p(𝒛t|𝒄)(p(𝒛t|𝒄)/p(𝒛t))w]=𝒛t[(1+w)logp(𝒛t|𝒄)wlogp(𝒛t)]=(1+w)𝒛tlogp(𝒛t|𝒄)w𝒛tlogp(𝒛t). This matches (CFG Score), confirming the correspondence. The sharpening effect follows because the likelihood ratio p(𝒛t|𝒄)/p(𝒛t) is highest in regions where the conditioning is most informative, and raising this ratio to the power w>0 amplifies these peaks.

Remark 33.

The optimal guidance scale w varies significantly across audio generation tasks:

  • Sound effects: w[2.0,5.0]. Sound effects are typically well-described by their captions (e.g., “a door slamming shut”), so moderate guidance suffices.

  • Music: w[3.0,7.0]. Music captions are often vague (e.g., “an upbeat pop song”), requiring stronger guidance to produce outputs that match the described genre and mood.

  • Speech: w[1.0,3.0]. Speech has strong intrinsic structure (phonemes, prosody), and excessive guidance can produce robotic, over-articulated outputs.

A common diagnostic is to listen to outputs at several guidance scales and select the one that balances text adherence against naturalness. Too high a guidance scale produces “over-saturated” audio analogous to the over-saturated images produced by high-CFG image generation.

Historical Note.

The application of latent diffusion to audio followed the image domain by approximately one year. Stable Diffusion (Rombach et al., 2022) demonstrated latent diffusion for images in late 2022; AudioLDM (Liu et al., 2023) applied the same framework to audio in early 2023. This timeline reflects a recurring pattern in generative modelling: techniques are first developed for images (which benefit from large datasets and well-understood evaluation metrics), then adapted to audio (where data is scarcer but the mathematical framework transfers directly). The adaptation is not merely engineering: audio-specific innovations such as beat-synchronous mixup (MusicLDM), timing conditioning (Stable Audio), and the LOA framework (AudioLDM2) demonstrate that the audio domain demands its own theoretical contributions.

Exercises

Exercise 11 (Latent Compression and Generation Quality).

Consider an audio VAE with encoder :Ts×FsTz×Cz and compression ratio r=(TsFs)/(TzCz).

  1. For a 10-second audio clip at 16,kHz with mel-spectrogram parameters Ts=1000, Fs=64, compute the latent dimensionality for compression ratios r{4,16,32,64}. Assuming Cz=8 in all cases, what is Tz for each?

  2. The VAE reconstruction error (measured by multi-resolution STFT loss) increases approximately as recon(r)αrβ for constants α,β>0. If the error doubles when going from r=16 to r=32, compute β. What reconstruction error would you predict at r=64?

  3. Argue that the optimal compression ratio r balances two competing effects: (i) the diffusion model's sample quality improves as r increases (easier denoising problem), and (ii) the autoencoder's reconstruction quality degrades as r increases. Sketch a qualitative plot of total generation quality (e.g., FAD) as a function of r, and argue that a minimum exists.

  4. (Challenging.) Let H(𝒙|𝒛0) denote the conditional entropy of the waveform given the latent code. Show that the total information loss of the three-stage pipeline is bounded by H(𝒙|𝒙^)H(𝒙|𝒛0)+H(𝒛0|𝒛^0)+H(𝒛^0|𝐒^)+H(𝐒^|𝒙^), where each term corresponds to the information lost at one stage of the pipeline. Which terms depend on the compression ratio?

Exercise 12 (Noise Schedule Design for Audio).

Consider the cosine noise schedule from (Cosine Schedule) with offset s=0.008 and T=1000 discrete steps.

  1. Compute αt for t{0,250,500,750,1000} and the corresponding SNR(t) values. At which timestep is SNR(t)1 (equal signal and noise energy)?

  2. Now consider the linear schedule with βmin=104 and βmax=0.02. Compute αt for the same timesteps. Compare the rate of information destruction between the two schedules.

  3. Design a “music-optimised” noise schedule that spends 60% of the timesteps in the range SNR(t)>1 (preserving global structure) and 40% in the range SNR(t)1 (refining details). Express your schedule as a function α(t) and verify that it satisfies the boundary conditions α(0)=1 and α(T)0.

  4. (Research direction.) Propose a method for learning the noise schedule from data, by treating α(t) as a parameterised function and optimising it jointly with the denoising network. What objective should be used? What constraints must the parameterisation satisfy to ensure a valid noise schedule?

Exercise 13 (Parameterisation Equivalence and Loss Weighting).

Let 𝒛t=αt𝒛0+1αt𝝐 with 𝝐Normal(0,𝐈).

  1. Starting from the 𝝐-prediction loss 𝝐=𝔼[𝝐𝝐^θ22], derive the equivalent 𝒙0-prediction loss with its implicit SNR weighting. Show that 𝝐=𝔼[SNR(t)𝒛0𝒛^0,θ22].

  2. Show that the 𝒗-parameterisation target 𝒗=αt𝝐1αt𝒛0 can be written as 𝒗=(𝒛t2αt𝒛0)/1αt, and verify that 𝒗22=𝝐22+𝒛0222αt(1αt)𝝐,𝒛0.

  3. Consider a hypothetical “SNR-balanced” parameterisation with loss bal=𝔼[SNR(t)1/2𝝐𝝐^θ22]. Show that this is equivalent to the 𝒗-loss up to a constant factor, confirming that 𝒗-parameterisation achieves balanced weighting.

  4. For a 1-dimensional latent (𝒛0,𝝐) and the linear schedule, numerically compute and plot the loss weighting functions w𝝐(t), w𝒙0(t), and w𝒗(t) for t=1,,1000. Verify that w𝒗(t)1 for all t.

Exercise 14 (Cascaded vs. Single-Stage Audio Diffusion).

Consider a cascaded system (Noise2Music-style) with a base model 𝝐θ(base) and a super-resolution model 𝝐ϕ(sr), and a single-stage latent system (AudioLDM-style) with model 𝝐ψ.

  1. The base model generates a 64×64 mel spectrogram and the SR model upsamples to 1024×64. Both use T=200 diffusion steps. If both models have the same architecture (same number of parameters), but the base model operates on 16× fewer spatial dimensions, estimate the ratio of total inference FLOPs for the cascaded system vs. a single AudioLDM model operating on 256×8 latent space.

  2. Using the error propagation bound from Proposition 27, suppose the base model achieves ebase=0.1 and the SR model has Lipschitz constant Lsr=2.0 with ideal SR error esr(ideal)=0.05. Compute the total cascaded error. How does this compare to a hypothetical single-stage error of esingle=0.15?

  3. Argue that the cascaded approach has an advantage for controllability: one can swap the base model (changing the coarse structure) while keeping the SR model fixed. Give a concrete example of how this modularity could be exploited in a music generation application.

Flow Matching for Music Generation

The diffusion models studied in the preceding sections learn to reverse a stochastic noising process: training teaches a network to predict the noise added at each timestep, and generation proceeds by iteratively denoising a sample drawn from pure Gaussian noise. While this framework has produced remarkable audio and music generation systems, it carries an inherent inefficiency. The stochastic trajectories connecting noise to data are highly curved in the latent space, requiring many small denoising steps for accurate traversal. Flow matching, introduced in ch:flows and connected to optimal transport in ch:sinkhorn, offers a compelling alternative: learn a deterministic velocity field that transports noise to data along straighter paths, enabling fewer integration steps and simpler training objectives.

In this section we develop the theory and practice of flow matching for audio and music generation. We begin by recalling the essential ideas from continuous normalising flows and explaining why straighter paths translate directly to faster audio synthesis (From Diffusion to Flow Matching). We then examine two influential systems: MusicFlow, which cascades flow matching models from text to semantic to acoustic tokens (MusicFlow: Cascaded Flow Matching), and MelodyFlow, which adds melody conditioning in a non-autoregressive framework (MelodyFlow: Melody-Conditioned Flow Matching). Finally, we study optimal transport paths tailored to audio latent spaces and the reflow procedure that further straightens learned trajectories (Optimal Transport Paths for Audio).

From Diffusion to Flow Matching

Recall from ch:flows that a continuous normalising flow (CNF) defines a time-dependent velocity field vθ:d×[0,1]d whose integral curves transport a source distribution p0 to a target distribution p1. Given an initial point 𝒛0p0, the generated sample is (CNF ODE)𝒛1=𝒛0+01vθ(𝒛t,t)dt, where 𝒛t follows the ODE d𝒛t/dt=vθ(𝒛t,t). In audio generation, p0=Normal(0,𝐈) is a standard Gaussian over the latent space of an audio codec or spectrogram encoder, and p1 is the distribution of clean audio latents.

The flow matching training objective avoids the expensive simulation required by maximum-likelihood training of CNFs. Instead, it constructs conditional probability paths pt(𝒛|𝒛0,𝒛1) that interpolate between the source and target, and trains vθ to match the conditional velocity field of these paths. The simplest choice is the linear interpolation path: (Linear Interp)𝒛t=(1t)𝒛0+t𝒛1,t[0,1], whose conditional velocity field is simply the displacement 𝒛1𝒛0, independent of t.

Key Idea.

Straighter paths, fewer steps, simpler training. The fundamental advantage of flow matching over diffusion is geometric. Diffusion models follow curved stochastic trajectories because the forward process adds noise incrementally and the reverse process must undo each increment. Flow matching constructs deterministic paths that can be made nearly straight, so that a first-order ODE solver (Euler method) with very few steps approximates the integral in (CNF ODE) accurately. For audio generation, where real-time synthesis demands fast inference, this translates directly to practical speedups.

Remark 34.

Flow matching and diffusion are not entirely distinct frameworks. As shown in ch:flows, the probability flow ODE of a diffusion model defines a deterministic velocity field, and flow matching can be viewed as training this field directly. The key difference is in the choice of paths: diffusion models inherit their paths from the forward noising process (typically variance-preserving or variance-exploding schedules), while flow matching allows the practitioner to choose paths that are geometrically simpler, such as linear interpolations or optimal transport maps.

To make the comparison concrete, consider the number of function evaluations (NFE) needed to generate a 10-second audio clip at acceptable quality. A typical diffusion model requires 50 to 200 NFE with a DDPM-style sampler, or 20 to 50 NFE with an accelerated sampler such as DPM-Solver. A flow matching model with near-linear trajectories can achieve comparable quality with 5 to 20 NFE, representing a 5 to 10× speedup in wall-clock generation time.

Example 20 (NFE comparison for audio generation).

Consider generating 10 seconds of 24,kHz audio using a latent model with temporal compression factor 320 (i.e., 750 latent frames). Each function evaluation requires a forward pass of a transformer with 400M parameters. At half-precision on an A100 GPU, each forward pass takes approximately 15,ms.

table Wall-clock generation time for 10,s of audio at various NFE counts.

MethodNFETime (s)RTF
DDPM2003.000.30
DPM-Solver++300.450.045
Flow matching100.150.015
Rectified flow50.0750.0075
Here, the real-time factor (RTF) is defined as the ratio of generation time to audio duration; RTF <1 indicates faster-than-real-time synthesis. All methods shown achieve RTF 1, but the flow matching variants offer an additional order of magnitude of headroom, which is critical for interactive applications such as live accompaniment or game audio.

Gaussian-to-data paths and the audio latent geometry.

Audio latent spaces exhibit structure that is atypical of natural image latent spaces: strong temporal correlations (adjacent latent frames encode overlapping waveform windows), a near-periodic substructure induced by musical rhythm, and highly non-isotropic marginals arising from the varying energy distribution across frequency bands. These properties have two consequences for flow matching. First, the linear interpolation (Linear Interp) may not be the most efficient path, because the intermediate points 𝒛t for t(0,1) may lie far from the data manifold. Second, the velocity field learned by the flow model inherits the temporal structure of the audio, so transformer architectures with causal or windowed attention along the time axis are natural choices for parameterising vθ.

Definition 43 (Gaussian-preserving interpolation for audio).

A Gaussian-preserving interpolation between p0=Normal(0,𝐈) and p1 with mean 𝝁 and covariance 𝚺 defines the path (Gauss Interp)𝒛t=((1t)𝐈+t𝚺1/2)𝒛0+t𝝁,t[0,1], so that 𝒛tNormal(t𝝁,((1t)𝐈+t𝚺1/2)2) at each time t. This path preserves the Gaussian family of distributions along the trajectory, which can be beneficial when the data distribution is approximately Gaussian in the latent space (as is often the case for well-trained variational autoencoders).

MusicFlow: Cascaded Flow Matching

MusicFlow applies the flow matching framework to music generation through a cascaded architecture that mirrors the semantic-to-acoustic pipeline of earlier diffusion-based systems, but replaces the diffusion backbone with conditional flow matching at each stage. The cascade consists of two stages:

  1. Text Semantic. A flow matching model vθ(sem) maps Gaussian noise to semantic token embeddings, conditioned on a text embedding 𝒄text obtained from a pretrained language model (e.g., T5 or CLAP).

  2. Semantic Acoustic. A second flow matching model vθ(ac) maps Gaussian noise to acoustic latents (e.g., EnCodec or DAC embeddings), conditioned on the semantic tokens generated in stage (i).

Definition 44 (Audio Flow Matching Objective).

Let 𝒛0Normal(0,𝐈) be a noise sample, let 𝒛1pdata be a clean audio latent, and let 𝒄 denote the conditioning signal (text embedding, semantic tokens, or both). Define the interpolant 𝒛t=(1t)𝒛0+t𝒛1 for t[0,1]. The audio flow matching objective is (FM LOSS)FM=𝔼t𝒰(0,1)𝔼𝒛0Normal(0,𝐈)𝔼𝒛1pdata[vθ(𝒛t,t,𝒄)(𝒛1𝒛0)2]. The target 𝒛1𝒛0 is the constant velocity of the linear interpolation path from 𝒛0 to 𝒛1.

Remark 35.

Comparing (FM LOSS) with the standard 𝝐-prediction objective of diffusion models, 𝔼[𝝐θ(𝒛t,t)𝝐2], we note two key differences. First, the flow matching target 𝒛1𝒛0 is a displacement rather than a noise sample; conceptually, the network predicts “where to go” rather than “what noise was added”. Second, the flow matching objective does not require specifying a noise schedule (αt,σt); the interpolation path is explicit and fixed. This simplifies hyperparameter tuning and removes a common source of instability in audio diffusion training.

The two stages of MusicFlow are trained independently using (FM LOSS) with different conditioning signals. At inference, text is first encoded by the language model, then passed through the semantic flow model to produce semantic tokens, which in turn condition the acoustic flow model. Both stages use an Euler ODE solver with a small number of steps (typically 10 to 25).

Optimal transport conditional paths.

The linear interpolation in (Linear Interp) pairs each noise sample 𝒛0 with a data sample 𝒛1 uniformly at random. This independent coupling can produce crossing paths: two distinct pairs (𝒛0(a),𝒛1(a)) and (𝒛0(b),𝒛1(b)) may have trajectories 𝒛t(a) and 𝒛t(b) that intersect, forcing the velocity field to be multi-valued at the intersection point. Such crossings increase the curvature of the learned velocity field and necessitate more integration steps.

MusicFlow mitigates this by using optimal transport (OT) conditional paths, as described in ch:sinkhorn. Within each training minibatch of size B, the B noise samples and B data samples are paired by solving a discrete optimal transport problem: (OT Coupling)π=arg minπΠ(𝒛01:B,𝒛11:B)i,jπij𝒛0(i)𝒛1(j)2, where Π denotes the set of permutation matrices (or, more generally, doubly stochastic matrices). The OT-paired paths are less likely to cross, yielding a smoother velocity field that can be integrated with fewer steps.

Proposition 36 (OT pairing reduces path curvature).

Let vθind and vθOT denote the velocity fields learned with independent and OT-paired couplings, respectively, on the same dataset. Then for any δ>0, the expected curvature κ=𝔼t[tvθ(𝒛t,t)] satisfies (OT Curvature Bound)κOTκind. Equality holds only in the degenerate case where both couplings coincide. In particular, for Gaussian source and data distributions p0=Normal(0,𝐈), p1=Normal(𝝁,𝚺), the OT-paired velocity field is exactly linear (constant curvature zero), while the independently paired field has non-zero curvature unless 𝚺=𝐈 and 𝝁=0.

Proof.

Consider the conditional velocity at time t for a fixed pair (𝒛0,𝒛1): it is u(𝒛t,t|𝒛0,𝒛1)=𝒛1𝒛0, a constant independent of t. The marginal velocity field is the average over the coupling: v(𝒛,t)=𝔼(𝒛0,𝒛1)π[(𝒛1𝒛0)|𝒛t=𝒛]. The curvature tv(𝒛t,t) arises entirely from the change in the conditional distribution p(𝒛0,𝒛1|𝒛t=𝒛) as t varies. When paths cross, the posterior over (𝒛0,𝒛1) given 𝒛t changes rapidly near the crossing point, producing large tv. OT pairing minimises the total squared displacement 𝔼π[𝒛1𝒛02], which by the Brenier theorem (see ch:sinkhorn) produces non-crossing paths when the source is absolutely continuous. Non-crossing paths ensure that the posterior p(𝒛0,𝒛1|𝒛t) varies smoothly, yielding κOTκind.

For the Gaussian case, the OT map is the affine transformation T(𝒛0)=𝝁+𝚺1/2𝒛0, and the conditional path is 𝒛t=(1t)𝒛0+tT(𝒛0)=((1t)𝐈+t𝚺1/2)𝒛0+t𝝁. The velocity field is v(𝒛,t)=(𝚺1/2𝐈)((1t)𝐈+t𝚺1/2)1(𝒛t𝝁)+𝝁, which has constant-in-𝒛 curvature. Direct computation shows that tv=0 when 𝚺1/2=𝐈, i.e., the paths are perfectly straight when the data covariance matches the source.

MelodyFlow: Melody-Conditioned Flow Matching

While MusicFlow demonstrates the viability of flow matching for text-to-music generation, many creative applications require finer control over the generated output. MelodyFlow extends the flow matching paradigm to accept melody conditioning: a reference pitch contour or hummed melody that guides the harmonic content of the generated music, while text controls higher-level attributes such as genre, instrumentation, and mood.

A distinguishing feature of MelodyFlow is its non-autoregressive architecture. Unlike the autoregressive token-based systems studied in earlier sections, MelodyFlow generates the entire audio segment in parallel by solving the ODE (CNF ODE) from t=0 to t=1. This parallel generation naturally accommodates global conditioning signals (such as a melody contour spanning the full duration) without the left-to-right bottleneck of autoregressive models.

Conditioning mechanism.

The melody conditioning signal 𝒄melT×dmel is a time-frequency representation of the reference melody, typically a chromagram or a pitch-class profile extracted by a pretrained pitch tracker. This is combined with the text embedding 𝒄textdtext through cross-attention layers in the flow model's transformer backbone: (Melodyflow Attention)𝒉=𝒉1+CrossAttn(𝒉1,𝒄text)+CrossAttn(𝒉1,𝒄mel), where 𝒉 is the hidden state at layer of the transformer. The two cross-attention modules operate independently, allowing the model to attend to text and melody information at different spatial and temporal scales.

Classifier-free guidance with dual conditions.

When both text and melody conditions are present, MelodyFlow applies a two-scale classifier-free guidance scheme. During training, the text condition is dropped independently with probability pdrop(text) and the melody condition with probability pdrop(mel). At inference, the guided velocity combines three predictions: (DUAL CFG)v~θ(𝒛t,t)=vθ(𝒛t,t,,)+wtext(vθ(𝒛t,t,𝒄text,)vθ(𝒛t,t,,))+wmel(vθ(𝒛t,t,𝒄text,𝒄mel)vθ(𝒛t,t,𝒄text,)), where wtext and wmel are separate guidance scales. Setting wmel=0 recovers text-only generation; setting wtext=0 and wmel>0 produces melody-conditioned generation without text influence.

Proposition 37 (Flow matching sample efficiency for audio).

Let v^NFM and 𝝐^NDM denote the flow matching and diffusion velocity (resp. noise) estimators trained on N i.i.d. samples from the audio data distribution p1 supported on a compact set 𝒦d. Suppose both estimators use function classes of equal capacity (e.g., transformers with the same architecture). Then the L2 estimation error of the flow matching estimator satisfies (FM Sample Bound)𝔼[01vv^NFML2(pt)2dt]C1dlogNN+C2κOT2δdisc2, where v is the true marginal velocity field, κOT is the path curvature under OT pairing, δdisc is the discretisation error of the minibatch OT solver, and C1,C2 are constants depending on 𝒦 and the function class.

The corresponding bound for the diffusion estimator is (DM Sample Bound)𝔼[01𝝐𝝐^NDML2(pt)2dt]C1dlogNN+C2κdiff2, where κdiff is the average curvature of the diffusion probability flow ODE trajectories.

Since κOTκdiff (by Proposition 36 and the fact that diffusion paths are a special case of independent coupling), and δdisc0 as the minibatch size grows, flow matching achieves a smaller or equal approximation error for the same sample size N.

Proof.

The proof proceeds in two parts.

Part 1: Statistical error. Both estimators minimise a squared-error objective over the training set. By standard results in nonparametric regression (see, e.g., Tsybakov, 2009), the statistical component of the risk for a function class with metric entropy log𝒩(δ,,)Cd(1/δ)d/(d+2) is bounded by O(dlogN/N). This bound applies equally to both estimators, since they use function classes of equal capacity.

Part 2: Approximation error. The approximation error measures how well the function class can represent the true target. For flow matching with OT pairing, the target velocity field v has curvature κOT. A smoother target is easier to approximate: the approximation error of a Lipschitz function class with Lipschitz constant L scales as O(L2κ2) in the regime where the function class is rich enough to capture the dominant modes. Since κOTκdiff, the approximation error of the flow matching estimator is no larger than that of the diffusion estimator. The additional δdisc2 term accounts for the finite-batch approximation of the OT coupling; it vanishes as the minibatch size tends to infinity.

Optimal Transport Paths for Audio

The OT pairing described in MusicFlow: Cascaded Flow Matching operates within minibatches and provides only an approximation to the true optimal transport map between the noise and data distributions. In this subsection, we develop the theory of rectified flows for audio, which iteratively straighten the learned trajectories toward the true OT map, achieving near-linear paths that can be traversed in very few steps.

The key insight, due to Liu et al. (2023), is that the reflow procedure takes a trained flow model and uses its own generated trajectories as training data for a new model, whose paths are provably straighter.

Definition 45 (Reflow procedure for audio).

Let vθ(k) denote the velocity field after k rounds of reflow. The procedure is:

  1. Initialise: Train vθ(0) using the flow matching objective (FM LOSS) with independent coupling.

  2. Generate pairs: For each noise sample 𝒛0(i)Normal(0,𝐈), solve the ODE d𝒛t/dt=vθ(k)(𝒛t,t) from t=0 to t=1 to obtain 𝒛^1(i).

  3. Retrain: Train vθ(k+1) using the flow matching objective with the deterministic coupling (𝒛0(i),𝒛^1(i)).

  4. Repeat steps (2)–(3).

Theorem 2 (Rectified flow convergence).

Let v(k) denote the population-optimal velocity field at reflow iteration k, i.e., the minimiser of 𝔼[v(k)(𝒛t,t)(𝒛^1(k)𝒛0)2] over all measurable functions, where 𝒛^1(k) is generated by the previous iterate. Define the straightness of the flow as (Straightness)S(v)=𝔼𝒛0[01v(𝒛t,t)dt(𝒛1𝒛0)2], where 𝒛1 is the endpoint of the trajectory starting at 𝒛0. Then:

  1. The straightness is non-increasing: S(v(k+1))S(v(k)) for all k0.

  2. The transport cost is non-increasing: 𝔼[𝒛^1(k+1)𝒛02]𝔼[𝒛^1(k)𝒛02].

  3. If S(v(k))0 as k, then the coupling (𝒛0,𝒛^1(k)) converges weakly to the optimal transport coupling, and the one-step approximation 𝒛0+v(k)(𝒛0,0) converges to 𝒛^1(k) in L2.

Proof.

Part (i). By construction, v(k+1) is trained on pairs (𝒛0,𝒛^1(k)) that are deterministically coupled by the ODE of v(k). The conditional path from 𝒛0 to 𝒛^1(k) is a straight line (the linear interpolant), but the actual trajectory under v(k) may be curved. The key observation is that the flow matching loss with deterministic coupling forces v(k+1) to match the straight-line velocity 𝒛^1(k)𝒛0, which is the straightest possible field for this coupling. Therefore, S(v(k+1))S(v(k)).

Part (ii). The transport cost 𝔼[𝒛^1(k)𝒛02] equals the expected squared displacement of the flow. By Cauchy–Schwarz, 𝒛^1(k)𝒛02=01v(k)(𝒛t,t)dt201v(k)(𝒛t,t)2dt. The flow matching training of v(k+1) with the coupling (𝒛0,𝒛^1(k)) produces a field whose integrated squared norm 01v(k+1)(𝒛t,t)2dt is minimised over the class of fields reproducing the same marginals. Since straightening the trajectories can only reduce the integrated kinetic energy (by Jensen's inequality applied to the convex function 2), we obtain 𝔼[𝒛^1(k+1)𝒛02]𝔼[𝒛^1(k)𝒛02].

Part (iii). When S(v(k))0, the flow becomes exactly a straight line from 𝒛0 to 𝒛^1(k). By part (ii), the coupling (𝒛0,𝒛^1(k)) has monotonically decreasing cost, and since it preserves the marginals p0 and p1, any accumulation point must be an optimal coupling (by the Kantorovich characterisation of OT). The one-step convergence follows because S(v(k))0 implies v(k)(𝒛0,0)𝒛^1(k)𝒛0 in L2.

Example 21 (Reflow iterations for music generation).

In practice, one or two reflow iterations suffice for music generation. Starting from a flow matching model trained with independent coupling (reflow iteration k=0), the first reflow (k=1) typically reduces the number of ODE steps needed for acceptable quality from 20 to 5, and the second reflow (k=2) further reduces it to 2 or 3. Beyond k=2, the gains are marginal and do not justify the additional training cost. The following table illustrates typical results on a text-to-music benchmark:

table Effect of reflow iterations on generation quality (FAD) and required ODE steps for music at 32,kHz.

Reflow kSteps = 1Steps = 5Steps = 10Steps = 25
012.43.82.11.7
15.61.91.71.7
23.11.81.71.7
Values are Fréchet Audio Distance (FAD, lower is better). After two reflow iterations, even a single-step generation achieves FAD = 3.1, which is within the acceptable range for many applications.

Caution.

Reflow is computationally expensive for audio. Each reflow iteration requires generating a complete synthetic dataset by solving the ODE for every training sample. For a dataset of one million 10-second audio clips at 24,kHz, with 20 ODE steps per sample and 15,ms per step, the generation phase alone takes approximately 106×20×0.015s=300,000s83hours on a single A100. Parallelising across 64 GPUs reduces this to roughly 1.3 hours, but the subsequent retraining phase adds another full training run. Two reflow iterations thus approximately triple the total training budget. For most practical audio systems, a single reflow combined with minibatch OT pairing during initial training provides the best cost–quality trade-off.

Comparison of sample trajectories in a 2D latent space. Left: Diffusion model reverse trajectories are curved and may cross, requiring many discretisation steps. Middle: Flow matching with independent coupling produces straight per-pair paths, but these may cross in the aggregate, forcing the marginal velocity field to be complex. Right: Flow matching with OT pairing produces nearly straight, non-crossing paths that can be traversed in as few as 1–5 Euler steps. In audio latent spaces, this difference translates to a 5–10× reduction in generation time.

Acceleration: Consistency and Distillation

Flow matching reduces the number of ODE steps needed for audio generation from dozens to a handful. But even 5 or 10 neural network forward passes may be too many for applications that demand true real-time performance: live music accompaniment, interactive sound design, streaming text-to-speech, or in-game audio synthesis. These applications require generating audio faster than it is consumed, with latency measured in milliseconds rather than seconds.

In this section we study two families of techniques that push audio generation toward the single-step regime: consistency models, which learn to map any point along a diffusion or flow trajectory directly to the trajectory's endpoint, and progressive distillation, which iteratively halves the number of required steps by training a student model to match a teacher's two-step output in a single step.

The Real-Time Imperative

Definition 46 (Real-Time Factor).

The real-time factor (RTF) of an audio generation system is the ratio (RTF)RTF=TcomputeTaudio, where Tcompute is the wall-clock time to generate the audio and Taudio is the duration of the generated audio. A system with RTF<1 generates audio faster than real time; RTF=1 is the real-time boundary.

For non-interactive applications (e.g., generating a soundtrack for a video), RTF 1 is desirable but not strictly required. For interactive applications, the constraint is more nuanced: the system must produce the next chunk of audio before the current chunk finishes playing. If the chunk size is L samples at sample rate fs, the maximum allowable latency per chunk is Tmax=L/fs. For a chunk of 1024 samples at 24,kHz, this is approximately 43,ms.

Key Idea.

Audio has stricter real-time requirements than video. A dropped video frame (at 30 fps, a 33,ms gap) is barely noticeable; a 33,ms gap in audio produces an audible click or stutter that is immediately perceived as an artefact. Human auditory temporal resolution is approximately 2,ms, roughly 15 times finer than visual temporal resolution. Consequently, audio generation systems must sustain consistent low-latency output with much tighter jitter budgets than their video counterparts.

Remark 36.

RTF is hardware-dependent: a model with RTF =0.1 on an A100 GPU may have RTF =1.5 on a consumer laptop GPU. When reporting RTF, it is essential to specify the hardware, batch size, and numerical precision. For deployment on edge devices (phones, embedded systems), even models that are “real-time” on server hardware may fail to meet the RTF <1 threshold, making the acceleration techniques in this section practically necessary.

Decomposing generation latency.

The total generation time for an audio diffusion or flow model can be decomposed as (Latency Decomp)Tcompute=NstepsTfwd+Tcond+Tdecode, where Nsteps is the number of denoising or ODE steps, Tfwd is the time for a single forward pass of the backbone network, Tcond is the time to compute conditioning embeddings (text encoder, CLAP, etc.), and Tdecode is the time for the audio codec decoder. The acceleration techniques in this section target Nsteps; complementary approaches (model pruning, quantisation, efficient attention) target Tfwd. In practice, the product NstepsTfwd dominates, so reducing Nsteps from 50 to 1 yields a near-50× end-to-end speedup.

AudioLCM: Consistency Distillation for Audio

Consistency models, introduced by Song et al. (2023), learn a function fθ(𝒛t,t) that maps any point 𝒛t along the probability flow ODE trajectory to the trajectory's origin 𝒛0 (the clean data sample). The defining property is self-consistency: for any two points 𝒛s and 𝒛t on the same trajectory, fθ(𝒛s,s)=fθ(𝒛t,t). AudioLCM adapts this framework to latent audio diffusion models, achieving dramatic speedups.

Definition 47 (Audio Consistency Function).

Let {𝒛t}t[ϵ,T] be the probability flow ODE trajectory of a pretrained audio diffusion model, where 𝒛ϵ𝒛0 (clean audio latent) and 𝒛T𝝐Normal(0,𝐈) (noise). An audio consistency function is a neural network fθ:d×[ϵ,T]d satisfying:

  1. Boundary condition: fθ(𝒛ϵ,ϵ)=𝒛ϵ for all 𝒛ϵ.

  2. Self-consistency: fθ(𝒛s,s)=fθ(𝒛t,t) for all s,t[ϵ,T] and all (𝒛s,𝒛t) on the same ODE trajectory.

The boundary condition (i) ensures that the function acts as the identity near t=ϵ, while self-consistency (ii) ensures that applying fθ at any timestep yields the same clean output. In practice, fθ is parameterised using the pretrained diffusion model's architecture with a skip connection that enforces the boundary condition: (Consistency Param)fθ(𝒛t,t)=cskip(t)𝒛t+cout(t)Fθ(𝒛t,t,𝒄), where cskip(ϵ)=1, cout(ϵ)=0, and Fθ is the base network.

Consistency distillation training.

The consistency function is trained by distilling the pretrained diffusion (teacher) model. Given a pair of timesteps tn+1>tn and a noisy sample 𝒛tn+1, the teacher model performs one ODE step to estimate 𝒛^tn. The consistency distillation loss then enforces self-consistency: (CD LOSS)CD=𝔼n,𝒛tn+1[d(fθ(𝒛tn+1,tn+1),fθ(𝒛^tn,tn))], where d(,) is a distance metric (typically the LPIPS perceptual distance adapted for spectrograms, or a combination of L1 and mel-spectrogram losses), 𝒛^tn is the teacher's one-step estimate, and θ denotes the exponential moving average (EMA) of the student's parameters (the “target network”, updated as θμθ+(1μ)θ).

Proposition 38 (AudioLCM speedup bound).

Let Nteacher denote the number of ODE steps required by the teacher diffusion model to generate audio of acceptable quality (FAD τ). After consistency distillation, the student AudioLCM model achieves the same quality threshold in Nstudent steps, where (Audiolcm Speedup)NteacherNstudentNteacher2 in the favourable case where Nstudent2. For typical teacher models with Nteacher=200 (DDPM-style) or Nteacher=50 (DPM-Solver), this yields speedup factors of 100× to 333× while maintaining FAD within 10% of the teacher's score.

Proof.

The bound follows from the definition of consistency distillation. The teacher requires Nteacher sequential forward passes to traverse the ODE from t=T to t=ϵ. The consistency function, by property (ii) of Definition 47, maps any point on the trajectory directly to the endpoint. A single application of fθ(𝒛T,T) therefore approximates the teacher's Nteacher-step output, achieving a speedup ratio of Nteacher/1.

In practice, a single step incurs some quality loss because the learned consistency function is not perfectly self-consistent (finite-sample training error and limited network capacity). Using Nstudent=2 steps (applying the consistency function at two timesteps with a single intermediate step) recovers most of the quality. The speedup ratio is then Nteacher/2. For Nteacher=200, this gives 100×; for Nteacher=50, this gives 25×. The “333×” figure cited in the literature uses Nstudent=1 (single-step generation) with a teacher requiring approximately 200 steps with a first-order sampler, but incorporating the overhead of classifier-free guidance (which doubles the effective NFE), yielding 200×2/1.2333.

The quality preservation (FAD within 10% of the teacher) is an empirical observation validated on standard benchmarks (AudioCaps, MusicCaps) and is not guaranteed theoretically. It depends on the consistency function's capacity and the distillation schedule.

Example 22 (AudioLCM for text-to-audio).

Consider a latent diffusion model for text-to-audio generation based on the AudioLDM2 architecture. The teacher model uses 200 DDPM steps with classifier-free guidance (scale w=3.5), requiring 400 forward passes (200 conditional + 200 unconditional) per sample. Each forward pass takes 8,ms on an A100 GPU, yielding a total generation time of 3.2,s for a 10-second audio clip (RTF = 0.32).

After consistency distillation with the loss in (CD LOSS), the AudioLCM student achieves comparable quality with 2 steps and guidance scale w=2.0. Using a single-step guidance approximation (no separate unconditional evaluation), the total forward passes reduce to 2, and the generation time drops to 16,ms (RTF = 0.0016), more than sufficient for real-time streaming.

ConsistencyTTA: Single-Step Text-to-Audio

ConsistencyTTA pushes the consistency model paradigm to its logical extreme: generating audio in a single forward pass. While AudioLCM achieves near-single-step generation through distillation from a pretrained teacher, ConsistencyTTA explores consistency training (CT), where the consistency property is enforced directly during training without a separate teacher model.

Definition 48 (Single-Step Audio Generation).

A single-step audio generation model is a function Gθ:d×𝒞d that maps a single noise sample 𝒛Normal(0,𝐈) and a conditioning signal 𝒄𝒞 directly to a clean audio latent: (Single STEP)𝒛^audio=Gθ(𝒛,𝒄),𝒛Normal(0,𝐈). The generation requires exactly one forward pass of the network, with no iterative refinement.

Quality-speed trade-offs.

Single-step generation fundamentally changes the quality-speed landscape. In multi-step models, generation quality improves monotonically with the number of steps (up to the teacher's quality ceiling). In single-step models, there is no such knob to turn at inference time; all quality must be baked into the model at training time. This creates a different set of design trade-offs:

  1. Model capacity. Single-step models typically require larger networks than their multi-step counterparts to compensate for the loss of iterative refinement. A 1B-parameter single-step model may match the quality of a 400M-parameter 50-step model.

  2. Training cost. Consistency training requires careful annealing of the timestep schedule during training and is more sensitive to hyperparameters than standard diffusion training. The total training compute is typically 1.5 to 3× that of the corresponding multi-step model.

  3. Diversity. Single-step models tend to produce less diverse outputs than multi-step models, because the stochastic exploration that occurs during iterative sampling is absent. This manifests as a slight reduction in the range of timbres, rhythmic patterns, and harmonic progressions observed across multiple generations from the same text prompt.

  4. Guidance integration. Classifier-free guidance, which requires evaluating the model twice per step (conditional and unconditional), doubles the cost of each step. For single-step models, this overhead is absolute: the model must either forgo guidance or use distilled guidance (where the guidance signal is absorbed into the model weights during training).

Caution.

Single-step models are not universal replacements for multi-step models. While the speedup is dramatic, single-step audio generation models currently lag behind their multi-step counterparts in terms of fine-grained controllability (e.g., precise timing of musical events) and perceptual quality on challenging prompts (e.g., complex polyphonic arrangements with multiple instruments). For applications where quality is paramount and latency is flexible (e.g., offline music production), multi-step models remain the preferred choice.

Progressive Distillation for Audio

Progressive distillation offers an alternative path to few-step audio generation that is conceptually simpler than consistency training. The idea is to iteratively halve the number of required sampling steps by training a student model to match the output of two teacher steps in a single student step.

Algorithm 5 (Progressive Distillation for Audio Generation).

Input: Pretrained teacher model 𝝐ϕ(0) requiring N0 steps; number of distillation rounds R; learning rate η; EMA decay μ.

Output: Student model 𝝐ϕ(R) requiring N0/2R steps.

  1. for r=1,2,,R
  2. NrN0/2r Halve the number of steps
  3. Initialise student 𝝐ϕ(r) from teacher 𝝐ϕ(r1)
  4. for each training iteration
  5. Sample 𝒛1pdata, 𝝐Normal(0,𝐈), timestep index n{1,,Nr}
  6. tnn/Nr, tn+1(n+1)/Nr, tn+1/2(n+1/2)/Nr
  7. 𝒛tn+1αtn+1𝒛1+σtn+1𝝐 Noisy sample at coarse step
  8. Teacher two-step:
  9. 𝒛^tn+1/2DDIM-step(𝝐ϕ(r1),𝒛tn+1,tn+1tn+1/2)
  10. 𝒛^tnteacherDDIM-step(𝝐ϕ(r1),𝒛^tn+1/2,tn+1/2tn)
  11. Student one-step:
  12. 𝒛^tnstudentDDIM-step(𝝐ϕ(r),𝒛tn+1,tn+1tn)
  13. w(tn)𝒛^tnstudent𝒛^tnteacher2 Weighted MSE loss
  14. Update ϕ(r) using ϕ with learning rate η
  15. return 𝝐ϕ(R)

Remark 37.

Each distillation round halves the step count: starting from N0=128 steps, four rounds yield an 8-step model, and seven rounds yield a single-step model (128/27=1). However, quality degrades progressively with each round, and the degradation accelerates in later rounds. For audio generation, three to four rounds (reducing from 100 steps to 8–12 steps) provide the best quality-speed trade-off. Going below 4 steps typically requires combining progressive distillation with consistency fine-tuning.

Example 23 (Progressive distillation schedule for music).

Starting from a teacher model that uses N0=64 DDIM steps for text-conditioned music generation at 32,kHz (10-second clips, FAD = 1.8 on MusicCaps), the following distillation schedule is typical:

table Progressive distillation rounds for a 64-step teacher music generation model. Each round halves the step count.

Round rSteps NrTraining itersFADNotes
0 (teacher)641.8Baseline
13250k1.9Negligible quality loss
21650k2.0Still excellent quality
3880k2.3Minor degradation
44100k2.9Noticeable loss on complex prompts
52150k4.1Significant degradation
61200k6.8Poor; use consistency fine-tuning
Observe that rounds 1–3 are nearly free in quality terms, while rounds 5–6 incur substantial degradation. The sweet spot for pure progressive distillation is 4–8 steps; reaching 1–2 steps benefits from additional consistency training.

Proposition 39 (Progressive distillation error accumulation).

Let ϵr denote the L2 distillation error introduced at round r, i.e., ϵr=𝔼[𝒛^rstudent𝒛^rteacher2]1/2 where the expectation is over the noisy inputs and timesteps. Then the cumulative error after R rounds satisfies (Progressive Error)𝔼[𝒛^(R)𝒛^(0)2]1/2r=1RϵrRmax1rRϵr. In practice, ϵr tends to increase with r (later rounds are harder because the student must compress more of the trajectory into each step), so the error growth is super-linear in R. This explains the accelerating quality degradation observed in tab:audiogen:progressive-schedule.

Proof.

By the triangle inequality applied to the chain 𝒛^(0)𝒛^(1)𝒛^(R): 𝒛^(R)𝒛^(0)r=1R𝒛^(r)𝒛^(r1). Taking expectations and applying linearity of expectation, we obtain 𝔼[𝒛^(R)𝒛^(0)]r=1R𝔼[𝒛^(r)𝒛^(r1)]=r=1Rϵr. The bound rϵrRmaxrϵr follows immediately. The squared version uses the elementary inequality (rar)2Rrar2 (Cauchy–Schwarz) to obtain 𝔼[𝒛^(R)𝒛^(0)2]Rrϵr2.

Speed-quality Pareto frontier for 10-second audio generation on the AudioCaps benchmark (A100 GPU, batch size 1). Each point represents a different number of sampling steps for the given method. The dashed curve indicates the approximate Pareto frontier. Methods closer to the bottom-left corner offer the best trade-off between generation speed and audio quality. Consistency and flow-based methods push the frontier significantly toward lower latency while maintaining competitive FAD scores.

Long-Form Audio and Music Generation

The audio and music generation systems studied so far produce segments of fixed, modest duration: typically 5 to 30 seconds. This window is dictated by the model's context length (for autoregressive systems) or the latent tensor size (for diffusion and flow models), both of which are constrained by GPU memory and the quadratic cost of attention. Yet the audio that humans actually want to generate, a complete song, a podcast episode, a film soundtrack, spans minutes or hours. Bridging the gap between a model's native generation window and the durations demanded by real applications is one of the central open challenges in audio generation.

This section analyses the long-form challenge from both theoretical and practical perspectives. We begin by characterising the hierarchical structure of music and audio (The Long-Form Challenge), then examine hierarchical generation approaches (Hierarchical Approaches), autoregressive chunking with overlap (Autoregressive Chunking with Overlap), and song-level structure modelling (Song-Level Structure Modelling). The reader may find it helpful to compare these approaches with the analogous strategies for long video generation discussed in sec:vdiff:long-video.

The Long-Form Challenge

Music possesses a natural hierarchical structure spanning multiple timescales. Understanding these timescales is essential for designing systems that can generate coherent long-form content.

Definition 49 (Musical Timescale Hierarchy).

We identify five principal timescales in Western music, each approximately an order of magnitude longer than the previous:

  1. Note level (50–500,ms): Individual note onsets, attacks, and decays. The pitch, duration, and dynamics of a single note.

  2. Beat/bar level (0.5–4,s): Rhythmic patterns, chord changes, and local harmonic progressions within a measure.

  3. Phrase level (4–16,s): Melodic phrases, harmonic cadences, and call-and-response patterns spanning several bars.

  4. Section level (16–60,s): Verse, chorus, bridge, solo, and other structural sections of a song. Each section has a distinct harmonic, rhythmic, and textural character.

  5. Song level (2–10,min): The overall form of a piece: the sequence of sections, the arc of tension and release, the introduction and coda.

Current generation models with a 10-second window operate comfortably at the phrase level but struggle at the section level and fail entirely at the song level. The fundamental difficulty is that coherence at higher levels requires information about events that occurred many seconds or minutes in the past.

Insight.

The information bottleneck at chunk boundaries. When generating long audio by concatenating chunks, the model must encode all relevant information about the past into a finite-dimensional conditioning signal (e.g., the last few seconds of the previous chunk, or a summary embedding). This creates an information bottleneck: details about musical form, key modulations, and thematic development are lost when they cannot be represented in the conditioning signal. The result is music that sounds locally coherent but globally aimless, lacking the large-scale structure that distinguishes a composed piece from an improvisation.

Example 24 (Timescale vs. model context).

Consider a transformer-based music generation model operating on latent tokens with a temporal compression factor of 320 (at 24,kHz, each token represents 13,ms of audio). With a context length of 2048 tokens, the model's receptive field spans 2048×13ms27s. This is sufficient for phrase-level coherence but inadequate for maintaining section-level structure in a 4-minute song (requiring 18,500 tokens). Even with 8192-token context (a significant memory investment), the model sees only 107,s, or roughly one-half of a typical pop song.

Memory and compute scaling.

The computational cost of extending the generation window depends on the model architecture. For a transformer with full self-attention over L tokens, the memory scales as O(L2) and the compute per step as O(L2d), where d is the model dimension. Generating a 4-minute song at 24,kHz with compression factor 320 requires L18,500 tokens. Compared to a 10-second window (L750), this represents a (18,500/750)2608× increase in attention memory and compute. Even with linear-attention variants or sparse attention patterns, the scaling remains at least O(LlogL), making brute-force extension of the generation window impractical for song-length audio.

Hierarchical Approaches

One strategy for overcoming the context-length bottleneck is to generate music at multiple levels of the timescale hierarchy, using the output of coarser levels to condition finer levels. This mirrors the hierarchical approach used in long video generation (see sec:vdiff:long-video) and in cascade-based image generation.

Definition 50 (Hierarchical Music Generation).

A two-level hierarchical music generation system consists of:

  1. A structure model pθ1(𝒔|𝒄text) that generates a compact structural representation 𝒔Ls×ds, where Ls is the number of structural tokens and ds is the token dimension. The structural representation encodes high-level attributes: section boundaries, key centers, tempo changes, and instrumentation transitions.

  2. An audio model pθ2(𝒛|𝒔,𝒄text) that generates the full audio latent 𝒛La×da conditioned on the structural plan 𝒔 and (optionally) the text prompt. The audio model operates at a finer temporal resolution (LaLs) and is responsible for local details: note-level timing, timbre, dynamics, and mixing.

The full generation process is: (Hierarchical GEN)𝒔pθ1(|𝒄text),𝒛pθ2(|𝒔,𝒄text),𝒙^=Decode(𝒛), where Decode is the audio codec decoder (e.g., EnCodec, DAC).

MusicWeaver.

MusicWeaver is an exemplar of the hierarchical approach. The structure model is a small transformer that generates a sequence of “structure tokens” at a rate of approximately one token per bar (roughly 2 seconds of music). Each structure token encodes a low-dimensional embedding of the harmonic, rhythmic, and timbral characteristics of the corresponding bar. The structure tokens are obtained during training by applying a learned quantiser to bar-level summaries extracted from the training data.

The audio model is a larger diffusion transformer that generates the full-resolution audio latent conditioned on the structure tokens via cross-attention. Because the structure tokens span the entire song, the audio model has access to global structural information even when it generates individual chunks.

Remark 38.

Training hierarchical systems requires aligned data at both levels. The structure model needs annotations (or automatically extracted features) indicating section boundaries, chord progressions, and instrumentation changes. Obtaining such annotations at scale is challenging; current systems rely on a combination of music information retrieval (MIR) tools and self-supervised feature extraction. The quality of the structural annotations directly limits the quality of the hierarchical generation.

Proposition 40 (Hierarchical compression advantage).

Let La denote the length of the audio latent sequence and Ls the length of the structural sequence, with compression ratio ρ=La/Ls. In a hierarchical system where the structure model uses full self-attention over Ls tokens and the audio model uses full self-attention over chunks of Lc tokens (with LcLa), the total attention cost for generating a piece of length La is (Hierarchical COST)𝒞hier=Ls2+LaLcδLc2=La2ρ2+LaLc2Lcδ, where δ is the overlap between adjacent chunks. Compared to the monolithic cost 𝒞mono=La2, the hierarchical cost satisfies 𝒞hier𝒞mono=1ρ2+Lc2La(Lcδ)1ρ2+LcLa for δLc. With typical values ρ=75 (one structure token per bar at 2,s, audio tokens at 75/s) and Lc/La=750/18,5000.04, the hierarchical cost is approximately 0.04 of the monolithic cost, a 25× reduction.

Proof.

The structure model attends over Ls=La/ρ tokens with full self-attention, giving cost Ls2=La2/ρ2. The audio model generates La/(Lcδ) chunks, each with attention cost Lc2. Summing gives (Hierarchical COST). The ratio to 𝒞mono=La2 follows by algebraic simplification.

Hierarchical music generation pipeline. The structure model (top) generates a compact sequence of structural tokens at a coarse temporal resolution (approximately one token per bar), capturing high-level musical form. The audio model (middle) generates full-resolution audio latents conditioned on both the structural plan and the text prompt. The codec decoder (bottom) converts the latents to a waveform.

Autoregressive Chunking with Overlap

The most widely used approach to long-form audio generation is conceptually simple: generate the audio one chunk at a time, conditioning each new chunk on the tail of the previous chunk to ensure continuity at the boundary. This mirrors the autoregressive chunk generation strategy for long video (see sec:vdiff:long-video), adapted to the particular demands of audio coherence.

Let 𝒙(k)L denote the k-th chunk of audio (in the latent space, with L tokens per chunk), and let δ{1,,L1} be the overlap parameter. The generation factorises as (AR Chunk)p(𝒙1:K)=p(𝒙(1))k=2Kp(𝒙(k)|𝒙Lδ:L(k1)), where 𝒙Lδ:L(k1) denotes the last δ tokens of chunk k1, and K=(Nδ)/(Lδ) is the total number of chunks for an N-token output.

Cross-fading.

The overlap region between consecutive chunks is generated twice: once as the tail of chunk k1 and once as the head of chunk k. To avoid discontinuities, the two versions are blended using a cross-fade: (Crossfade)𝒙^n={𝒙n(k1)if n<Lδ,(1αn)𝒙n(k1)+αn𝒙n(k)if Lδn<L,𝒙n(k)if nL, where αn=(n(Lδ))/δ is a linear blend weight. More sophisticated cross-fade functions (e.g., raised cosine, Hann window) can reduce audible artefacts at the boundary.

Proposition 41 (Overlap-add coherence).

Let 𝒙(k1) and 𝒙(k) be audio chunks generated by a model with conditional distribution p(𝒙(k)|𝒙Lδ:L(k1)), and let 𝒙^ be the cross-faded output as in (Crossfade). Define the boundary discontinuity as (Boundary DISC)D(δ)=𝔼[𝒙1:δ(k)𝒙Lδ:L(k1)2]. Then:

  1. D(δ) is non-increasing in δ: more overlap reduces the expected boundary discontinuity.

  2. For a model with stationary conditional statistics, D(δ)Ceγδ for constants C,γ>0 depending on the model's temporal autocorrelation length.

  3. After cross-fading, the residual error in the overlap region satisfies (Crossfade Error)Ecf(δ)=𝔼[𝒙^Lδ:L𝒙Lδ:L2]D(δ)4, where 𝒙 is the “ground truth” (the audio that would be generated by a model with infinite context). The factor of 1/4 arises from the linear blending weights.

Proof.

Part (i). Increasing δ provides the model with more context from the previous chunk. By the data-processing inequality (or simply the observation that more conditioning information can only reduce the conditional variance), the conditional distribution p(𝒙1:δ(k)|𝒙Lδ:L(k1)) becomes more concentrated around 𝒙Lδ:L(k1) as δ increases. Hence D(δ) is non-increasing.

Part (ii). For a stationary model, the mutual information between the overlap region of the new chunk and the conditioning region decays exponentially with the temporal distance from the conditioning boundary (by the exponential decay of autocorrelation in most audio models). The expected squared deviation is bounded by the conditional variance, which inherits this exponential decay: D(δ)Ceγδ.

Part (iii). In the overlap region, the cross-faded output at position n is 𝒙^n=(1αn)𝒙n(k1)+αn𝒙n(k), with αn[0,1]. The ground truth 𝒙n lies (approximately) at the mean of 𝒙n(k1) and 𝒙n(k). The squared error is 𝒙^n𝒙n2=(1αn)(𝒙n(k1)𝒙n)+αn(𝒙n(k)𝒙n)2(1αn)2𝒙n(k1)𝒙n2+αn2𝒙n(k)𝒙n2+cross terms. Taking expectations and noting that the deviations of 𝒙n(k1) and 𝒙n(k) from 𝒙n are approximately equal in magnitude (both are O(D(δ))) and approximately uncorrelated (since they are generated from independent noise), the expected squared error integrates over αn[0,1] to Ecf=01((1α)2+α2)dαD(δ)2=13D(δ)2=D(δ)6. The stated bound D(δ)/4 is a looser but simpler upper bound that holds without the independence assumption (using the triangle inequality directly).

Example 25 (Practical chunking parameters).

Consider a latent diffusion model generating audio at 24,kHz with compression factor 320. Each chunk contains L=750 latent tokens (10,s of audio). With overlap δ=75 tokens (1,s), generating a 3-minute song requires K=180s75tok/s7575075=13,425675=20 chunks. At 10 ODE steps per chunk with 15,ms per step, the total generation time is 20×10×0.015=3.0,s, yielding RTF =3.0/180=0.017. The overhead from overlap is modest: 75/750=10% of tokens are re-generated for cross-fading.

Temporal inpainting for chunk boundaries.

An alternative to cross-fading is temporal inpainting: the model generates the overlap region conditioned on both the tail of the previous chunk and the head of the next chunk (or a preliminary version of it). This produces a smoother transition because the model can “see” both sides of the boundary. The inpainting formulation treats the non-overlap regions as observed and the overlap region as latent: (Inpainting)p(𝒙overlap|𝒙left,𝒙right)p(𝒙left,𝒙overlap,𝒙right), where 𝒙left and 𝒙right are the observed segments. In practice, this posterior is approximated using the Replace-then-Denoise strategy: at each denoising step, the known regions are replaced with their noised versions, and only the overlap region is updated by the model.

Song-Level Structure Modelling

Neither hierarchical generation nor autoregressive chunking directly models the large-scale form of a song. Musical form, the arrangement of sections into a coherent whole, is a defining feature of composed music, distinguishing it from aimless improvisation. Common forms include:

  • AABA form: Two statements of a theme (A), a contrasting bridge (B), and a return to the theme (A). Common in jazz standards and Tin Pan Alley songs.

  • Verse–chorus form: Alternating verses (new lyrics over similar music) and a recurring chorus (fixed lyrics and melody). Dominant in pop and rock music.

  • Rondo form: A principal theme alternates with contrasting episodes: ABACADA. Common in classical music.

  • Through-composed: No repeated sections; the music evolves continuously. Common in art songs and some film scores.

Self-similarity matrices.

A powerful tool for analysing and imposing musical form is the self-similarity matrix (SSM). Given a sequence of feature vectors {𝒇t}t=1T extracted from an audio recording (or generated audio), the SSM is the T×T matrix 𝐒 with entries (SSM)Sij=κ(𝒇i,𝒇j), where κ is a similarity kernel (e.g., cosine similarity, RBF kernel). For music with clear sectional structure, the SSM exhibits a block-diagonal pattern: elements within the same section have high similarity, and repeated sections (e.g., two occurrences of the chorus) produce off-diagonal blocks of high similarity.

Definition 51 (Structure-Conditioned Generation).

A structure-conditioned generation model takes as input a target self-similarity matrix 𝐒T×T (or an equivalent structural specification, such as a list of section labels and boundaries) and generates audio whose SSM approximates 𝐒: (Structure COND GEN)𝒙^pθ(|𝐒,𝒄text),subject to𝐒(𝒙^)𝐒Fϵ, where 𝐒(𝒙^) is the SSM computed from the generated audio and ϵ is a tolerance.

Remark 39.

The SSM constraint in (Structure COND GEN) can be enforced during sampling via classifier guidance. At each denoising step, the predicted clean audio 𝒙^0 is used to compute an approximate SSM 𝐒(𝒙^0), and the gradient of the Frobenius-norm discrepancy 𝒛t𝐒(𝒙^0)𝐒F2 is added to the score function. This “SSM guidance” nudges the generation toward producing the desired sectional structure without retraining the model. The approach is analogous to image generation with classifier guidance, but the guiding signal is a structural property of the output rather than a class label.

Conjecture 2 (Recurrence structure improves long-form quality).

Let pθ be a music generation model and let 𝒙^ denote a generated song with self-similarity matrix 𝐒(𝒙^). Define the recurrence score as (Recurrence Score)R(𝒙^)=1|𝒪|(i,j)𝒪Sij, where 𝒪={(i,j):|ij|>Lphrase,Sij>τ} is the set of off-diagonal high-similarity pairs beyond a phrase-length distance. We conjecture that for human-composed music, R(𝒙^) is positively correlated with human preference scores for “musical coherence” and “song-like quality”, and that explicitly maximising R during generation (via guidance or reward-weighted training) improves subjective quality ratings for long-form outputs.

Remark 40.

Current state-of-the-art systems are far from achieving reliable song-level structure modelling. The principal limitations are:

  1. Training data scarcity. Most music datasets contain short clips (10–30,s), not full songs with structural annotations. Models trained on short clips never learn the statistical patterns of large-scale form.

  2. Evaluation metrics. Standard audio quality metrics (FAD, KL divergence, inception score) evaluate perceptual quality at short timescales and do not capture structural coherence. No widely accepted metric measures whether generated music has coherent song-level form.

  3. Computational cost. Full-song generation at CD quality (44.1,kHz, stereo) requires handling millions of latent tokens, straining the memory and computational budgets of current hardware.

  4. Human preferences. Human evaluation of long-form music is expensive and subjective. Listeners' assessments of structural coherence depend heavily on musical training and genre familiarity, making it difficult to obtain reliable ground-truth judgements.

Addressing these limitations is an active area of research, and we expect significant progress in the coming years as training datasets grow and evaluation methodology matures.

Multi-Track Generation and Source Separation

Music, in its most natural form, is not a monolithic waveform but a superposition of individual stems: drums, bass, vocals, keyboards, guitars, strings, and other instruments. Professional music production treats each stem separately, adjusting its volume, panning, equalization, and effects before combining the stems into a final mix. This section studies the generative modelling of multi-track music and its dual problem, source separation, from a unified probabilistic perspective.

Multi-Track Generation

Multi-track generation aims to produce a set of K coherent stems {𝒙1,,𝒙K} that sound musically meaningful both individually and when mixed together. The simplest approach is to model the joint distribution p(𝒙1,,𝒙K|𝒄) directly, treating the multi-track audio as a single high-dimensional object. More practical approaches decompose this joint distribution into manageable factors.

Definition 52 (Multi-Track Audio Model).

A multi-track audio model generates K stems 𝒙1,,𝒙KT (each a T-sample waveform or latent sequence) conditioned on a prompt 𝒄, such that:

  1. Each stem 𝒙k is a plausible audio signal for its assigned instrument or role.

  2. The mixture 𝒙mix=k=1K𝒙k is a plausible musical piece.

  3. The stems are temporally aligned: onsets, beats, and harmonic changes are synchronised across stems.

The model can be parameterised in one of three ways:

  1. Joint: pθ(𝒙1,,𝒙K|𝒄), modelling all stems simultaneously. The latent space has dimension K times that of a single-stem model.

  2. Sequential (cascade): pθ(𝒙1|𝒄)k=2Kpθ(𝒙k|𝒙1:k1,𝒄), generating stems one at a time, each conditioned on previously generated stems.

  3. Independent with coupling: k=1Kpθ(𝒙k|𝒄,𝒛), where 𝒛 is a shared latent code that enforces coherence (e.g., a harmonic plan, a rhythmic grid, or a structural skeleton generated by a separate model).

Joint generation.

The joint approach concatenates or interleaves the stems along the channel dimension: 𝑿=[𝒙1;𝒙2;;𝒙K]K×T is treated as a K-channel audio signal. A diffusion or flow model then generates 𝑿 as if it were a single multi-channel audio. The advantage is that the model can capture arbitrary dependencies between stems (e.g., the bass following the kick drum, the vocals interacting with the harmonic progression). The disadvantage is the increased dimensionality: for K=4 stems, the latent space is 4 times larger, increasing memory and compute requirements by roughly the same factor.

Sequential generation.

The sequential approach generates stems in a fixed order (e.g., drums bass harmony vocals), conditioning each new stem on the previously generated ones. This reduces the per-step dimensionality to that of a single stem but introduces an order dependence: the first stem receives no musical context from the others, which can lead to suboptimal results if the “most important” stem (often vocals or melody) is not generated first.

Example 26 (Joint vs. sequential multi-track generation).

Consider generating a 4-stem mix (drums, bass, piano, vocals) for a 10-second clip at 24,kHz with a latent compression factor of 320.

table Comparison of joint and sequential multi-track generation for K=4 stems, 10 seconds at 24,kHz.

ApproachLatent dimNFETime (s)Total time (s)
Joint (K-channel)4×750201.21.2
Sequential (4 stages)1×750200.31.2
Indep. + coupling1×750200.30.3
5@l@ Parallel generation on 4 GPUs or using batched inference.
The joint and sequential approaches have similar total generation times, but the sequential approach can begin playing the first stem before the last stem is generated, enabling a form of streaming output. The independent approach is fastest when parallel hardware is available, but relies on the shared latent code 𝒛 to ensure coherence, which may limit musical interaction between stems.

Source Separation as Inverse Generation

Source separation, the task of extracting individual stems 𝒙1,,𝒙K from a mixture 𝒙mix=k=1K𝒙k, is the inverse problem dual to multi-track generation. A generative model that can produce plausible stems provides a natural prior for this inverse problem, leading to a Bayesian formulation of source separation.

Proposition 42 (Separation via posterior sampling).

Let p(𝒙k) be the prior distribution over stem k, learned by a pretrained generative model. Assume the mixing model is linear and noiseless: 𝒙mix=k=1K𝒙k. Then the posterior distribution of stem k given the mixture is (Separation Posterior)p(𝒙k|𝒙mix)p(𝒙mix|𝒙k)p(𝒙k), where the likelihood is (Separation Likelihood)p(𝒙mix|𝒙k)=jkp(𝒙j)δ(𝒙mix𝒙kjk𝒙j)jkd𝒙j. In the two-source case (K=2), the likelihood simplifies to p(𝒙mix|𝒙1)=p2(𝒙mix𝒙1), where p2 is the prior for the second source.

Proof.

By Bayes' theorem, p(𝒙k|𝒙mix)p(𝒙mix|𝒙k)p(𝒙k). The likelihood p(𝒙mix|𝒙k) is obtained by marginalising over all other sources: p(𝒙mix|𝒙k)=p(𝒙mix|𝒙1,,𝒙K)jkp(𝒙j)jkd𝒙j=δ(𝒙mixj=1K𝒙j)jkp(𝒙j)jkd𝒙j, where we used the deterministic mixing model p(𝒙mix|𝒙1,,𝒙K)=δ(𝒙mixj𝒙j). For K=2, p(𝒙mix|𝒙1)=δ(𝒙mix𝒙1𝒙2)p(𝒙2)d𝒙2=p2(𝒙mix𝒙1).

Diffusion-based source separation.

The posterior in (Separation Posterior) is intractable in general, but it can be approximated using the machinery of diffusion models. The key idea is to run the reverse diffusion process for each source 𝒙k while enforcing the mixture constraint at each denoising step. Concretely, at each timestep t of the reverse process:

  1. Predict the clean estimate 𝒙^k(t) for each source k using source-specific diffusion models (or a shared model with source-class conditioning).

  2. Project the estimates onto the mixture constraint: replace {𝒙^k(t)} with the nearest set of signals (in L2 norm) that sum to 𝒙mix: (Mixture Projection)𝒙~k(t)=𝒙^k(t)+1K(𝒙mixj=1K𝒙^j(t)). This additive correction distributes the residual equally among all sources.

  3. Continue the reverse diffusion from the projected estimates.

Lemma 3 (Optimality of equal redistribution).

The projection in (Mixture Projection) is the orthogonal projection of the unconstrained estimates {𝒙^k(t)}k=1K onto the affine subspace ={(𝒙1,,𝒙K):k=1K𝒙k=𝒙mix}.

Proof.

We seek {𝒙~k} minimising k=1K𝒙~k𝒙^k2 subject to k=1K𝒙~k=𝒙mix. Introducing a Lagrange multiplier 𝝀 for the constraint, the KKT conditions give 𝒙~k=𝒙^k+𝝀/2 for each k. Substituting into the constraint: k(𝒙^k+𝝀/2)=𝒙mix, so 𝝀/2=(𝒙mixk𝒙^k)/K, which yields (Mixture Projection).

Remark 41.

The use of diffusion priors for inverse problems is a general technique studied in ch:vision for image restoration tasks (denoising, inpainting, super-resolution). Audio source separation is an instance of the same paradigm, with the “forward model” being linear mixing rather than image degradation. The mathematical framework is identical; only the domain-specific priors differ.

Algorithm 6 (Diffusion-Based Source Separation).

Input: Mixture 𝒙mixT; pretrained score models {sθk}k=1K for each source class; noise schedule {σt}t=Tt=0; number of steps N.

Output: Estimated sources {𝒙^k}k=1K.

  1. for k=1,,K
  2. 𝒛k(T)Normal(0,σT2𝐈) Initialise from noise
  3. for n=N,N1,,1
  4. tnn/NT
  5. for k=1,,K
  6. 𝒙^k(0)Tweedie(sθk,𝒛k(tn),tn) Predict clean source
  7. Mixture projection:
  8. for k=1,,K
  9. 𝒙~k(0)𝒙^k(0)+1K(𝒙mixj𝒙^j(0))
  10. Noise and step:
  11. for k=1,,K
  12. 𝒛k(tn1)DDPM-step(𝒛k(tn),𝒙~k(0),tntn1)
  13. return {𝒙~k(0)}k=1K

Style Transfer and Timbre Control

Beyond generating audio from text, many applications require transforming existing audio: changing the instrumentation of a piano recording to an orchestral arrangement, transferring the “feel” of one drum pattern to another, or converting a speaking voice to a singing voice while preserving the linguistic content. These tasks fall under the broad umbrella of audio style transfer.

Timbre transfer.

Timbre, often described as the “colour” of a sound, is the perceptual quality that distinguishes two notes of the same pitch and loudness played by different instruments. Formally, timbre encompasses all spectral and temporal characteristics not captured by pitch, loudness, and duration. Timbre transfer aims to change the timbre of an audio signal while preserving its pitch, rhythm, and temporal structure.

In the diffusion/flow framework, timbre transfer is implemented as constrained sampling: the model generates audio that matches a target timbre distribution while remaining close to the source audio in a content-preserving feature space.

Definition 53 (Audio Style Transfer via Constrained Posterior).

Let 𝒙in be the source audio, 𝒄style the target style descriptor, and p(𝒙|𝒄style) a style-conditioned generative prior. The audio style transfer posterior is (Constrained Style)p(𝒙out|𝒙in,𝒄style)p(𝒙out|𝒄style)exp(λdcontent(𝒙out,𝒙in)), where dcontent(,) is a content-preserving distance (e.g., based on pitch contour similarity, onset timing, or learned content embeddings) and λ>0 controls the strength of the content constraint.

Insight.

Constrained sampling unifies style transfer tasks. Timbre transfer, genre transfer, and other audio style transfer tasks can all be formulated via (Constrained Style) by choosing an appropriate content distance dcontent. For timbre transfer, dcontent preserves pitch, onset timing, and temporal envelope while allowing spectral shape changes. For genre transfer, dcontent preserves only coarse features (harmonic progression, melodic contour, song structure) while allowing changes to rhythm, timbre, and production style. This formulation reduces style transfer to posterior sampling under a diffusion or flow prior, using the same machinery developed for inverse problems in ch:vision.

Example 27 (Timbre transfer via diffusion inversion).

A common approach to timbre transfer uses DDIM inversion (see ch:diffusion). Given a source audio latent 𝒛1 and a target timbre description 𝒄timbre:

  1. Invert: Run the DDIM forward process (deterministic noising) on 𝒛1 to obtain a noise latent 𝒛T at the terminal timestep.

  2. Regenerate: Run the DDIM reverse process from 𝒛T using the diffusion model conditioned on 𝒄timbre instead of the original conditioning.

  3. Blend: Optionally blend the regenerated audio with the original at a controllable ratio to preserve more of the source content.

The inversion step maps the source audio to a noise-space representation that preserves content information (temporal structure, onset patterns); the regeneration step reinterprets this representation through the lens of the target timbre. The fidelity of content preservation depends on the accuracy of the DDIM inversion, which in turn depends on the number of inversion steps (more steps yield more faithful inversion).

Proposition 43 (Content preservation under DDIM inversion).

Let 𝒛T=DDIMInv(𝒛1,N) denote the N-step DDIM inversion of a clean audio latent 𝒛1, and let 𝒛^1=DDIMRev(𝒛T,N,𝒄) denote the N-step DDIM reverse with conditioning 𝒄. Define the content preservation error Δ(N)=𝒛^1𝒛1 when 𝒄 equals the original conditioning. Then:

  1. Δ(N)=O(1/N2) as N (assuming the velocity field is Lipschitz continuous with constant L).

  2. For finite N, Δ(N)L2/(2N2)𝒛1T2, where T is the terminal noise level.

Proof.

The DDIM forward and reverse processes are both first-order Euler discretisations of the same ODE, traversed in opposite directions. By standard ODE theory, the one-step local truncation error of the Euler method applied to an ODE with Lipschitz-L right-hand side is O(h2), where h=T/N is the step size. The global error after N steps accumulates to O(h)=O(T/N) in general. However, DDIM inversion followed by DDIM reversal is a round-trip: the forward discretisation error at each step is approximately cancelled by the reverse discretisation error at the corresponding step (since both use the same step size and the same velocity field evaluated at nearly the same point). The residual round-trip error arises from the second-order term in the Taylor expansion, giving Δ(N)=O(h2)=O(T2/N2).

For the explicit bound, the local round-trip error at step n is bounded by (L2h2/2)𝒛tn, where 𝒛tn𝒛1 (approximately, for variance-preserving schedules). Summing over N steps (with the round-trip cancellation ensuring no error accumulation beyond the local terms) gives Δ(N)N(L2h2/2)𝒛1=L2T2𝒛1/(2N2)(1/N)N=L2T2𝒛1/(2N2). (The extra factor of N in the numerator from summing N terms is cancelled by the round-trip cancellation, which means errors do not propagate multiplicatively between steps.)

Genre transfer.

Genre transfer generalises timbre transfer to higher-level musical attributes: converting a jazz piano trio performance into a rock band arrangement, or transforming an electronic dance track into an acoustic folk rendition. This requires changing not only the timbres but also the rhythmic patterns, harmonic voicings, and production style. The constrained sampling framework in (Constrained Style) applies, but the content distance dcontent must preserve only the most abstract features (overall harmonic progression, melodic contour, song structure) while allowing the model freedom to modify lower-level details.

Historical Note.

From signal processing to generative separation. Source separation has a long history in signal processing, dating back to the cocktail party problem articulated by Cherry in 1953. Classical approaches, including independent component analysis (ICA), non-negative matrix factorisation (NMF), and Wiener filtering, rely on hand-crafted signal models. The deep learning era brought supervised separation networks (e.g., Wave-U-Net, Open-Unmix, Demucs), which train discriminatively on paired mixture-source data. The generative approach studied here represents a third paradigm: using pretrained generative priors for each source, with no need for paired training data. This “zero-shot” separation capability is a direct consequence of the Bayesian duality between generation and inference, and it improves automatically as the generative models improve, without retraining the separation system.

0.50.4pt

Exercise 15 (Flow matching vs. diffusion for audio).

Consider a 1D audio latent space (d=1) with data distribution p1=Normal(3,1) and source distribution p0=Normal(0,1).

  1. Compute the optimal transport map T: from p0 to p1.

  2. Write down the exact velocity field v(z,t) for the linear interpolation flow matching model with OT coupling. Verify that it is linear in z for all t.

  3. Compute the probability flow ODE velocity field for a variance-preserving diffusion model with schedule αt=cos(πt/2), σt=sin(πt/2). Show that it is not linear in z.

  4. Numerically integrate both velocity fields using the Euler method with N=1,2,5,10,50 steps, starting from z0=0. Compare the endpoint errors and verify that the flow matching field requires fewer steps for a given accuracy.

Exercise 16 (Consistency distillation loss).

Let fθ(𝒛t,t)=cskip(t)𝒛t+cout(t)Fθ(𝒛t,t) be a consistency function parameterised as in (Consistency Param), with cskip(t)=σt2/(σt2+σϵ2) and cout(t)=σϵσt/σt2+σϵ2, where σt is the noise level at time t and σϵ is the noise level at the boundary t=ϵ.

  1. Verify that fθ(𝒛ϵ,ϵ)=𝒛ϵ (the boundary condition).

  2. Show that the consistency distillation loss (CD LOSS) with d(𝒂,𝒃)=𝒂𝒃2 reduces to CD=𝔼[(cout(tn+1)Fθ(𝒛tn+1,tn+1)+cskip(tn+1)𝒛tn+1cout(tn)Fθ(𝒛^tn,tn)cskip(tn)𝒛^tn)2].

  3. Explain why the EMA target network (θ) is necessary for stable training. What happens if θ=θ (no EMA)?

Exercise 17 (Overlap-add analysis for long-form audio).

A music generation model produces chunks of L=750 latent tokens with an overlap of δ tokens. The model's temporal autocorrelation decays exponentially with decay constant γ=0.05 tokens1.

  1. Using Proposition 41, derive the minimum overlap δ such that the boundary discontinuity D(δ)103𝔼[𝒙(k)2].

  2. Compute the number of chunks K needed to generate a 3-minute song at 24,kHz with compression factor 320, using overlap δ.

  3. A raised-cosine cross-fade αn=12(1cos(πn/δ)) replaces the linear cross-fade in (Crossfade). Repeat the cross-fade error analysis and show that the raised-cosine window achieves a tighter bound: Ecfcos3D(δ)/16.

Exercise 18 (Bayesian source separation).

Consider the two-source separation problem with mixture 𝒙mix=𝒙1+𝒙2, where 𝒙1Normal(𝝁1,σ12𝐈) and 𝒙2Normal(𝝁2,σ22𝐈) are independent Gaussian sources in d.

  1. Derive the exact posterior p(𝒙1|𝒙mix) and show that it is Gaussian. Compute its mean and covariance.

  2. The posterior mean is the Wiener filter estimate. Express it in terms of 𝝁1, 𝝁2, σ12, σ22, and 𝒙mix.

  3. Now suppose 𝒙1 has a non-Gaussian prior (e.g., a mixture of Gaussians representing different instrument timbres). Explain how the diffusion-based separation procedure (project-and-denoise) of Source Separation as Inverse Generation approximates the true posterior, and discuss when it might fail.

  4. Derive the equal-redistribution projection (Mixture Projection) for the K-source case as the orthogonal projection onto the affine subspace {k𝒙k=𝒙mix}. Verify that the residual k𝒙~k𝒙mix=0 exactly.

Cross-Modal Generation

The previous sections addressed audio and music generation conditioned on text prompts, tags, or melodic hints. In many practical applications, however, the conditioning signal originates from an entirely different modality: a video stream that requires a matching soundtrack, a set of lyrics that must be sung with correct timing, or an existing audio clip that must be extended, inpainted, or super-resolved. Each of these scenarios demands architectures that bridge representational gaps between modalities while preserving the fine-grained temporal structure that characterises audio.

This section formalises three families of cross-modal audio generation: video-to-music (Video-to-Music Generation), lyrics-conditioned singing synthesis (Lyrics-Conditioned Generation), and audio-to-audio tasks such as inpainting and editing (Audio-to-Audio: Inpainting and Editing).

Video-to-Music Generation

A background music track must reflect the visual content, the emotional tone, and the temporal dynamics of its accompanying video. We begin with a precise problem statement.

Definition 54 (Video-Conditioned Music Generation).

Let 𝐕=(𝒗1,,𝒗F)F×dv be a sequence of F visual feature vectors extracted from a video at frame rate rv, and let 𝒙T denote a waveform of duration T/fs seconds at sample rate fs. The video-conditioned music generation problem is to learn a conditional distribution (V2M Marginal)pθ(𝒙|𝐕)=pθ(𝒙|𝒛)pθ(𝒛|𝐕)d𝒛, where 𝒛 denotes a latent representation (e.g., discrete audio tokens or a continuous latent spectrogram) and θ collects all learnable parameters. The model must satisfy the temporal alignment constraint: musically salient events (beat onsets, harmonic changes, dynamic swells) should correlate with visually salient events (scene cuts, motion peaks, lighting changes).

Visual feature extraction.

The visual encoder ϕv:H×W×3dv is typically a pretrained vision transformer (e.g., CLIP ViT-L/14 or DINOv2). For a video of F frames, we obtain 𝐕=(ϕv(𝐈1),,ϕv(𝐈F)). Two strategies dominate the literature:

  1. Frame-level features. Each frame is encoded independently, yielding 𝐕F×dv. This preserves fine-grained temporal information but ignores inter-frame motion.

  2. Clip-level features. A video encoder such as VideoMAE or TimeSformer processes short clips (e.g., 16 frames) and produces a single vector per clip. This captures motion but reduces temporal resolution.

In practice, Diff-BGM [27] and VidMuse [28] combine both: frame-level features provide a dense temporal signal, while a clip-level summary vector captures global scene semantics.

Temporal alignment mechanisms.

The mismatch between video frame rate (rv, typically 24 fps) and audio token rate (ra, often 50 Hz for neural codec tokens) necessitates an alignment module. Let 𝐀La×F denote a soft alignment matrix, where La=Tra/fs is the number of audio tokens. The aligned visual features are (V2M Align)𝐕~=𝐀𝐕La×dv, so that each audio token position receives a weighted combination of visual features. Diff-BGM learns 𝐀 via cross-attention within a latent diffusion model; VidMuse employs a dedicated temporal alignment network that predicts 𝐀 from rhythm and motion energy signals.

Key Idea.

Rhythm-aware alignment. Music is organised around a beat grid, while video has no such intrinsic periodicity. Effective video-to-music systems first estimate the target tempo (beats per minute) from the visual motion energy, then construct 𝐀 so that beat positions in the audio coincide with high-motion frames in the video. This produces soundtracks that “breathe” with the visual dynamics.

Diff-BGM.

Diff-BGM [27] uses a two-stage approach:

  1. Video analysis. A pretrained scene classifier and motion energy estimator produce a scene label sequence and a continuous arousal curve a(t)[0,1].

  2. Conditional latent diffusion. A U-Net denoiser operates on mel-spectrogram latents 𝒛C×T (where T is the compressed time dimension), conditioned on both the scene label embedding and the arousal curve via cross-attention and FiLM layers.

The training objective follows the standard latent diffusion loss: (Diffbgm LOSS)Diff-BGM=𝔼t,𝝐,𝒛0[𝝐𝝐θ(𝒛t,t,𝐕)22], where 𝒛t=αt𝒛0+1αt𝝐 is the noised latent and 𝐕 summarises the visual conditioning.

VidMuse.

VidMuse [28] extends the framework to long-form video by introducing a hierarchical conditioning scheme. A global video embedding captures scene-level semantics (genre, mood), while local frame-level embeddings guide temporal details. The generation model is a transformer-based language model over EnCodec tokens, with the visual features injected via cross-attention at every layer: (Vidmuse AR)pθ(𝒙|𝐕)=n=1Npθ(xn|x<n,𝐕~), where xn denotes the n-th audio token and 𝐕~ is the temporally aligned visual feature sequence from (V2M Align).

Video-to-music generation pipeline. Visual features are extracted per frame, temporally aligned to the audio token rate, and injected into the generation model via cross-attention. The final waveform is produced by an audio decoder (vocoder or neural codec decoder).

Remark 42 (Connection to audio-visual generation).

Video-to-music generation is one direction of the broader audio-visual generation problem. The reverse direction (music-to-video or audio-driven video synthesis) is treated in sec:vdiff:app:audiovisual. The two directions share the temporal alignment challenge but differ in their output modalities and perceptual evaluation criteria.

Proposition 44 (Alignment optimality).

Let 𝐕F×dv and 𝐇La×dh denote the visual feature matrix and the audio hidden state matrix, respectively. The soft alignment matrix that maximises the total cross-modal attention energy is (Align OPT)𝐀=softmax(𝐇𝐖q𝐖k𝐕/dk), where 𝐖qdh×dk and 𝐖kdv×dk are the query and key projection matrices and softmax is applied row-wise. Each row of 𝐀 is a probability distribution over video frames, giving a probabilistic correspondence between each audio token and the visual context.

Proof.

The cross-modal attention energy for audio position n is en=𝒉n𝐖q𝐖k𝐕𝒂n, where 𝒂nΔF1 is the n-th row of 𝐀. We wish to find 𝒂n that maximises en subject to 𝒂n0 and 𝟙𝒂n=1. Let 𝒔n=𝐕𝐖k𝐖q𝒉nF. Then en=𝒔n𝒂n, which is a linear function of 𝒂n over the simplex. The maximum of a linear function over a simplex is attained at a vertex (hard attention). The softmax with temperature dk provides the standard smooth relaxation used in practice, approaching the hard maximum as dk0+.

Lyrics-Conditioned Generation

Generating singing voice that matches given lyrics requires solving two intertwined problems: synthesising intelligible vocal content and producing musically coherent melodic and rhythmic patterns. This subsection analyses the conditioning mechanisms, alignment strategies, and the interplay with large-scale music models such as Jukebox (sec:vae2:case-jukebox).

Character-level vs. word-level conditioning.

The choice of text granularity has significant architectural implications:

  • Character-level conditioning. Each character (or phoneme) is treated as a separate token, enabling fine-grained control over pronunciation and duration. The conditioning sequence is long, which increases the computational cost of cross-attention but allows the model to learn sub-word timing patterns (e.g., melisma, where a single syllable spans multiple notes).

  • Word-level conditioning. Each word is mapped to a single embedding vector, reducing the conditioning sequence length. This simplifies the alignment problem but sacrifices control over intra-word timing. Word-level models typically require a separate duration model to predict how many audio frames correspond to each word.

Definition 55 (Lyrics-to-Singing Alignment).

Let 𝒍=(l1,,lM) be a sequence of M linguistic units (characters or phonemes) and 𝒙=(x1,,xN) be a sequence of N audio frames. A lyrics-to-singing alignment is a monotone non-decreasing function α:{1,,N}{1,,M} such that α(n) gives the linguistic unit active at audio frame n. Equivalently, α defines a partition of {1,,N} into M contiguous segments, where segment m has duration dm=|{n:α(n)=m}|.

The monotonicity constraint reflects the fact that, in normal singing, lyrics are produced in order. The duration vector 𝒅=(d1,,dM)>0M with mdm=N is the natural parameterisation of the alignment.

Duration modelling.

Given lyrics 𝒍 and a target melody contour 𝒇0=(f0,1,,f0,N) (fundamental frequency), a duration model predicts the duration of each phoneme: (Duration Model)𝒅=gψ(𝒍,𝒇0)>0M, where gψ is typically a small transformer or convolutional network. The predicted durations are rounded and used to expand the phoneme embeddings to the audio frame rate via repetition: (Expand)l~n=lα(n),where α(n)=min{m:j=1mdjn}.

Jukebox and VQ-VAE approaches.

Jukebox [29] demonstrated that lyrics can be incorporated as conditioning for a hierarchical VQ-VAE music model. The lyrics are encoded by a small transformer and injected via addition to the audio token embeddings at each level of the hierarchy. While Jukebox produces impressive results, the alignment between lyrics and audio is largely learned implicitly, without an explicit duration model. This leads to occasional misalignment artefacts (skipped words, repeated phrases) that more structured approaches avoid.

Example 28 (Lyrics conditioning in Jukebox).

In Jukebox, the lyrics encoder produces a sequence of embeddings 𝐄lM×d. These are aligned to the audio token sequence via a learned attention mechanism at each level of the VQ-VAE hierarchy. At the top level (lowest resolution, 8 Hz), the attention is coarse and captures phrase-level alignment. At the bottom level (highest resolution, 44.1 kHz), the attention refines to phoneme-level alignment. The hierarchical structure allows the model to capture both global lyrical structure and local pronunciation details.

Insight.

Monotonic attention for lyrics alignment. Standard soft attention allows the model to attend to any position in the lyrics at any point in time, which can produce non-monotonic alignments (e.g., singing words out of order). Enforcing monotonicity through mechanisms such as monotonic attention [3] or the Connectionist Temporal Classification (CTC) loss significantly improves alignment quality. The trade-off is reduced flexibility: monotonic attention cannot handle musical structures where lyrics are intentionally repeated or reordered (as in some choral works).

Audio-to-Audio: Inpainting and Editing

Audio-to-audio generation encompasses tasks where both the input and output are audio signals: inpainting missing regions, extending existing clips, performing super-resolution, and making localised edits. These tasks are unified by conditioning the generation on a partially observed audio signal.

Definition 56 (Audio Inpainting).

Let 𝒙T be a complete audio signal and let Ωobs{1,,T} denote the set of observed sample indices, with Ωmiss={1,,T}Ωobs the missing indices. The audio inpainting problem is to sample from the conditional distribution (Inpainting EDIT)𝒙Ωmissp(𝒙Ωmiss|𝒙Ωobs,𝒄), where 𝒄 is an optional auxiliary conditioning signal (text description, surrounding context features). The inpainted signal must be (i) perceptually natural, (ii) temporally consistent with the observed regions at the boundaries, and (iii) semantically consistent with the conditioning 𝒄.

Diffusion-based inpainting.

In the diffusion framework, inpainting is implemented by constraining the observed region at each denoising step. Let 𝒛t=αt𝒛0+1αt𝝐 be the forward diffusion at time t. At each reverse step, after computing the denoised estimate 𝒛^0 from the network prediction, we replace the observed region with the corresponding noised observation: (Inpaint Replace)𝒛t1(n)={αt1𝒛0obs+1αt1𝝐nif nΩobs,𝒛^t1(n)if nΩmiss, where 𝒛^t1 is the standard reverse-step output and 𝝐nNormal(0,1) is fresh noise. This repaint strategy [4] ensures the observed region remains faithful to the input throughout the denoising process.

Remark 43 (Boundary artefacts in naive inpainting).

The replacement strategy in (Inpaint Replace) can create discontinuities at the boundary between Ωobs and Ωmiss, because the generated region and the observed region evolve under different noise schedules during the reverse process. Several remedies exist: (i) applying a smooth transition mask at the boundary, (ii) resampling several denoising steps (the “repaint” jumps of [4]), or (iii) using a harmonisation network that post-processes the boundary.

Audio super-resolution.

Super-resolution is a special case of audio-to-audio generation where Ωobs corresponds to the low-frequency content. Given a band-limited signal 𝒙lowT with bandwidth Blow, the goal is to synthesise the high-frequency band [Blow,Bhigh] conditioned on 𝒙low. In the frequency domain, this can be formulated as (Superres)X^(f)={Xlow(f)if |f|Blow,Gθ(f|Xlow)if Blow<|f|Bhigh, where Gθ is a generative model for the missing bandwidth.

Audio extension (outpainting).

Audio extension (also called outpainting or continuation) generates new audio that seamlessly continues an existing clip. Formally, given 𝒙1:T0, we wish to sample 𝒙T0+1:Tp(𝒙T0+1:T|𝒙1:T0,𝒄). Autoregressive models handle this naturally by simply continuing the token generation. For diffusion models, the continuation is treated as a right-boundary inpainting problem: the observed region is Ωobs={1,,T0} and the missing region is Ωmiss={T0+1,,T}.

Example 29 (Instruction-based audio editing).

Instruction-based audio editing systems (e.g., InstructME, AUDIT) accept a source audio clip and a natural language instruction such as “remove the drums” or “add reverb to the vocals.” The system must identify the relevant audio region, apply the requested modification, and leave the remaining content unchanged. This requires a joint understanding of audio semantics and natural language, typically achieved by conditioning a diffusion model on both the source audio and a text embedding of the instruction: (Instruct EDIT)𝒙editpθ(𝒙|𝒙source,𝒄instr). The conditioning on 𝒙source is implemented via concatenation in the input channels of the U-Net denoiser, while 𝒄instr is injected through cross-attention.

Proposition 45 (Inpainting as constrained posterior sampling).

Let p(𝒙) be the prior distribution of complete audio signals. The audio inpainting posterior decomposes as (Inpaint Bayes)p(𝒙Ωmiss|𝒙Ωobs)=p(𝒙Ωobs,𝒙Ωmiss)p(𝒙Ωobs)=p(𝒙)p(𝒙Ωobs). When the prior p(𝒙) is modelled by a diffusion model, sampling from this posterior requires running the reverse process while constraining the observed indices. The replacement strategy in (Inpaint Replace) approximates this posterior sampling by enforcing the observation constraint at each step.

Proof.

The decomposition follows directly from the definition of conditional probability. For the diffusion approximation, note that at each reverse step t, the distribution of 𝒛t1 given 𝒛t and the observation 𝒛0obs factorises over the observed and missing regions under the Gaussian noise model: (Inpaint Factor)p(𝒛t1|𝒛t,𝒛0obs)=p(𝒛t1miss|𝒛t)p(𝒛t1obs|𝒛0obs)=pθ(𝒛t1miss|𝒛t)q(𝒛t1obs|𝒛0obs), where q(𝒛t1obs|𝒛0obs) is the forward process marginal, which is Gaussian and can be sampled exactly. The replacement step realises this factorisation.

Symbolic Music Generation

All generation methods discussed so far operate on audio waveforms or their spectral representations. An older but complementary tradition represents music symbolically, as sequences of discrete note events. Symbolic representations capture pitch, timing, duration, and velocity with exact precision, at the cost of discarding timbre, room acoustics, and performance nuances. This section covers the dominant symbolic representation (MIDI), transformer-based generation models, and recent diffusion approaches that operate directly on piano-roll matrices.

MIDI and Symbolic Representations

The Musical Instrument Digital Interface (MIDI) standard, introduced in 1983, remains the most widely used symbolic music representation. A MIDI file encodes a piece of music as a sequence of timestamped events.

Definition 57 (MIDI Event Sequence).

A MIDI event sequence is an ordered list of events 𝒆=(e1,e2,,eL), where each event ei is a tuple (MIDI Event)ei=(τi,typei,pitchi,veli,chi), with:

  • τi0: the timestamp (in ticks or seconds),

  • typei{𝚗𝚘𝚝𝚎_𝚘𝚗,𝚗𝚘𝚝𝚎_𝚘𝚏𝚏,𝚌𝚘𝚗𝚝𝚛𝚘𝚕_𝚌𝚑𝚊𝚗𝚐𝚎,}: the event type,

  • pitchi{0,1,,127}: the MIDI pitch number (where 60 = middle C, 69 = A4 = 440,Hz),

  • veli{0,1,,127}: the velocity (loudness), and

  • chi{0,1,,15}: the MIDI channel (instrument track).

Events are ordered by timestamp: τ1τ2τL.

Piano roll representation.

For computational purposes, it is often convenient to convert a MIDI event sequence into a piano roll: a binary matrix that indicates which pitches are active at each time step.

Definition 58 (Piano Roll).

Given a MIDI event sequence with a fixed time quantisation of Δt (e.g., a sixteenth note), the piano roll is a binary matrix (Pianoroll)𝐏{0,1}T×128, where T=τL/Δt is the number of time steps and Pt,p=1 if and only if pitch p is sounding at time step t. The piano roll can be augmented with velocity information by replacing binary values with Pt,p=vt,p/127[0,1], yielding a continuous-valued piano roll.

Piano roll representation of a short musical passage. Each filled rectangle indicates an active note: blue blocks represent chord tones and orange blocks represent the melody. The horizontal axis is quantised time, the vertical axis is MIDI pitch. The matrix 𝐏{0,1}T×128 is a binary encoding of this visualisation.
Alternative symbolic encodings.

Several alternatives to the raw MIDI event sequence have been proposed to improve the efficiency and learnability of symbolic music models:

  • REMI (REvamped MIDI-derived) [5]: Introduces explicit bar and position tokens, encoding each note as a sequence of sub-tokens: [Bar], [Position], [Pitch], [Duration], [Velocity]. This makes the bar structure explicit and improves long-range coherence.

  • Compound Word (CP) [6]: Groups the sub-tokens of REMI into a single compound token, where each compound token is a concatenation of embeddings for position, pitch, duration, and velocity. This reduces the sequence length by a factor of 4–5 compared to REMI.

  • Octuple [7]: Extends the compound word idea to eight attributes (time, bar, position, instrument, pitch, duration, velocity, tempo), enabling multi-track representation in a single token per note.

Remark 44 (Sequence length trade-offs).

A 4-minute piano piece at sixteenth-note resolution contains roughly T=960 time steps. In the raw MIDI event representation, each note requires at least two events (note_on and note_off), and a typical piece has 1000–3000 notes, yielding sequences of 2000–6000 tokens. The REMI encoding expands this further (5 sub-tokens per note), while the Compound Word encoding compresses it back to one compound token per note. These sequence lengths are well within the capacity of modern transformers with relative or rotary positional encodings, but they highlight the importance of efficient tokenisation for symbolic music.

Transformer Models for Symbolic Music

The application of transformer architectures to symbolic music generation has been remarkably successful. The key insight is that a MIDI token sequence, after appropriate encoding, is structurally similar to a natural language sentence: it is a sequence of discrete tokens with both local dependencies (notes within a chord) and long-range dependencies (harmonic progressions, melodic repetitions, verse-chorus structures). We refer the reader to ch:autoreg for the general treatment of autoregressive models.

Music Transformer.

The Music Transformer [30] adapts the transformer decoder architecture for symbolic music generation, with one crucial modification: relative attention.

Standard absolute positional encodings (sinusoidal or learned) add a fixed position-dependent bias to each token. In music, however, the important relationships are often relative: a perfect fifth interval (7 semitones) has the same harmonic character regardless of its absolute pitch, and a rhythmic pattern at the beginning of a phrase has the same metrical function when it recurs four bars later.

Definition 59 (Relative Attention for Music).

Let 𝐐,𝐊,𝐕L×d be the query, key, and value matrices for a sequence of length L. The relative attention mechanism computes (Relattn)RelAttn(𝐐,𝐊,𝐕)=softmax(𝐐𝐊+𝐒reld)𝐕, where 𝐒relL×L is the relative position bias matrix with entries (Relattn Entry)Si,jrel=𝒒i𝒓ji, and 𝒓kd is a learned embedding for relative position offset k. To keep the number of parameters finite, offsets are clipped to [Kmax,Kmax].

Proposition 46 (Relative attention captures periodic structure).

Let 𝒒i=𝒒 for all i (a constant query, representing a repeated motif) and suppose the relative embeddings are periodic: 𝒓k+P=𝒓k for some period P and all k. Then the relative attention scores satisfy (Periodic Score)Si,jrel=Si,j+Prelfor all i,j, i.e., the attention pattern is periodic with the same period P. In particular, if P corresponds to one bar of music, the model naturally attends to the same metrical position in all bars.

Proof.

By definition, Si,jrel=𝒒𝒓ji and Si,j+Prel=𝒒𝒓j+Pi=𝒒𝒓ji+P=𝒒𝒓ji, where the last equality uses the periodicity of 𝒓.

Compound word tokenisation.

As noted in the previous subsection, the Compound Word (CP) encoding [6] represents each musical note as a single compound token, where each compound token consists of multiple attribute embeddings that are summed or concatenated: (Compound)𝒆nCP=𝒆npos+𝒆npitch+𝒆ndur+𝒆nvel, where each 𝒆n()d is a learned embedding for the corresponding attribute. The transformer then operates on the compound sequence (𝒆1CP,,𝒆NCP), with separate prediction heads for each attribute at the output.

Example 30 (Pop music generation with compound words).

Pop Piano Transformer [6] trains a 12-layer transformer on the POP909 dataset (909 pop songs) using CP encoding. At generation time, the model produces a sequence of compound tokens autoregressively, predicting all attributes of each note in parallel at each step. The generated pieces exhibit clear verse-chorus structure, appropriate chord progressions, and rhythmic consistency over spans of 30–60 seconds. Beyond this duration, the model tends to drift harmonically, a limitation shared by most fixed-context-window autoregressive approaches.

Longer context and hierarchical generation.

Standard transformer architectures with context window W can capture dependencies spanning at most W tokens. For a CP-encoded piece, W=2048 covers approximately 2 minutes of music. To generate longer pieces, several strategies have been explored:

  1. Sliding window with memory. The model generates in overlapping windows, carrying forward a compressed memory (e.g., the hidden states of the last k tokens).

  2. Hierarchical generation. A high-level model generates a structural plan (e.g., chord progression, section labels), and a low-level model fills in the notes conditioned on this plan. This decomposes long-range structure from local note-level detail.

  3. Sparse attention. Attention patterns such as Longformer or BigBird allow the model to attend to distant tokens at reduced computational cost, extending the effective context to tens of thousands of tokens.

Diffusion for Symbolic Music

While autoregressive transformers have been the dominant paradigm for symbolic music, diffusion models offer an attractive alternative: they can generate all time steps in parallel and naturally handle tasks like inpainting and arrangement. The challenge is that the piano roll 𝐏{0,1}T×128 is a discrete object, whereas standard diffusion operates on continuous spaces.

Definition 60 (Symbolic Music Diffusion).

Let 𝐏[0,1]T×128 be a relaxed (continuous) piano roll, obtained by treating each entry as the probability that pitch p is active at time t. The symbolic music diffusion model defines a forward process (SYM Forward)q(𝐏t|𝐏0)=Normal(𝐏t;αt𝐏0,(1αt)𝐈) and a learned reverse process (SYM Reverse)pθ(𝐏t1|𝐏t)=Normal(𝐏t1;𝝁θ(𝐏t,t),σt2𝐈), where 𝝁θ is parameterised by a neural network (typically a U-Net operating on the T×128 “image”). At generation time, the continuous output 𝐏0T×128 is binarised by thresholding at 0.5 to recover a valid piano roll.

Challenges of the discrete nature.

The relaxation from {0,1} to [0,1] introduces a mismatch between the training data (binary piano rolls) and the diffusion process (continuous). Several issues arise:

  1. Boundary effects. The Gaussian noise process can push values outside [0,1], requiring clipping or a sigmoid transformation.

  2. Thresholding artefacts. The binarisation step at generation time is non-differentiable and can produce noisy results near the 0.5 threshold.

  3. Sparse activations. A typical piano roll is very sparse: at any given time step, at most 10–15 of the 128 pitches are active (and usually fewer than 6). The diffusion model must learn to produce sharp, sparse outputs from a dense Gaussian process.

Remark 45 (Discrete diffusion alternatives).

An alternative to the continuous relaxation is to use discrete diffusion models that operate directly on binary or categorical data; see sec:diffusion_language for a general treatment. D3PM [8] and related methods define forward processes that corrupt discrete tokens via transition matrices rather than Gaussian noise. For piano rolls, the absorbing-state variant is natural: each pitch independently transitions to an “unknown” state with probability that increases over the forward process, and the reverse process learns to reconstruct the original binary values.

Architecture considerations.

The piano roll 𝐏{0,1}T×128 is a two-dimensional structure that can be treated as a single-channel image. This observation motivates the use of 2D U-Net architectures, where the “height” dimension is pitch and the “width” dimension is time. However, the two axes have very different semantics:

  • The pitch axis has harmonic structure: notes separated by octaves (12 semitones) have similar function, and common intervals (thirds, fifths, octaves) have characteristic sizes.

  • The time axis has rhythmic structure: events are organised around a metrical grid (bars, beats, subdivisions).

Effective architectures exploit these distinct structures through asymmetric kernel sizes, separate pitch and time attention layers, or hybrid convolutional-attention designs.

Proposition 47 (Sparsity-aware loss for piano roll diffusion).

Let ρ=𝔼[Pt,p] denote the average activation rate of the piano roll, with ρ1 in practice (ρ0.03 for typical piano music). The standard MSE diffusion loss 𝝐𝝐θ(𝐏t,t)22 treats active and inactive entries equally. A reweighted loss that upweights active entries by w+=(1ρ)/ρ improves reconstruction of the sparse active notes: (Sparse LOSS)sparse=𝔼t,𝝐[t,pw(Pt,p(0))(ϵt,pϵθ;t,p(𝐏t,t))2], where w(1)=w+=(1ρ)/ρ and w(0)=1. This is equivalent to optimising a weighted Gaussian log-likelihood where the variance is scaled inversely with class frequency.

Proof.

The standard MSE loss assigns equal weight to all T×128 entries. Since a fraction ρ of entries are active and 1ρ are inactive, the gradient contribution from active entries is proportional to ρ, which is small. Setting w+=(1ρ)/ρ ensures that the total weight on active entries equals the total weight on inactive entries: ρw+=ρ(1ρ)/ρ=1ρ. This balances the gradient contributions and prevents the model from learning a trivial all-zeros prediction.

Example 31 (Piano roll diffusion for accompaniment generation).

Consider generating an accompaniment piano roll 𝐏acc conditioned on a given melody piano roll 𝐏mel. This is a conditional generation task where the conditioning signal has the same dimensionality as the output. The conditioning is implemented by concatenating 𝐏mel as an additional input channel to the U-Net: (Accomp COND)𝝐θ(𝐏tacc,t,𝐏mel)=U-Net([𝐏tacc;𝐏mel],t), where [;] denotes channel-wise concatenation. The generated accompaniment respects the harmonic context of the melody, producing chords and bass lines that are harmonically consistent.

Commercial and Industrial Systems

The research advances described in previous sections have been translated into commercial products that can generate full-length songs, complete with vocals, lyrics, instrumentation, and mixing. This section surveys the major commercial and industrial systems as of 2024–2025, analyses their known (or inferred) architectures, and provides a comparative evaluation. We emphasise that many of these systems are closed-source, so the architectural details presented here are based on published papers, patent filings, blog posts, and empirical observation.

Caution.

Closed-source limitations. The commercial systems described in this section are proprietary. Architectural details are inferred from published information and may not reflect the actual implementations. Quantitative comparisons should be interpreted with caution, as the systems are continually updated and the evaluation conditions may differ. We include this section because these systems represent the state of the art in user-facing music generation and illustrate the practical engineering decisions that complement the theoretical foundations developed earlier.

Google MusicLM / MusicFX

MusicLM [31] was one of the first systems to demonstrate high-quality, long-form text-to-music generation. Its architecture exemplifies the hierarchical token generation paradigm that has become the dominant approach for audio generation at scale.

Architecture overview.

MusicLM uses a three-stage hierarchical generation process:

  1. Semantic tokens. A first transformer maps a text description to a sequence of semantic tokens extracted from a MuLan [9] audio embedding. MuLan is a joint text-audio embedding model (analogous to CLIP for text-image), trained on 44M audio-text pairs. The semantic tokens capture high-level musical attributes (genre, mood, instrumentation) at a coarse temporal resolution (25,Hz).

  2. Coarse acoustic tokens. A second transformer maps the semantic tokens to coarse acoustic tokens from a SoundStream [10] neural audio codec. These tokens encode the broad spectral shape and harmonic content at 50,Hz.

  3. Fine acoustic tokens. A third transformer (or the same transformer with a different head) refines the coarse tokens by predicting the residual quantisation levels of the codec, capturing fine spectral detail and high-frequency content.

The full generation pipeline can be written as (Musiclm Hierarchy)p(𝒙|𝒄text)=p(𝒔|𝒄text)semantic stagep(𝒂c|𝒔)coarse acousticp(𝒂f|𝒂c,𝒔)fine acousticp(𝒙|𝒂f,𝒂c)codec decoder, where 𝒔 denotes semantic tokens, 𝒂c coarse acoustic tokens, 𝒂f fine acoustic tokens, and 𝒙 the output waveform.

MuLan conditioning.

The MuLan model projects both text and audio into a shared embedding space dmu, trained with a contrastive loss: (Mulan LOSS)MuLan=1Bi=1Blogexp(ϕt(𝒄i)ϕa(𝒙i)/τ)j=1Bexp(ϕt(𝒄i)ϕa(𝒙j)/τ), where ϕt and ϕa are the text and audio encoders, B is the batch size, and τ is a temperature parameter. At generation time, the text is encoded by ϕt, and the resulting embedding conditions the semantic token generator.

MusicFX and Gemini-era updates.

Following MusicLM, Google released MusicFX as a user-facing product integrated with the Gemini family of models. Key improvements include:

  • Longer generation. MusicFX supports generation of up to 70 seconds (compared to MusicLM's 30 seconds), achieved through sliding-window generation with overlapping context.

  • Improved conditioning. The MuLan embedding is supplemented with T5 text embeddings, providing richer semantic conditioning for complex prompts.

  • Quality filtering. A MuLan-based quality classifier filters generated candidates, selecting the output with the highest text-audio alignment score. For a batch of K candidates, the selected output is (Quality Filter)𝒙=arg max𝒙k,k=1,,Kϕt(𝒄)ϕa(𝒙k).

Historical Note.

From MusicLM to MusicFX. MusicLM (January 2023) established the three-stage hierarchical paradigm and demonstrated that text-to-music generation could achieve quality comparable to real recordings for short clips. MusicFX (late 2023, updated 2024) refined this approach for production use, incorporating safety filters (to avoid generating copyrighted material or harmful content), latency optimisations (streaming the coarse-to-fine stages), and user interface features (prompt suggestions, style controls). The Gemini integration (2024–2025) further improved prompt understanding by leveraging the LLM's reasoning capabilities to expand and disambiguate user prompts before feeding them to the music generation pipeline.

Suno

Suno (versions 3 through 4) is a commercial system that generates complete songs with vocals, lyrics, and full instrumentation. It represents a significant leap in scope compared to earlier systems that focused on instrumental music or short clips.

Inferred architecture.

Based on public information and empirical analysis, Suno appears to employ a two-stage architecture:

  1. Structure planning. Given a text prompt (or user-provided lyrics), a language model generates a structural plan that specifies: itemize

  2. Song structure (intro, verse, chorus, bridge, outro),

  3. Lyrical content for each section (if not user-provided),

  4. Approximate duration and tempo,

  5. Genre and stylistic tags. itemize This stage can be understood as sampling from a conditional distribution over structural descriptions: (SUNO Structure)𝒔pθ1(𝒔|𝒄prompt), where 𝒔 encodes the song structure and 𝒄prompt is the user's input.

  6. Audio synthesis. A second model generates the audio waveform conditioned on the structural plan: (SUNO Synthesis)𝒙pθ2(𝒙|𝒔). This stage likely combines a transformer-based token generator (for coarse structure) with a diffusion-based refinement stage (for high-fidelity audio). The vocal track and instrumental track may be generated jointly or in separate streams that are mixed in a final step.

Full-song generation.

A distinguishing feature of Suno is its ability to generate songs of 2–4 minutes in length with coherent large-scale structure. This requires:

  • Long-range coherence. The model must maintain harmonic, melodic, and rhythmic consistency across the entire song. The structure planning stage addresses this by providing a high-level blueprint.

  • Vocal-instrumental alignment. The generated vocals must be synchronised with the instrumental accompaniment, both in timing (lyrics align with the beat) and in pitch (vocals follow the chord progression).

  • Dynamic range. Real songs have dynamics (quiet verses, loud choruses). The model must vary energy, density, and spectral content across sections.

Remark 46 (Training data considerations).

The training data for Suno reportedly includes a large corpus of licensed and/or public-domain music. The scale and composition of this dataset are significant factors in the system's performance. Copyright concerns surrounding AI-generated music trained on copyrighted material remain an active legal and ethical issue that is beyond the scope of this text.

Udio

Udio, developed by former members of Google DeepMind, is a competing commercial system that also generates full songs with vocals and instrumentation.

Architecture overview.

Udio's architecture is believed to centre on a diffusion-based generation model operating in a latent space, conditioned on text embeddings from a large language model. Key design choices (inferred from public demonstrations and interviews) include:

  • Latent diffusion. Audio is encoded into a compressed latent space via a neural audio codec, and the diffusion process operates in this latent space (similar to the approach of AudioLDM and Stable Audio).

  • LLM-based text understanding. The text prompt is processed by a large language model that extracts structured attributes (genre, mood, tempo, instrumentation) before conditioning the diffusion model.

  • Iterative refinement. The generation may involve multiple passes: a first pass generates a draft, and subsequent passes refine specific aspects (vocal clarity, instrumental balance, dynamic range).

Comparison with Suno.

While both systems target the same use case (full-song generation from text prompts or lyrics), empirical evaluations by users and independent reviewers suggest the following qualitative differences:

  • Vocal quality. Suno tends to produce more natural-sounding vocals with better pronunciation, while Udio excels at certain vocal styles (e.g., operatic, choral).

  • Instrumental variety. Udio shows greater diversity in instrumentation and arrangement, particularly for genres outside mainstream pop.

  • Structural coherence. Both systems occasionally produce songs with structural anomalies (e.g., abrupt endings, missing transitions), though the rate of such anomalies has decreased with successive versions.

Comparative Analysis

We summarise the key features of the major commercial and research systems in the following table.

1.2

SystemDur.VocalsConditioningArchitectureOpenYear
MusicLM30,sNoText (MuLan)Hier. ARNo2023
MusicGen30,sNoText (T5)Single ARYes2023
AudioLDM 210,sNoText (CLAP+GPT2)Latent diff.Yes2023
Stable Audio95,sNoText (CLAP)Latent diff.Partial2024
Suno v3/v4240,sYesText / lyricsAR + diff.No2024
Udio240,sYesText / lyricsLatent diff.No2024
MusicFX70,sNoText (MuLan+T5)Hier. ARNo2024
tableComparison of major music generation systems. “Duration” indicates the maximum generation length. “AR” denotes autoregressive. “Open Source” indicates whether model weights are publicly available. All systems support text-based conditioning; Suno and Udio additionally accept lyrics as input.

Remark 47 (Rapid evolution).

The commercial music generation landscape evolves rapidly. Between 2023 and 2025, the maximum generation duration increased from 30 seconds to over 4 minutes, vocal synthesis went from non-existent to near-human quality, and the number of controllable parameters (genre, mood, tempo, instrumentation, structure) expanded considerably. Any snapshot comparison, including the one above, is likely outdated by the time it is read. The reader is encouraged to consult online benchmarks and user evaluations for the most current comparisons.

Insight.

The two-stage pattern. A recurring architectural pattern in commercial systems is the separation of planning (generating a high-level structural description) from synthesis (generating the audio waveform conditioned on the plan). This mirrors the plan-then-execute paradigm in language model agents and reflects a fundamental principle: long-range coherence is better achieved by planning at a coarse temporal resolution than by attempting to maintain it through thousands of fine-grained autoregressive steps.

Evaluation Metrics for Audio Generation

Evaluating generated audio is challenging because perceptual quality, fidelity to a conditioning signal, and musical coherence are all multidimensional attributes that resist reduction to a single number. This section provides a rigorous treatment of the major evaluation metrics used in the audio generation literature, including their mathematical definitions, statistical properties, and practical limitations.

Fréchet Audio Distance

The Fréchet Audio Distance (FAD) is the audio analogue of the Fréchet Inception Distance (FID) used in image generation. It measures the distance between the distribution of real audio and the distribution of generated audio in a learned feature space.

Definition 61 (Fréchet Audio Distance).

Let {𝒙i(r)}i=1Nr and {𝒙j(g)}j=1Ng be sets of real and generated audio clips, respectively. Let ψ:Td be a feature extractor (e.g., VGGish) that maps each audio clip to a d-dimensional embedding. Compute the feature statistics: (FAD Stats R)𝝁r=1Nri=1Nrψ(𝒙i(r)),𝚺r=1Nr1i=1Nr(ψ(𝒙i(r))𝝁r)(ψ(𝒙i(r))𝝁r),𝝁g=1Ngj=1Ngψ(𝒙j(g)),𝚺g=1Ng1j=1Ng(ψ(𝒙j(g))𝝁g)(ψ(𝒙j(g))𝝁g). The Fréchet Audio Distance is (FAD)FAD=𝝁r𝝁g22+trace(𝚺r+𝚺g2(𝚺r𝚺g)1/2).

The matrix square root (𝚺r𝚺g)1/2 is the unique positive semidefinite matrix 𝐌 satisfying 𝐌2=𝚺r𝚺g. When 𝚺r and 𝚺g commute (i.e., share the same eigenvectors), this simplifies to the element-wise square root of their eigenvalue products.

Theorem 3 (FAD as W22 between Gaussians).

The FAD is equal to the squared 2-Wasserstein distance between two multivariate Gaussian distributions: (FAD W2)FAD=W22(Normal(𝝁r,𝚺r),Normal(𝝁g,𝚺g)). In particular, FAD0 with equality if and only if 𝝁r=𝝁g and 𝚺r=𝚺g.

Proof.

The squared 2-Wasserstein distance between two probability measures μ and ν on d is (W2 DEF)W22(μ,ν)=infγΓ(μ,ν)d×d𝒙𝒚22dγ(𝒙,𝒚), where Γ(μ,ν) is the set of all couplings of μ and ν. For Gaussians μ=Normal(𝝁1,𝚺1) and ν=Normal(𝝁2,𝚺2), the optimal transport map is the affine map T(𝒙)=𝝁2+𝐀(𝒙𝝁1), where 𝐀=𝚺11/2(𝚺11/2𝚺2𝚺11/2)1/2𝚺11/2. Substituting and simplifying (see ch:sinkhorn for the general theory of optimal transport), we obtain (W2 Gaussian)W22(μ,ν)=𝝁1𝝁222+trace(𝚺1+𝚺22(𝚺11/2𝚺2𝚺11/2)1/2)=𝝁1𝝁222+trace(𝚺1+𝚺22(𝚺1𝚺2)1/2), where the last equality uses the identity trace((𝚺11/2𝚺2𝚺11/2)1/2)=trace((𝚺1𝚺2)1/2), which follows from the cyclic property of the trace and the fact that (𝐁𝐁)1/2 and (𝐁𝐁)1/2 have the same nonzero eigenvalues for any matrix 𝐁. Setting (𝝁1,𝚺1)=(𝝁r,𝚺r) and (𝝁2,𝚺2)=(𝝁g,𝚺g) recovers (FAD).

Non-negativity follows from the general property W220. For the equality case, W22=0 if and only if the two distributions are identical, which for Gaussians requires 𝝁r=𝝁g and 𝚺r=𝚺g.

Feature extractors for FAD.

The choice of feature extractor ψ significantly affects the FAD score:

  • VGGish [11]: A CNN trained on AudioSet for audio event classification. This is the most common choice but may not capture music-specific features (melody, harmony, rhythm) effectively.

  • PANN (PANNs-CNN14): A larger CNN also trained on AudioSet, providing a richer feature space (d=2048) compared to VGGish (d=128).

  • CLAP embeddings: Using the audio encoder of a CLAP model as ψ yields a FAD variant that is sensitive to semantic audio attributes. This variant is sometimes called the “CLAP-FAD.”

  • EnCodec features: Using intermediate representations from a neural audio codec captures perceptual quality at a fine-grained level.

FAD computation pipeline. Real and generated audio clips are passed through a shared feature extractor ψ. The mean and covariance of each feature set are computed, and the FAD is calculated as the squared 2-Wasserstein distance between the fitted Gaussian distributions.

Remark 48 (Sample size effects on FAD).

The FAD estimate is biased: with finite samples, the covariance estimates 𝚺r and 𝚺g are noisy, leading to upward-biased FAD values. The bias scales as O(d/N) where d is the feature dimension and N is the sample size. For VGGish (d=128), at least N1000 samples are recommended to obtain stable FAD estimates. For higher-dimensional feature extractors, more samples are needed. The bias can be partially corrected using the formula (FAD BIAS)FAD^unbiasedFADdNrdNg, though this correction is only approximate and assumes the feature distributions are close to Gaussian.

Perceptual Quality Metrics

Unlike distributional metrics such as FAD that compare populations of audio clips, perceptual quality metrics evaluate individual audio signals against a reference or in isolation.

Definition 62 (PESQ: Perceptual Evaluation of Speech Quality).

The Perceptual Evaluation of Speech Quality (PESQ; ITU-T P.862) is a full-reference metric that compares a degraded signal 𝒙d against a clean reference 𝒙r. The PESQ algorithm operates as follows:

  1. Level and time alignment. Both signals are level-adjusted and temporally aligned using a correlation-based delay estimator.

  2. Perceptual modelling. Both signals are transformed to a perceptual domain using a model of the human auditory system, yielding loudness-density representations 𝐋r,𝐋dT×F (time-frequency with Bark-scale frequency resolution).

  3. Disturbance computation. The signed disturbance is 𝐃=𝐋d𝐋r, decomposed into positive (additive) and negative (subtractive) components.

  4. Aggregation. The disturbances are aggregated over time and frequency using Minkowski norms to yield the PESQ score: (PESQ)PESQ=a0+a1Dsym+a2Dasym, where Dsym and Dasym are the symmetric and asymmetric disturbance measures, and a0,a1,a2 are regression coefficients calibrated against human judgments.

The PESQ score ranges from 0.5 to 4.5, with higher values indicating better quality.

Definition 63 (ViSQOL: Virtual Speech Quality Objective Listener).

ViSQOL [12] is a full-reference metric designed for a broader range of audio signals (not just speech). It operates by:

  1. Computing gammatone spectrograms of the reference and degraded signals.

  2. Extracting neurogram similarity index measure (NSIM) patches, which compare local spectro-temporal patterns using a structural similarity index.

  3. Aggregating patch similarities across the signal to produce a MOS-LQO (Mean Opinion Score, Listening Quality Objective) estimate on a 1–5 scale.

Definition 64 (PEAQ: Perceptual Evaluation of Audio Quality).

PEAQ (ITU-R BS.1387) is a full-reference metric designed for general audio (music, speech, environmental sounds). It computes multiple model output variables (MOVs) including:

  • Bandwidth differences between reference and test signals,

  • Noise-to-mask ratio (NMR): the ratio of distortion energy to the masking threshold at each time-frequency point,

  • Harmonic structure of distortions,

  • Temporal envelope differences.

These MOVs are combined by a neural network trained on subjective test data to produce an Objective Difference Grade (ODG) on a scale from 4 (very annoying) to 0 (imperceptible difference).

Remark 49 (Applicability to generated audio).

PESQ, ViSQOL, and PEAQ are full-reference metrics: they require a clean reference signal to compare against. For audio generation tasks, a reference signal often does not exist (the generated audio is novel). These metrics are therefore most applicable to:

  • Codec evaluation. Comparing the output of a neural audio codec (e.g., EnCodec) against the original input.

  • Enhancement tasks. Evaluating denoising, super-resolution, or bandwidth extension where a clean reference is available.

  • Reconstruction quality. In VAE-based systems, comparing the reconstruction 𝒙^ against the original 𝒙.

For unconditional or text-conditioned generation, reference-free metrics (see below) or human evaluation are more appropriate.

CLAP Score

The CLAP (Contrastive Language-Audio Pretraining) score measures the alignment between a generated audio clip and its conditioning text prompt, analogous to the CLIP score in image generation.

Definition 65 (CLAP Score).

Let ϕt:𝒯dc and ϕa:Tdc be the text and audio encoders of a pretrained CLAP model, with dc the shared embedding dimension. The CLAP score for a text-audio pair (𝒄,𝒙) is the cosine similarity of their embeddings: (CLAP Score EVAL)CLAP(𝒄,𝒙)=ϕt(𝒄)ϕa(𝒙)ϕt(𝒄)2ϕa(𝒙)2[1,1]. The average CLAP score over a test set of N text-audio pairs is (CLAP AVG)CLAP=1Ni=1NCLAP(𝒄i,𝒙i).

The CLAP score is related to the CLAP contrastive loss (CLAP: Contrastive Language–Audio Pretraining) used during CLAP pretraining. A higher CLAP score indicates that the generated audio is more semantically aligned with the text description.

Proposition 48 (CLAP score and generation likelihood).

Under the assumption that the CLAP model defines a Gibbs distribution over audio given text, (CLAP Gibbs)pCLAP(𝒙|𝒄)exp(ϕt(𝒄)ϕa(𝒙)/τ), where τ>0 is the temperature, the CLAP score is proportional to the log-likelihood: (CLAP Loglik)CLAP(𝒄,𝒙)=τϕt(𝒄)2ϕa(𝒙)2(logpCLAP(𝒙|𝒄)+logZ(𝒄)), where Z(𝒄)=exp(ϕt(𝒄)ϕa(𝒙)/τ)d𝒙 is the partition function. This shows that maximising the CLAP score is equivalent to maximising the likelihood under the CLAP energy-based model, up to a prompt-dependent constant.

Proof.

From (CLAP Gibbs), logpCLAP(𝒙|𝒄)=ϕt(𝒄)ϕa(𝒙)/τlogZ(𝒄). Rearranging and dividing by ϕt(𝒄)2ϕa(𝒙)2: ϕt(𝒄)ϕa(𝒙)ϕt(𝒄)2ϕa(𝒙)2=τϕt(𝒄)2ϕa(𝒙)2(logpCLAP(𝒙|𝒄)+logZ(𝒄)), which is exactly (CLAP Loglik).

Mean Opinion Score

Despite the development of numerous objective metrics, subjective human evaluation remains the gold standard for assessing audio quality and musicality. The Mean Opinion Score is the most widely used protocol.

Definition 66 (Mean Opinion Score).

In a Mean Opinion Score (MOS) evaluation, K human raters each assign an integer score sk,i{1,2,3,4,5} to each of N audio stimuli, where the scale is:

ScoreQuality
5Excellent
4Good
3Fair
2Poor
1Bad
The MOS for stimulus i is the arithmetic mean over raters: (MOS)MOSi=1Kk=1Ksk,i. The overall MOS for a system is the grand mean MOS=1Ni=1NMOSi.

Confidence intervals.

Under the assumption that the ratings for stimulus i are i.i.d. draws from a distribution with mean μi and variance σi2, the 95% confidence interval for MOSi is (MOS CI)MOSi±1.96σ^iK, where σ^i=1K1k=1K(sk,iMOSi)2 is the sample standard deviation. The i.i.d. assumption is often violated in practice (raters have different internal scales and biases), which motivates more sophisticated analysis methods.

MUSHRA protocol.

The MUltiple Stimuli with Hidden Reference and Anchor (MUSHRA; ITU-R BS.1534) protocol is a variant designed for evaluating intermediate-quality audio (such as codec outputs or generated music). Key features include:

  • A hidden reference (the original high-quality audio) is included among the test stimuli.

  • A low-quality anchor (e.g., 3.5,kHz low-pass filtered version) is included to calibrate the scale.

  • Raters assign scores on a continuous 0–100 scale rather than a discrete 1–5 scale.

  • Raters can switch between stimuli in real time.

The MUSHRA score for stimulus i is (Mushra)MUSHRAi=1Kk=1Ksk,i,sk,i[0,100].

Statistical testing.

To determine whether two systems produce significantly different quality, we use the Wilcoxon signed-rank test, which is appropriate for paired ordinal data and does not assume normality.

Definition 67 (Wilcoxon Signed-Rank Test for MOS Comparison).

Given paired MOS scores (MOSi(A),MOSi(B)) for systems A and B on N stimuli, compute the differences di=MOSi(A)MOSi(B), discard any di=0, and rank the remaining |di| values. The test statistic is (Wilcoxon)W=i:di>0Ri, where Ri is the rank of |di|. Under the null hypothesis H0:median(di)=0, the distribution of W is known (tabulated for small N, approximately normal for large N with mean 𝔼[W]=N(N+1)/4 and variance 𝖵ar[W]=N(N+1)(2N+1)/24, where N is the number of non-zero differences). We reject H0 at significance level α if the p-value α.

Example 32 (MOS comparison of two music generation systems).

Suppose we evaluate systems A and B on N=50 prompts with K=20 raters each. The grand MOS scores are MOS(A)=3.82±0.12 and MOS(B)=3.61±0.14 (95% confidence intervals). The Wilcoxon signed-rank test on the per-stimulus MOS differences yields W=847, p=0.023. At the α=0.05 significance level, we reject the null hypothesis and conclude that system A produces significantly higher-rated output than system B. Note that the effect size is moderate: the MOS difference of 0.21 on a 5-point scale corresponds to roughly half a quality category.

Evaluation Summary

We summarise the major evaluation metrics and their properties in the following table.

1.2

MetricTypeReferenceMeasuresScaleAutomated
FADDistributionalPopulationFidelity[0,)Yes
CLAP ScorePairwiseNoneText alignment[1,1]Yes
PESQPairwiseFullSpeech quality[0.5,4.5]Yes
ViSQOLPairwiseFullAudio quality[1,5]Yes
PEAQ (ODG)PairwiseFullAudio quality[4,0]Yes
MOSAbsoluteNoneOverall quality[1,5]No
MUSHRAComparativeHiddenAudio quality[0,100]No
tableSummary of evaluation metrics for audio generation. “Type” describes the comparison paradigm. “Reference” indicates whether a clean reference signal is required. “Automated” indicates whether the metric can be computed without human raters.

Caution.

No single metric suffices. Each evaluation metric captures a different aspect of audio quality. FAD measures distributional similarity but is insensitive to per-sample quality variations. The CLAP score measures text-audio alignment but does not assess perceptual quality. PESQ and ViSQOL require a reference and focus on signal-level distortions rather than musical attributes. MOS captures overall quality but is expensive, subjective, and difficult to reproduce across studies. A thorough evaluation of an audio generation system should report multiple metrics (at minimum FAD, CLAP score, and MOS) and clearly describe the evaluation protocol, including the number of samples, the feature extractor used for FAD, the CLAP model version, and the MOS rater pool characteristics. Relying on a single metric is a common source of misleading conclusions in the literature.

Proposition 49 (Metric disagreement).

There exist pairs of audio generation systems (A,B) such that FAD(A)<FAD(B) (system A has lower FAD, hence better distributional similarity) but MOS(A)<MOS(B) (system B is preferred by human raters).

Proof.

We construct an explicit example. Let system A generate audio clips that are individually mediocre but collectively diverse, covering the real data distribution well. Let system B generate audio clips that are individually excellent but concentrated in a narrow region of the audio space (e.g., a single genre). System A will have lower FAD because its feature distribution closely matches the real distribution, but system B will have higher MOS because each individual sample sounds better. This is the audio analogue of the well-known precision-recall trade-off in image generation: FAD conflates precision and recall, while MOS primarily measures precision (per-sample quality).

Musicality and Perceptual Quality

The evaluation metrics in Evaluation Metrics for Audio Generation assess generic audio quality and text-audio alignment. For music generation specifically, we also need to evaluate attributes that are central to musicality: rhythmic regularity, harmonic coherence, melodic contour, structural organisation, and stylistic diversity. This section introduces domain-specific metrics for these attributes and analyses the fundamental trade-off between diversity and quality.

Musical Structure Metrics

Musical structure operates at multiple temporal scales: individual notes (milliseconds), beats and bars (seconds), phrases (tens of seconds), and sections (minutes). We discuss metrics at each scale.

Beat tracking accuracy.

A musically coherent piece should have a clear and consistent beat. Given a generated audio signal 𝒙, a beat tracker extracts a sequence of beat times 𝒃^=(b^1,,b^B). If a reference beat grid 𝒃=(b1,,bB) is available, the F-measure of beat tracking is (Fbeat)Fbeat=2PbeatRbeatPbeat+Rbeat, where precision Pbeat is the fraction of predicted beats within a tolerance window δ (typically δ=70,ms) of a true beat, and recall Rbeat is the fraction of true beats matched by a prediction.

For generated music without a reference, we instead compute beat consistency: the regularity of inter-beat intervals. Let Δn=b^n+1b^n be the n-th inter-beat interval. The normalised tempo deviation is (Tempo DEV)σtempo=std(Δ1,,ΔB1)mean(Δ1,,ΔB1), where std and mean denote the sample standard deviation and mean, respectively. Lower values indicate more consistent tempo. Professional recordings typically have σtempo<0.05 (excluding intentional rubato), while generated music from early systems often exhibits σtempo>0.15.

Key and chord estimation.

Harmonic coherence can be assessed by estimating the key and chord progression of the generated music and evaluating their consistency. Let k^{C,C,D,,B}×{major,minor} be the estimated global key and 𝒄^chord=(c^1,,c^M) the estimated chord sequence. Two metrics are relevant:

  1. Key consistency. The fraction of estimated chords that are diatonic to (i.e., belong to the scale of) the estimated key: (Keycons)KeyCons=1Mm=1M𝟏[c^mDiatonic(k^)]. Professional music typically has KeyCons0.80 (allowing for secondary dominants, borrowed chords, and modulations).

  2. Chord transition smoothness. The average “harmonic distance” between consecutive chords, measured by the number of common tones or the distance in the Tonnetz (a graph-theoretic representation of pitch relationships): (Chordsmooth)ChordSmooth=1M1m=1M1dTonnetz(c^m,c^m+1). Lower values indicate smoother harmonic progressions.

Self-similarity analysis.

Musical form (verse-chorus structure, ABA form, rondo) creates characteristic patterns of repetition and contrast. These can be quantified using the self-similarity matrix (SSM).

Definition 68 (Self-Similarity Matrix).

Given a sequence of feature vectors 𝒇=(𝒇1,,𝒇N)N×d (e.g., chroma features, MFCC features, or learned embeddings) extracted from an audio signal at a fixed hop size, the self-similarity matrix is 𝐒[0,1]N×N with entries (SSM EVAL)Si,j=𝒇i𝒇j𝒇i2𝒇j2, i.e., the cosine similarity between features at positions i and j.

In the SSM of well-structured music, one observes:

  • Diagonal blocks. Sections of the piece that are internally consistent (e.g., a verse) appear as bright blocks along the main diagonal.

  • Off-diagonal stripes. Repeated sections (e.g., the chorus returning) appear as bright stripes parallel to the main diagonal.

  • Checkerboard patterns. Alternating sections (ABAB form) produce a checkerboard-like pattern.

The structural complexity of a piece can be quantified as the entropy of the SVD spectrum of 𝐒: (Struct Entropy)Hstruct=i=1Nσ~ilogσ~i, where σ~i=σi/jσj is the normalised i-th singular value of 𝐒. A piece with highly repetitive structure (low complexity) will have a few dominant singular values and low entropy, while a piece with no repetition (high complexity) will have a flat spectrum and high entropy.

Remark 50 (Domain-specific nature of structure metrics).

The metrics above are tailored to tonal, rhythmic music in the Western tradition. They may not be appropriate for:

  • Atonal music (e.g., serialist compositions), where key consistency is meaningless.

  • Ambient or drone music, where beat tracking is inapplicable.

  • Non-Western traditions with different tonal systems (e.g., maqam, raga) or rhythmic structures (e.g., tala).

Developing culturally inclusive evaluation metrics for music generation is an important open problem.

Diversity and Coverage

A good music generation system should not only produce high-quality samples but also exhibit diversity: generating different outputs for different prompts (inter-prompt diversity) and varied outputs for the same prompt across different random seeds (intra-prompt diversity).

Intra-class diversity.

For a fixed prompt 𝒄, generate K samples {𝒙1,,𝒙K} and compute pairwise distances in a feature space: (Intra DIV)Divintra(𝒄)=2K(K1)i<jd(ψ(𝒙i),ψ(𝒙j)), where d is a distance function (e.g., cosine distance) and ψ is a feature extractor. Low intra-class diversity indicates mode collapse: the model produces nearly identical outputs regardless of the random seed.

Genre coverage.

For a set of genre labels 𝒢={g1,,gG}, generate N samples from genre-specific prompts and classify them using a pretrained genre classifier. The genre coverage is the fraction of genres for which the classifier assigns the correct label to at least one generated sample: (Genre Coverage)Coverage=1Gg=1G𝟏[i:g^(𝒙i)=g and prompt(𝒙i)=g].

Proposition 50 (Diversity-quality trade-off).

Let pθ(𝒙|𝒄) be a conditional generative model parameterised by θ, and let Q(θ)=𝔼𝒄,𝒙pθ[q(𝒙)] and D(θ)=𝔼𝒄[𝖧(pθ(|𝒄))] denote the expected quality and the expected conditional entropy (a measure of diversity), respectively, where q(𝒙) is a per-sample quality function and 𝖧 is the differential entropy. Then, for the class of tempered models pθ(β)(𝒙|𝒄)pθ(𝒙|𝒄)q(𝒙)β, the diversity D is a non-increasing function of the temperature parameter β0: (DIV Decrease)dDdβ0.

Proof.

The tempered distribution p(β)(𝒙|𝒄)p(𝒙|𝒄)q(𝒙)β concentrates on high-quality regions as β increases. The differential entropy of p(β) is 𝖧(p(β))=p(β)(𝒙)logp(β)(𝒙)d𝒙. Taking the derivative with respect to β: (Entropy Deriv)d𝖧dβ=p(β)β(1+logp(β)(𝒙))d𝒙=𝖢ovp(β)[logq(𝒙),logp(β)(𝒙)]. Under mild regularity conditions, logq(𝒙) and logp(β)(𝒙) are positively correlated (since p(β) assigns more mass to regions where q is large), so the covariance is non-negative, giving d𝖧/dβ0.

Example 33 (Measuring diversity in MusicGen).

In evaluating MusicGen, one can generate K=10 samples for each of N=100 text prompts and compute the average intra-prompt diversity. Using CLAP embeddings as the feature space ψ and cosine distance as d, typical values are Divintra0.35 for MusicGen-Large with standard sampling, and Divintra0.22 for the same model with classifier-free guidance at scale w=3.0. This confirms the diversity-quality trade-off: higher guidance improves CLAP scores but reduces sample diversity.

Exercise 19 (FAD Computation and Bias).

Let 𝝁r=(1,0), 𝝁g=(0,1), and 𝚺r=(2112),𝚺g=(3001).

  1. Compute FAD=𝝁r𝝁g22+trace(𝚺r+𝚺g2(𝚺r𝚺g)1/2). (Hint: diagonalise the product 𝚺r𝚺g first.)

  2. Show that 𝚺r and 𝚺g do not commute, so the matrix square root (𝚺r𝚺g)1/2 cannot be computed by simply taking the square root of each eigenvalue of 𝚺r𝚺g. Explain why the trace identity trace((𝐀𝐁)1/2)=trace((𝐀1/2𝐁𝐀1/2)1/2) still holds.

  3. If you have only Nr=Ng=50 samples from each distribution (with d=2), estimate the expected bias of the FAD estimate using (FAD BIAS).

Exercise 20 (Piano Roll Diffusion).

Consider a simplified piano roll with T=16 time steps and P=12 pitches (one octave). The activation rate is ρ=1/12 (exactly one pitch active per time step, representing a monophonic melody).

  1. Compute the sparsity-aware weights w+ and w0 from Proposition 47.

  2. Write down the forward diffusion process for the relaxed piano roll 𝐏[0,1]16×12 at noise level t with αt=0.5. What is the expected value and variance of each entry?

  3. Suppose the denoiser predicts a continuous output 𝐏^016×12. Describe two strategies for converting this to a valid monophonic piano roll (exactly one active pitch per time step). Which strategy preserves the monophonic constraint better?

  4. Argue that discrete diffusion (absorbing state) is a more natural choice than continuous diffusion for this problem. What is the forward transition matrix for the absorbing-state process on a vocabulary of {0,1,,12} (12 pitches plus one absorbing “mask” token)?

Exercise 21 (CLAP Score Properties).

Let ϕt(𝒄)d and ϕa(𝒙)d be CLAP embeddings with ϕt(𝒄)2=ϕa(𝒙)2=1 (unit normalised).

  1. Show that the CLAP score satisfies CLAP(𝒄,𝒙)=112ϕt(𝒄)ϕa(𝒙)22.

  2. A generation system produces audio 𝒙 that maximises the CLAP score: 𝒙=arg max𝒙CLAP(𝒄,𝒙). Assuming the audio encoder ϕa is surjective onto the unit sphere 𝕊d1, show that ϕa(𝒙)=ϕt(𝒄).

  3. Explain why maximising the CLAP score alone is insufficient for high-quality music generation. Give a concrete example of an audio signal with CLAP(𝒄,𝒙)1 but poor perceptual quality.

  4. Propose a combined objective that balances CLAP score with a quality metric. Write it as an optimisation problem and discuss the trade-offs.

Exercise 22 (Video-to-Music Alignment).

Consider a video with F=120 frames (5 seconds at 24 fps) and an audio target of La=250 tokens (5 seconds at 50 Hz).

  1. Write down the dimensions of the soft alignment matrix 𝐀 from (V2M Align). How many parameters does this matrix contain?

  2. Suppose the video contains a single scene cut at frame f=60 (the midpoint). Design an alignment matrix 𝐀 that assigns each audio token to the nearest video frame. Is this matrix a valid soft alignment (i.e., do its rows sum to 1)?

  3. The scene cut should correspond to a musical change (e.g., a chord change or a dynamic accent) at audio token n=125. Write down a loss term that encourages this alignment.

  4. In the cross-attention formulation of (Align OPT), the alignment is determined by the learned projections 𝐖q and 𝐖k. Explain why this is more flexible than the fixed alignment in part (b) and discuss potential failure modes.

Detection and Watermarking

The generative audio systems developed in the preceding sections can synthesise speech, music, and sound effects of remarkable quality. This very capability raises an urgent question: how can we distinguish AI-generated audio from recordings of real-world events? The stakes are considerable. Synthetic speech can impersonate individuals for fraud; generated music can circumvent copyright protections; fabricated sound effects can manipulate evidence. In this section, we formalise the detection problem, study two families of solutions (passive detection and active watermarking), and analyse their information-theoretic limits.

The Detection Problem

We begin by casting detection as a binary hypothesis test.

Definition 69 (Audio Deepfake Detection).

Let 𝒳=N denote the space of discrete-time audio signals of length N. An audio deepfake detector is a measurable function h:𝒳{0,1}, where h(𝒙)=0 indicates “real” (naturally recorded) and h(𝒙)=1 indicates “synthetic” (AI-generated). The detector is parameterised by a threshold τ applied to a continuous scoring function sϕ:𝒳: (Detector)h(𝒙)=𝟏[sϕ(𝒙)τ]. The performance of h is characterised by the probability of false alarm PFA=Pr(h(𝒙)=1|𝒙 is real) and the probability of detection PD=Pr(h(𝒙)=1|𝒙 is synthetic).

Remark 51 (Threat models).

The difficulty of detection depends fundamentally on what the detector knows about the generator.

  1. White-box setting. The detector has access to the generator's architecture and weights. This enables powerful attacks: one can compute the likelihood pθ(𝒙) under the generator and use it as a detection score.

  2. Black-box setting. The detector can query the generator with arbitrary inputs but cannot inspect its internals. Detection must rely on statistical properties of the output distribution.

  3. Closed-box setting. The detector has no access to the generator at all; it must detect synthetic audio using only properties learned from a training set of real and (possibly different) synthetic examples. This is the hardest and most practically relevant setting.

The Neyman–Pearson lemma provides the optimal detector when the two distributions are known.

Proposition 51 (Optimal Detection via Likelihood Ratio).

Let preal and psyn denote the distributions of real and synthetic audio, respectively. Among all detectors satisfying PFAα for a given α(0,1), the one maximising PD is the likelihood ratio test (LR TEST)h(𝒙)=𝟏[psyn(𝒙)preal(𝒙)η], where η>0 is chosen so that Pr(h(𝒙)=1|𝒙preal)=α.

Proof.

This is a direct application of the Neyman–Pearson lemma. Define the log-likelihood ratio (𝒙)=logpsyn(𝒙)logpreal(𝒙). The region {𝒙:(𝒙)logη} is the most powerful critical region at level α for testing H0:𝒙preal against H1:𝒙psyn. No other test at level α can achieve a higher detection probability, as proved in standard references on hypothesis testing (see, e.g., ch:probability).

Caution.

In the closed-box setting, neither preal nor psyn is available. The detector must learn a scoring function sϕ from data, and there is no guarantee that the learned score will approximate the likelihood ratio, especially when the synthetic audio comes from a generator not seen during training. This generalisation gap is the central challenge of practical deepfake detection.

SynthID for Audio

Active watermarking takes a fundamentally different approach from passive detection: rather than trying to identify synthetic audio after the fact, the generator embeds an imperceptible signal in every output, which a dedicated decoder can later extract. Google DeepMind's SynthID for audio operates in the spectrogram domain, embedding a binary payload into the time-frequency representation of the generated waveform.

Definition 70 (Spectrogram Watermarking).

Let 𝐒F×K denote the magnitude spectrogram of an audio signal (with F time frames and K frequency bins), and let 𝒃{0,1}L be a binary payload of L bits. A spectrogram watermarking system consists of an encoder θ:F×K×{0,1}LF×K and a decoder 𝒟ψ:F×K{0,1}L satisfying:

  1. Imperceptibility: dperceptual(𝐒,θ(𝐒,𝒃))δ for a perceptual distance dperceptual and tolerance δ>0.

  2. Payload recovery: 𝒟ψ(θ(𝐒,𝒃))=𝒃 with high probability.

  3. Robustness: 𝒟ψ(𝒜(θ(𝐒,𝒃)))=𝒃 for all attacks 𝒜 in a specified attack set (e.g., MP3 compression, resampling, additive noise, time stretching).

The encoded spectrogram is 𝐒~=θ(𝐒,𝒃)=𝐒+Δ𝐒θ(𝐒,𝒃), where Δ𝐒θ is a perturbation network that outputs the watermark signal.

The training objective balances imperceptibility and recoverability: (Synthid LOSS)SynthID=𝔼𝐒,𝒃[Δ𝐒θ(𝐒,𝒃)2]imperceptibility+λ𝔼𝐒,𝒃,𝒜[=1LBCE([𝒟ψ(𝒜(𝐒~))],b)]recovery under attack, where BCE denotes the binary cross-entropy loss and λ>0 controls the trade-off.

Proposition 52 (Watermark Capacity Bound).

Consider a spectrogram watermarking channel where the host signal 𝐒 has power PS, the watermark perturbation has power PW (constrained by PWδ2 for imperceptibility), and the attack channel introduces additive noise of power PA. Under Gaussian assumptions, the maximum achievable payload rate is bounded by (Capacity Bound)LFK2log2(1+PWPA)bits per spectrogram frame, where FK is the total number of time-frequency bins. For a fixed perceptual distortion budget δ, this establishes a fundamental trade-off between payload length L and robustness to attacks of power PA.

Proof.

Model the watermarking problem as a communication channel. The encoder transmits L bits through the spectrogram by adding a perturbation Δ𝐒 with power PW to the host 𝐒. The attack adds noise 𝒏 with power PA. The decoder observes 𝐒+Δ𝐒+𝒏. Treating 𝐒 as a known “side information” at both encoder and decoder (the informed embedding model, sometimes called “writing on dirty paper”), the effective channel reduces to a Gaussian channel with input power PW and noise power PA. By Shannon's channel coding theorem, the capacity per time-frequency bin is 12log2(1+PW/PA), and summing over FK bins yields the bound.

Remark 52.

In practice, SynthID achieves payloads of L32 to 128 bits at sample rates of 16–48 kHz, which is well below the information-theoretic capacity. The gap arises because the Gaussian assumption is only approximate, the perceptual constraint is more restrictive than a simple power constraint, and the attack set is broader than additive Gaussian noise.

AudioSeal

While SynthID operates globally on the spectrogram, Meta's AudioSeal introduces localized watermarking that operates at the sample level, enabling temporal localisation of which portions of an audio signal are synthetic.

Definition 71 (Localized Audio Watermarking).

A localized audio watermarking system consists of a generator Gθ:N×{0,1}LN and a detector Dψ:N[0,1]N such that:

  1. The generator produces a watermarked signal 𝒙~=Gθ(𝒙,𝒃)=𝒙+𝒘θ(𝒙,𝒃), where 𝒘θ is an imperceptible watermark signal.

  2. The detector outputs a per-sample detection probability: for each sample index n{1,,N}, [Dψ(𝒙~)]n[0,1] indicates the probability that sample n belongs to a watermarked segment.

  3. An optional message decoder Mψ:N{0,1}L recovers the embedded payload.

The per-sample detection capability enables temporal localisation: given an audio signal that is a mixture of real and synthetic segments, the detector can identify the synthetic portions.

The AudioSeal architecture uses a convolutional encoder-decoder structure with a multi-scale discriminator for perceptual quality.

AudioSeal architecture. The generator Gθ takes an input waveform 𝒙 and payload 𝒃 and produces an imperceptible watermark signal 𝒘θ. After potential attacks 𝒜 (compression, noise, etc.), the detector Dψ outputs per-sample detection probabilities, enabling temporal localisation. An optional message decoder Mψ recovers the embedded payload.

The AudioSeal training loss combines several terms: (Audioseal LOSS)AudioSeal=𝒘θ(𝒙,𝒃)2imperc.+λ1adv(Gθ,Ddisc)quality+λ2𝔼[n=1Nynlog[Dψ(𝒙~)]n+(1yn)log(1[Dψ(𝒙~)]n)]detection+λ3𝔼[=1LBCE([Mψ(𝒙~)],b)]recovery, where yn{0,1} indicates whether sample n is watermarked, Ddisc is a multi-scale waveform discriminator (similar to the HiFi-GAN discriminator), and λ1,λ2,λ3>0 are balancing hyperparameters.

Lemma 4 (AudioSeal Detection Error Rate).

Suppose the detector Dψ achieves per-sample binary cross-entropy loss n on watermarked and unwatermarked segments. For a segment of M consecutive samples, the segment-level detection score is the average D=1Mn=1M[Dψ(𝒙~)]n. If the per-sample scores are i.i.d. with mean μ1 under watermarked audio and μ0 under unwatermarked audio (with μ1>μ0) and common variance σ2, then the segment-level false positive rate at threshold τ decays as (Audioseal FPR)PFA(M)=Pr(Dτ|unwatermarked)exp(M(τμ0)22σ2), by Hoeffding's inequality. Thus, aggregating over longer segments exponentially improves detection reliability.

Proof.

Under the i.i.d. assumption, D is a sample mean of M bounded random variables in [0,1]. Hoeffding's inequality states that Pr(Dμ0t)exp(2Mt2/(ba)2) for variables in [a,b]. Setting t=τμ0 and (ba)2=1 yields the stated bound. For sub-Gaussian variables with variance proxy σ2, we obtain the sharper form with 2σ2 in the denominator. As M increases, the bound decays exponentially, confirming that longer segments permit more reliable detection.

Key Idea.

Proactive versus reactive detection. Watermarking (AudioSeal, SynthID) is a proactive approach: the generator cooperates by embedding a signal at generation time. Passive detection is a reactive approach: the detector must identify synthetic audio without any cooperation from the generator. Proactive methods are more reliable when applicable (the generator is under the control of a responsible party), but reactive methods are necessary when the generator is adversarial or unknown. A robust detection ecosystem requires both.

Statistical Detection Methods

When watermarking is not available, passive detection must rely on statistical artifacts that distinguish synthetic audio from real recordings. These artifacts arise from the generation process itself: the architecture, the training data, and the sampling procedure all leave traces in the output distribution.

Proposition 53 (Spectral Signature of Diffusion Artifacts).

Let 𝒙N be an audio signal generated by a diffusion model with T denoising steps and step size Δt=1/T. In the high-frequency regime (angular frequency ω>ωc for a model-dependent cutoff ωc), the power spectral density of the generated signal exhibits a characteristic roll-off: (Spectral Rolloff)Φsyn(ω)Φreal(ω)(1CT2ω2)for ω>ωc, where C>0 is a constant depending on the noise schedule and the Lipschitz constant of the denoiser. This high-frequency attenuation is a consequence of the finite number of denoising steps: each step can only partially restore high-frequency content, and the cumulative effect is a slight suppression relative to the natural audio distribution.

Proof sketch.

Consider the probability flow ODE discretised with step size Δt=1/T. At each step, the denoiser 𝝐θ(𝒙t,t) approximates the score function 𝒙logpt(𝒙). The discretisation error at each step is O(Δt2) in the L2 norm. In the frequency domain, this error concentrates at high frequencies because the score function is smoother (lower-frequency) than the data. Summing the per-step frequency-domain errors over T steps and applying the Parseval relation yields a cumulative high-frequency deficit of O(TΔt2)=O(1/T) in the power spectral density, which gives the C/(T2ω2) factor after accounting for the ω-dependent sensitivity.

Remark 53 (Practical detection features).

Beyond spectral roll-off, practical detectors exploit a variety of statistical features:

  1. Phase regularity. Vocoders (HiFi-GAN, SoundStream) reconstruct phase from magnitude, producing phase spectra that are smoother than natural phase.

  2. Codec artifacts. Neural codec models (Encodec, DAC) introduce quantisation artifacts at codebook boundaries that differ from those of traditional codecs (MP3, AAC).

  3. Temporal micro-structure. Autoregressive models generate audio frame-by-frame, producing subtle periodic patterns at the frame boundary rate that are absent in natural recordings.

The adversarial robustness of detection methods is a critical concern. An adversary who knows the detection method can attempt to evade it.

Proposition 54 (Evasion Attack Bound).

Let sϕ:𝒳 be a detection scoring function that is Ls-Lipschitz with respect to the 2 norm on waveforms. For any synthetic audio 𝒙syn with detection score sϕ(𝒙syn)=s0>τ, there exists an adversarial perturbation 𝜹N with 𝜹(s0τ)/Ls such that the perturbed signal 𝒙=𝒙syn+𝜹 satisfies sϕ(𝒙)τ and thus evades detection.

Proof.

By the Lipschitz condition, |sϕ(𝒙)sϕ(𝒙syn)|Ls𝜹. Setting 𝜹=s0τLs𝒙sϕ(𝒙syn)𝒙sϕ(𝒙syn) (the steepest-descent direction) yields sϕ(𝒙)s0Lss0τLs=τ, achieving exact evasion with the minimum perturbation.

Theorem 4 (Detection–Quality Trade-off).

Let preal and pθ denote the distributions of real and generated audio, respectively. For any detector h, the sum of false negative and false positive rates satisfies (Detection TV)PFA+(1PD)1TV(pθ,preal), where TV(,) denotes the total variation distance. Consequently, as the generator improves so that TV(pθ,preal)0, no detector can simultaneously achieve low false alarm and high detection rates: PDPFA2TV(pθ,preal)0.

Proof.

By the variational representation of total variation, TV(pθ,preal)=supA|pθ(A)preal(A)| over all measurable sets A. Taking A={𝒙:h(𝒙)=1}, we obtain PDPFATV(pθ,preal). The bound on the error sum follows by noting that PFA+(1PD)=1(PDPFA)1TV(pθ,preal).

The Arms Race

Insight.

The detection arms race. Detection and generation exist in a perpetual cat-and-mouse dynamic. As generators improve (producing audio with fewer detectable artifacts), detectors must find increasingly subtle statistical signatures. As detectors improve, generators are trained adversarially to remove those signatures. This is not merely an engineering challenge; it has a precise information-theoretic formulation. Theorem 4 shows that at the limit where TV(pθ,preal)0, the optimal detector's advantage over random guessing vanishes. In this regime, only active watermarking (which modifies the generation process itself) can provide reliable detection.

Post-processing attacks represent a significant practical threat to both passive detection and watermarking.

Remark 54 (Post-processing attacks).

Common attacks against audio detectors and watermarks include:

  1. Lossy compression (MP3, AAC, Opus at low bitrates) removes high-frequency information where many artifacts reside.

  2. Resampling (e.g., 481648 kHz) applies anti-aliasing filters that destroy fine spectral structure.

  3. Time stretching and pitch shifting modify the temporal and frequency structure without significantly affecting perceived content.

  4. Additive noise masks low-energy watermark signals.

  5. Adversarial perturbations (as in Proposition 54) apply targeted perturbations to fool specific detectors.

Robust watermarking systems must be trained with these attacks in the augmentation pipeline. Even so, no system is provably robust against all possible attacks; this is a consequence of the information-theoretic capacity bound in Proposition 52.

Remark 55 (Generalisation to unseen generators).

A critical limitation of current passive detectors is their poor generalisation to generators not seen during training. A detector trained on WaveNet-generated speech may fail entirely on diffusion-generated speech. This is because the detector learns generator-specific artifacts rather than a universal notion of “synthetic-ness.” Cross-generator generalisation remains an active research challenge, with promising directions including self-supervised pretraining on audio representations and meta-learning across diverse generator families.

Historical Note.

From audio forensics to neural detection. The problem of detecting manipulated audio predates neural generation by decades. Early audio forensic techniques (dating to the 1970s) focused on detecting splices in analogue tape recordings by analysing the electrical network frequency (ENF) embedded in recordings made near power lines. Digital audio forensics in the 2000s developed techniques for detecting copy-move forgeries, double compression artifacts, and steganographic content. The arrival of neural text-to-speech (WaveNet, 2016) and neural voice conversion created a fundamentally new challenge: the synthetic audio is not a manipulation of an existing recording but a de novo generation, and the artifacts are statistical rather than physical. The ASVspoof challenge series (beginning in 2015) has been the primary benchmark driving progress in synthetic speech detection, evolving from detecting simple concatenative synthesis to detecting state-of-the-art neural TTS and voice conversion.

The technical capabilities developed in this chapter intersect with a web of ethical, legal, and societal concerns. In this section, we examine the principal areas of tension: copyright and intellectual property, voice cloning and deepfakes, and the emerging frameworks for attribution and provenance.

Generative music models are trained on large corpora of copyrighted recordings. This raises two distinct legal questions: (i) whether training on copyrighted material constitutes fair use, and (ii) whether the generated output infringes the copyright of training examples.

From a mathematical perspective, the question of whether a generated piece is “substantially similar” to a training example can be formalised as a distance computation.

Definition 72 (Musical Similarity Metrics).

Let 𝒙,𝒚𝒳 be two audio signals. Common similarity metrics include:

  1. Chroma cosine similarity: Extract chroma features (12-dimensional pitch class profiles) 𝒄𝒙(t),𝒄𝒚(t)12 at each time step, then compute (Chroma SIM)dchroma(𝒙,𝒚)=11Tt=1T𝒄𝒙(t)𝖳𝒄𝒚(t)𝒄𝒙(t)𝒄𝒚(t).

  2. Melodic contour distance: Extract the fundamental frequency trajectory f0𝒙(t) and f0𝒚(t), and compute the dynamic time warping (DTW) distance (DTW Melody)dmelody(𝒙,𝒚)=minπΠ(i,j)π|logf0𝒙(i)logf0𝒚(j)|2, where Π is the set of valid DTW alignment paths.

  3. Embedding distance: Compute representations via a pretrained audio model ϕ and measure the Euclidean distance demb(𝒙,𝒚)=ϕ(𝒙)ϕ(𝒚).

Remark 56 (Style versus substance).

The legal distinction between style mimicry (generating music “in the style of” an artist, which is generally not copyright infringement) and plagiarism (reproducing protected melodic or harmonic content) does not map cleanly onto any single distance metric. Style is a distributional property (captured by statistics over many works), while plagiarism concerns specific passages (captured by pairwise distances). The mathematical challenge of defining and detecting the boundary between these two regimes remains open and is likely to require both information-theoretic and legal reasoning.

Remark 57 (Legal landscape).

As of this writing, the legal status of training generative models on copyrighted audio varies by jurisdiction. In the United States, several lawsuits are pending that test whether model training constitutes fair use under Section 107 of the Copyright Act. The European Union's AI Act introduces specific requirements for transparency and disclosure of training data. Japan's copyright law contains an explicit exception for machine learning. These divergent approaches create uncertainty for both developers and creators, underscoring the need for technical solutions (watermarking, attribution) that can operate across legal regimes.

Voice Cloning and Deepfakes

Zero-shot voice cloning systems (as described in earlier sections of this chapter) can synthesise speech in any person's voice from a few seconds of reference audio. While this enables legitimate applications (accessibility, dubbing, voice preservation for medical patients), it also creates serious risks.

  1. Identity fraud. Cloned voices can be used to bypass voice authentication systems, impersonate individuals in phone calls, or create fraudulent audio evidence.

  2. Non-consensual content. An individual's voice can be used without consent to generate speech they never uttered, creating reputational and psychological harm.

  3. Misinformation. Fabricated speech attributed to public figures can spread misinformation, undermine trust in audio evidence, and destabilise democratic processes.

Technical countermeasures include:

  1. Speaker verification. Deploying robust speaker verification systems that can detect cloned voices by comparing against enrolled voiceprints.

  2. Mandatory watermarking. Requiring all voice synthesis systems to embed watermarks (as in sec:audiogen:detection:synthid,sec:audiogen:detection:audioseal) in their output.

  3. Consent mechanisms. Requiring explicit consent from the voice owner before cloning is permitted, enforced through API access controls and voice registration databases.

Remark 58 (The dual-use dilemma).

Voice cloning technology is inherently dual-use: the same system that enables a person with ALS to preserve their voice for future communication also enables a scammer to impersonate a CEO. No purely technical solution can resolve this tension. Effective governance requires a combination of technical safeguards (watermarking, verification), legal frameworks (consent laws, liability for misuse), and platform policies (usage monitoring, abuse reporting). The mathematical frameworks for detection (Detection and Watermarking) provide the technical foundation, but their deployment is ultimately a societal choice.

Attribution and Provenance

The Coalition for Content Provenance and Authenticity (C2PA) standard provides a framework for attaching cryptographically signed metadata to media files, creating an unbroken chain of provenance from creation to consumption.

For audio, the C2PA manifest includes:

  1. A claim declaring that the content was generated by a specific AI system.

  2. A signature from the generating system's certificate authority, ensuring the claim is authentic.

  3. Action records documenting any transformations (editing, mixing, compression) applied after generation.

Watermarking provides a complementary layer of provenance that survives the removal of metadata (which can be stripped by simple file format conversion). The combination of cryptographic provenance (C2PA) and signal-level provenance (watermarking, as discussed in Detection and Watermarking) provides a defence-in-depth strategy for audio attribution.

Key Idea.

Provenance as a technical and social contract. Neither watermarking alone nor metadata alone is sufficient for reliable audio provenance. Watermarks can be attacked (see Remark 54); metadata can be stripped. A robust provenance system requires both technical mechanisms and social contracts: platforms that verify and display provenance information, legal frameworks that penalise provenance removal, and cultural norms that demand transparency about the origin of audio content.

Connections and Synthesis

Audio generation draws on and contributes to nearly every major topic in this book. In this section, we make these connections explicit, providing insight boxes that link the techniques of this chapter to their theoretical foundations elsewhere. The reader is encouraged to revisit the referenced chapters with the audio perspective in mind; the connections often illuminate both sides.

Variational Inference

Insight.

Audio autoencoders are variational autoencoders (ch:vi). The audio autoencoders at the heart of latent diffusion systems for audio (Stable Audio, AudioLDM) are trained with an objective that is a direct descendant of the evidence lower bound (ELBO) developed in ch:vi. Given an audio waveform 𝒙, the encoder produces a distribution qϕ(𝒛|𝒙) over latent codes, and the decoder reconstructs pψ(𝒙|𝒛). The training objective is (ELBO)ELBO=𝔼qϕ(𝒛|𝒙)[logpψ(𝒙|𝒛)]+𝖣KL(qϕ(𝒛|𝒙)p(𝒛)), augmented with perceptual and adversarial losses specific to audio. The KL term regularises the latent space to be smooth and approximately Gaussian, which is essential for the subsequent diffusion process to work well. The theoretical analysis of ELBO tightness, posterior collapse, and the information bottleneck from ch:vi directly governs the quality of the learned audio latent space: too-tight a bottleneck loses audio detail; too-loose a bottleneck makes diffusion harder.

Optimal Transport and Flow Matching

Insight.

FAD as W22 and flow matching as OT paths (ch:sinkhorn, ch:flows). The Fréchet Audio Distance (FAD), the standard quality metric for audio generation, is exactly the squared 2-Wasserstein distance between Gaussian-fitted feature distributions. Let 𝝁r,𝚺r and 𝝁g,𝚺g be the means and covariances of real and generated audio features extracted by a pretrained classifier (typically VGGish). Then (FAD W2)FAD=W22(Normal(𝝁r,𝚺r),Normal(𝝁g,𝚺g))=𝝁r𝝁g2+trace(𝚺r+𝚺g2(𝚺r𝚺g)1/2), connecting audio evaluation directly to the theory of optimal transport developed in ch:sinkhorn. Meanwhile, the flow matching training paradigm used in modern audio diffusion models learns a velocity field 𝒗t(𝒛) whose trajectories approximate optimal transport maps from noise to data. The conditional OT path 𝒛t=(1t)𝒛0+t𝒛1 traces a straight line in latent space, corresponding to the Benamou–Brenier dynamical formulation of the Wasserstein distance. This connection explains why flow matching produces straighter, more efficient sampling trajectories than classical DDPM noise schedules.

Generative Adversarial Networks

Insight.

HiFi-GAN vocoders and adversarial training for audio (ch:gan, ch:wgan). Generative adversarial networks, studied in depth in ch:gan and ch:wgan, play a crucial role in audio generation even within diffusion-based systems. The HiFi-GAN vocoder (and its descendants in SoundStream and Encodec) is a pure GAN: a generator network maps mel spectrograms to waveforms, trained against multi-period and multi-scale discriminators. The discriminator losses provide a learned perceptual metric that captures fine temporal structure (pitch periodicity, transient sharpness) that is poorly measured by 2 reconstruction loss. The theoretical analysis of GAN training dynamics from ch:gan (mode collapse, discriminator saturation, the role of gradient penalties from ch:wgan) directly applies to understanding the stability and quality of vocoder training. Historically, WaveGAN and GANSynth were among the first neural audio generators, preceding the diffusion era; the transition mirrors the image generation timeline, with diffusion models offering improved diversity and training stability at the cost of slower inference.

Vector Quantisation

Insight.

RVQ codecs as descendants of VQ-VAE (ch:vae2). The residual vector quantisation (RVQ) at the core of neural audio codecs (Encodec, SoundStream, DAC) is a direct extension of the VQ-VAE framework introduced in ch:vae2. The standard VQ-VAE uses a single codebook of K vectors; RVQ applies Q successive codebooks, each quantising the residual error from the previous stage. Formally, the RVQ encoding of a latent vector 𝒛 is the sequence of codebook indices (c1,c2,,cQ) with (RVQ)𝒛^=q=1Q𝒆cq(q),where cq=arg mink𝒓q1𝒆k(q)2,𝒓q=𝒓q1𝒆cq(q),𝒓0=𝒛. The codebook learning challenge (codebook collapse, the straight-through estimator, exponential moving average updates) analysed in ch:vae2 applies directly. The key innovation for audio is the residual structure: by stacking Q=8 to 32 codebooks, RVQ achieves the fine-grained quantisation needed for perceptually transparent audio compression at 1.5–24 kbps, far beyond what a single VQ codebook could provide.

Autoregressive Models

Insight.

MusicGen as AR over codec tokens, and WaveNet as pioneering AR audio (ch:autoreg). The autoregressive modelling framework of ch:autoreg is the foundation of two major audio generation paradigms. First, WaveNet (2016) demonstrated that modelling the raw waveform as an autoregressive sequence p(𝒙)=n=1Np(x[n]|x[1],,x[n1]) could produce remarkably natural speech, though at enormous computational cost (16,000 sequential steps per second of audio at 16 kHz). Second, modern systems like MusicGen and VALL-E apply autoregressive transformers not to raw samples but to the discrete tokens produced by neural audio codecs. With a typical codec operating at 50 tokens per second per codebook and Q=4 codebooks, the sequence length for 10 seconds of music is 50×10×4=2,000 tokens, well within the capacity of modern transformers. The codebook pattern strategies (delay, parallel, interleaved) studied earlier in this chapter are specific solutions to the challenge of modelling multi-codebook sequences autoregressively, a problem that does not arise in text-only autoregressive models but connects to the multi-head prediction strategies discussed in ch:autoreg.

Video Diffusion

Insight.

Parallel temporal challenges in audio and video (ch:video-diffusion). Audio and video generation share a fundamental structural challenge: both must produce coherent temporal sequences where local quality (per-frame sharpness, per-sample fidelity) and global structure (narrative coherence, musical form) must be simultaneously maintained. The factored attention strategies used in video diffusion (separate spatial and temporal attention layers) have direct analogues in audio diffusion, where separate frequency and temporal attention axes are used. The temporal caching strategies for efficient video sampling (ch:video-diffusion) are applicable to audio latent diffusion, exploiting the fact that the denoiser output changes slowly across adjacent latent frames. Perhaps most significantly, the frontier of audio-visual generation (producing video with synchronised sound) requires jointly modelling both modalities, connecting the two chapters at a systems level. Cross-modal attention between audio and video latents provides the synchronisation signal, as discussed in ch:video-diffusion.

Diffusion Language Models

Insight.

Discrete diffusion for symbolic music (sec:diffusion_language). The discrete diffusion framework developed in sec:diffusion_language for text generation has a natural application to symbolic music (MIDI, ABC notation), where the data consists of discrete tokens representing notes, durations, and dynamics. Rather than diffusing in a continuous waveform or spectrogram space, discrete diffusion corrupts a sequence of musical tokens by replacing them with random tokens according to a masking schedule, and the denoiser learns to restore the original sequence. This connects directly to the absorbing-state and uniform-noise diffusion processes studied in sec:diffusion_language. The advantage over standard autoregressive music generation is that discrete diffusion can generate all tokens in parallel (or with a small number of iterative refinement steps), enabling bidirectional context and non-left-to-right generation strategies that are natural for music (where, for example, a composer might write the chorus before the verses).

Connections Map

fig:audiogen:connection-map visualises the connections between audio generation and the other topics covered in this book.

Connection map linking audio and music generation to other chapters of this book. Each edge indicates a specific technical relationship discussed in the corresponding insight box above.

Remark 59 (The breadth of connections).

The connections enumerated above are not merely analogies; they represent shared mathematical structures. The ELBO, optimal transport distances, adversarial training, vector quantisation, autoregressive factorisation, and discrete diffusion are all instantiations of general frameworks that manifest differently in the audio domain due to the specific properties of audio signals (high dimensionality, strong temporal structure, perceptual sensitivity to phase and periodicity). Understanding these connections enables practitioners to transfer insights across modalities: an advance in image VAE training may improve audio autoencoders, a faster ODE solver for video diffusion may accelerate audio latent diffusion, and a better quantisation scheme for language models may yield better audio codecs.

Open Problems and Future Directions

Despite the remarkable progress documented in this chapter, audio and music generation systems remain far from the ideal creative tool. In this section, we identify five fundamental open problems that define the frontier of the field. For each problem, we provide a formal statement, discuss why it is hard, and outline promising directions of attack.

Real-Time Interactive Generation

Current state-of-the-art audio generation systems require several seconds to generate one second of audio on modern GPUs. Interactive applications (live performance, game audio, conversational agents) demand generation latency below the threshold of human perception.

Problem 1 (Real-Time Audio Generation).

Design a generative audio model that produces audio of quality comparable to current state-of-the-art systems while satisfying the following latency constraints:

  1. Conversational speech: end-to-end latency (from text input to first audio sample) below 200 ms.

  2. Interactive music: latency from control signal change (e.g., a key press, a tempo change) to audible response below 50 ms.

  3. Sound effects: latency from trigger event to onset below 20 ms (the threshold for perceived synchronisation in audiovisual media).

The total computational budget must fit within a single consumer GPU or (ideally) a mobile device's neural processing unit.

The latency budget can be decomposed into three stages, each of which must be optimised.

  1. Encoding stage: Computing the conditioning representation (text embedding, control signal processing). This is typically fast (<10 ms) and is not the bottleneck.

  2. Generation stage: Running the diffusion or autoregressive model to produce the latent representation. For a diffusion model with T denoising steps, each requiring a full forward pass through a large transformer, this is the dominant cost. Consistency distillation (reducing T to 1–4 steps) and model distillation (reducing the model size) are the principal approaches.

  3. Decoding stage: Converting the latent representation to a waveform via the decoder/vocoder. For a streaming codec decoder, this can operate in real-time with modest computation, but the latency depends on the codec's frame size.

Remark 60 (Streaming generation).

Real-time generation requires a streaming architecture that can begin outputting audio before the entire generation is complete. For autoregressive models (MusicGen, VALL-E), streaming is natural: each token can be decoded and output as soon as it is generated. For diffusion models, streaming is more challenging because the standard denoising process operates on the entire latent simultaneously. Promising approaches include chunked diffusion (generating overlapping segments and crossfading) and causal diffusion architectures that denoise from left to right.

Song-Length Coherence

Current music generation systems produce clips of at most 30 seconds to 2 minutes with acceptable quality. Generating a complete song (3 to 5 minutes) with consistent structure (verse, chorus, bridge, key changes, dynamic arc) remains an open problem.

Problem 2 (Song-Length Musical Coherence).

Generate a piece of music of duration D180 seconds (3 minutes) that satisfies:

  1. Local quality: Every 5-second segment, evaluated in isolation, is indistinguishable from a human-composed recording.

  2. Structural coherence: The piece exhibits recognisable musical form (e.g., AABA, verse-chorus) with thematic recurrence: melodic motifs introduced in the first 30 seconds reappear (possibly varied) later in the piece.

  3. Harmonic consistency: The piece maintains a coherent key centre (with possible modulations) and avoids unintended harmonic discontinuities.

  4. Dynamic arc: The piece has a perceptible narrative in terms of energy, texture, and instrumentation that builds, peaks, and resolves.

The difficulty lies in the mismatch between the model's context window and the time scales of musical structure. A typical diffusion model's context window covers 10–30 seconds, but musical structure operates at multiple hierarchical levels:

LevelTime scaleExamples
Sample/grain<50 msTimbre, onset transient
Beat/note100–500 msRhythm, melody
Phrase2–8 sMelodic phrase, chord progression
Section15–60 sVerse, chorus, bridge
Song3–5 minOverall form, dynamic arc
tableHierarchical time scales of musical structure.

Remark 61 (Analogy to long video).

The song-coherence problem is the audio analogue of the long video generation problem discussed in ch:video-diffusion. Both require maintaining global structure over time spans that far exceed the model's native context window. Autoregressive chunk generation, hierarchical planning, and global conditioning signals (e.g., a structural outline) are promising shared strategies.

Faithful Instrument and Vocal Rendering

Even the best current systems produce audio that experts can distinguish from real recordings. The gap is most apparent for solo instruments and vocals, where listeners have extremely sensitive expectations.

Conjecture 3 (Codec Bottleneck Hypothesis).

As neural audio codec quality approaches perceptual transparency (typically achieved at bitrates of approximately 128 kbps or above with current architectures), the quality bottleneck in generated audio shifts from the codec to the generative model. Formally, let codec denote the codec reconstruction error and gen denote the generation error (measured as the gap between the generated and real distributions in the latent space). We conjecture that there exists a codec quality threshold τc such that (Codec Bottleneck)codec<τcperceptual qualityf(gen), where f is a monotonically increasing function independent of the codec. Beyond this threshold, improving the codec yields diminishing returns; the dominant source of artifacts is the generative model's inability to capture the full complexity of the audio distribution.

This conjecture, if true, implies that future progress in audio generation quality requires advances in the generative model itself (better architectures, training procedures, and data) rather than better audio compression.

Remark 62 (Perceptual evaluation frontier).

The gap between generated and real audio is most reliably measured by trained listeners (MOS scores), but even human evaluation has limitations: judgements depend on the listening environment, familiarity with the instrument, and the evaluator's musical training. For solo piano, expert listeners can detect subtle timing irregularities of less than 5 ms and pitch deviations of less than 5 cents. Achieving imperceptible generation for such demanding material requires the generative model to capture the full joint distribution over pitch, timing, dynamics, and timbre at resolutions that approach the limits of human perception.

Unified Audio Generation

Current systems are specialised: separate models for speech synthesis, music generation, and sound effect production. A unified model that handles all audio modalities with a single architecture and a single set of weights would be both more efficient and more capable, enabling cross-modal generation (e.g., generating music with lyrics, or sound effects synchronised to speech).

Problem 3 (Unified Audio Generation).

Design a single generative model pθ(𝒙|𝒄) over the space of all audio signals 𝒙𝒳 conditioned on a multimodal prompt 𝒄 (text, audio, MIDI, video, etc.) such that:

  1. Speech quality: The model generates speech at quality matching or exceeding specialised TTS systems, with accurate prosody, emotion, and speaker identity.

  2. Music quality: The model generates music at quality matching or exceeding specialised music generation systems, with faithful instrumentation and musical structure.

  3. Sound effect quality: The model generates environmental sounds, Foley effects, and mechanical sounds with physical plausibility.

  4. Compositional generation: The model can generate mixtures (speech over background music, music with sound effects) with appropriate relative levels and temporal alignment.

The principal challenge is the distributional diversity: speech, music, and sound effects have very different statistical properties (spectral envelopes, temporal dynamics, harmonic structures). A single model must learn to represent all of these within a shared latent space without sacrificing quality in any modality.

Remark 63 (Connection to multimodal video generation).

The unified audio generation problem is a component of the broader unified multimodal generation problem discussed in ch:video-diffusion. A system that can generate video with synchronised audio, speech, music, and sound effects would subsume both the video and audio generation challenges.

Verifiable Provenance

As discussed in Detection and Watermarking, current watermarking systems face a fundamental tension between robustness and imperceptibility, bounded by the information-theoretic capacity of the watermarking channel (Proposition 52).

Problem 4 (Robust Unforgeable Audio Watermarking).

Design a watermarking system (θ,𝒟ψ) that embeds a payload 𝒃{0,1}L in audio signals with the following properties:

  1. Imperceptibility: The watermarked signal is perceptually indistinguishable from the original (PESQ degradation <0.1, ViSQOL degradation <0.1).

  2. Robustness: The payload is recoverable after any combination of the attacks in Remark 54 with bit error rate <103.

  3. Unforgeability: It is computationally infeasible (under standard cryptographic assumptions) for an adversary without the embedding key to produce audio that the decoder classifies as watermarked.

  4. Capacity: The payload length L64 bits, sufficient to encode a unique identifier for the generating system and a timestamp.

The system must operate in real-time (embedding and detection each require <10 ms per second of audio).

The unforgeability requirement connects to the cryptographic literature on digital signatures and commitment schemes. The challenge is that unlike traditional digital signatures (which protect a specific bitstring), audio watermarks must survive signal-processing transformations that alter the audio content while preserving its perceptual identity.

Key Idea.

The provenance frontier. Solving the verifiable provenance problem requires bridging information theory (capacity bounds), signal processing (robustness to transformations), perceptual science (imperceptibility), and cryptography (unforgeability). No existing system satisfies all four requirements simultaneously. Progress on this problem is essential for building a trustworthy audio ecosystem in which the provenance of every audio signal can be verified.

Exercises

Exercise 23 (STFT and Mel Spectrogram Frame Count).

Consider an audio signal 𝒙N sampled at rate fs (samples per second). The Short-Time Fourier Transform (STFT) is computed with a window of length W samples and a hop size of H samples. A mel filterbank with M triangular filters is applied to the magnitude STFT to produce the mel spectrogram 𝐒F×M.

  1. Show that the number of STFT frames is (Frame Count)F=NWH+1. Hint: The k-th frame is centred at sample kH+W/2 (for k=0,1,), and the last valid frame must have its window entirely within the signal.

  2. For an audio signal of duration D seconds at sample rate fs=22,050 Hz, window length W=1,024, and hop size H=256, compute the number of frames F as a function of D. What is F for D=10 s?

  3. The mel spectrogram has shape F×M. For M=128 mel bins, compute the total number of real-valued entries in the mel spectrogram for D=10 s. Compare this to the number of samples N=fsD in the original waveform. What is the compression ratio?

  4. The frequency resolution of the STFT is Δf=fs/W and the time resolution is Δt=H/fs. Show that their product satisfies the uncertainty principle: (Uncertainty)ΔfΔt=HW1, with equality when H=W (no overlap). Explain the trade-off between time and frequency resolution.

  5. The mel scale is approximately linear below 1 kHz and logarithmic above 1 kHz. Using the definition m=2595log10(1+f/700) (where f is in Hz and m is in mels), compute the mel values at f=1,000 Hz and f=8,000 Hz. Verify that the interval from 1 kHz to 8 kHz in Hz (a factor of 8) corresponds to a much smaller ratio in mel space (less than a factor of 2).

Exercise 24 (RVQ Approximation Error Bound).

Consider a residual vector quantiser (RVQ) with Q stages, each using a codebook of K vectors in d. Let 𝒛d be a latent vector to be quantised.

  1. Define the RVQ encoding recursively: 𝒓0=𝒛, and for q=1,,Q, (RVQ Recursive)cq=arg mink{1,,K}𝒓q1𝒆k(q)2,𝒓q=𝒓q1𝒆cq(q). Show that the reconstruction is 𝒛^=q=1Q𝒆cq(q) and the quantisation error is 𝒛𝒛^=𝒓Q.

  2. Suppose each codebook is “well-distributed” in the sense that for any vector 𝒓d with 𝒓R, there exists a codebook vector 𝒆k(q) satisfying 𝒓𝒆k(q)ρ𝒓 for a contraction factor ρ(0,1). Prove that the RVQ quantisation error satisfies (RVQ Error Bound)𝒓QρQ𝒛. Hint: Show by induction that 𝒓qρ𝒓q1.

  3. The contraction factor ρ depends on the codebook size K and dimension d. For a random codebook with K vectors drawn uniformly on the unit sphere in d, argue (using a covering number argument) that ρ1cK2/d for a dimension-dependent constant c>0. What does this imply about the number of codebook entries needed for a given target error in high dimensions?

  4. Compare the total number of bits used by RVQ (Qlog2K bits per vector) with scalar quantisation (db bits per vector, where b is the number of bits per dimension). For what range of Q,K,d,b does RVQ use fewer bits for the same reconstruction error?

  5. In practice, RVQ codebooks are trained jointly with the encoder and decoder using the straight-through estimator. Explain why the gradient 𝒛𝒓Q2 cannot be computed exactly through the arg min operation, and describe how the straight-through estimator approximates it.

Exercise 25 (Codebook Pattern Complexity).

Consider a neural audio codec with Q codebook levels producing T tokens per second. For a clip of duration D seconds, the total number of tokens per codebook is S=TD.

  1. Flattened pattern. All Q×S tokens are arranged in a single sequence in raster order (all codebooks for time step 1, then all codebooks for time step 2, etc.). The sequence length is Lflat=QS. For an autoregressive transformer with dmodel dimensions, the self-attention cost per layer is O(Lflat2dmodel). Express this in terms of Q, S, and dmodel.

  2. Delay pattern. Codebook q is delayed by q1 time steps. At each step, the model predicts one token per codebook, but codebook q uses context from time step t(q1). The effective sequence length is Ldelay=S+Q1. Show that the attention cost per layer is O((S+Q1)2Qdmodel) and compare to the flattened pattern.

  3. Parallel pattern. All Q codebooks at the same time step are predicted simultaneously (in parallel). The sequence length is Lpar=S. The model uses a single transformer forward pass per time step to predict Q tokens. Show that the attention cost per layer is O(S2dmodel), independent of Q. What is the trade-off in terms of modelling quality?

  4. For Q=8, T=50 tokens/second, D=10 s, and dmodel=2,048, compute the attention cost (in units of dmodel multiplications) for each of the three patterns. Which pattern is most efficient? By what factor?

  5. Discuss the quality-efficiency trade-off: the flattened pattern models all dependencies but is slowest; the parallel pattern is fastest but loses inter-codebook dependencies at the same time step. The delay pattern is a compromise. Under what assumptions about the statistical structure of codec tokens is the parallel pattern a good approximation?

Exercise 26 (OT Conditional Path for Audio Latents).

Consider a flow matching framework for audio latent diffusion. Let 𝒛0pdata be a clean audio latent and 𝒛1Normal(0,𝐈) be Gaussian noise. The optimal transport (OT) conditional path is defined as (OT PATH)𝒛t=(1t)𝒛0+t𝒛1,t[0,1].

  1. Compute the conditional velocity field 𝒗t(𝒛t|𝒛0,𝒛1)=ddt𝒛t and show that it is constant in t: (Const Velocity)𝒗t(𝒛t|𝒛0,𝒛1)=𝒛1𝒛0.

  2. Show that the conditional path has constant speed: 𝒗t(𝒛t|𝒛0,𝒛1)=𝒛1𝒛0 for all t. Interpret this geometrically: each sample travels along a straight line from data to noise at uniform speed.

  3. The flow matching training loss is (FM LOSS)FM(θ)=𝔼t,𝒛0,𝒛1[uθ(𝒛t,t)(𝒛1𝒛0)2], where tUniform[0,1], 𝒛0pdata, 𝒛1Normal(0,𝐈), and 𝒛t=(1t)𝒛0+t𝒛1. Show that this loss is equivalent (up to a constant) to the standard denoising score matching loss used in DDPM, by expressing the noise prediction target 𝝐=𝒛1 in terms of the velocity target 𝒛1𝒛0.

  4. At inference time, generation proceeds by solving the ODE d𝒛tdt=uθ(𝒛t,t) from t=1 (noise) to t=0 (data). Using the Euler method with N steps (step size Δt=1/N), write the update rule and show that the total number of model evaluations is exactly N.

  5. For audio applications, the latent 𝒛0 encodes a mel spectrogram of shape F×M (flattened to d with d=FM). The straightness of the OT path means that Euler integration with few steps (N=10 to 50) produces high-quality results. Explain why this is particularly advantageous for audio, where generation speed directly impacts usability.

Exercise 27 (Consistency Distillation for Audio Diffusion).

Let 𝒛t follow the probability flow ODE of an audio diffusion model: (PF ODE)d𝒛tdt=f(t)𝒛t+g(t)22σt𝝐teacher(𝒛t,t), where f(t),g(t) are the drift and diffusion coefficients of the forward SDE, σt=1e0t2f(s)ds is the noise standard deviation, and 𝝐teacher is the pretrained teacher denoiser.

  1. A consistency function fθ(𝒛t,t) maps any point on a PF-ODE trajectory to its origin 𝒛0. State the self-consistency property: for any s,t with 0s<t, fθ(𝒛t,t)=fθ(𝒛s,s) provided 𝒛s and 𝒛t lie on the same ODE trajectory.

  2. The consistency distillation loss is (CD LOSS)CD(θ)=𝔼tn,𝒛0,𝝐[d(fθ(𝒛tn+1,tn+1),fθ(𝒛^tn,tn))], where tn<tn+1 are adjacent time steps in a discretisation 0=t0<t1<<tT=1, 𝒛^tn is obtained by one ODE step from 𝒛tn+1 using the teacher, θ is the EMA of θ, and d is a distance function (e.g., 2 or LPIPS). Derive the gradient θCD and explain why the EMA target prevents collapse.

  3. After distillation, one-step generation produces 𝒛^0=fθ(𝒛T,T) from 𝒛TNormal(0,𝐈). The quality gap relative to the multi-step teacher depends on how well the consistency property is satisfied. Show that if the distillation loss achieves CDϵ, then the one-step output satisfies (CD Quality)𝔼[fθ(𝒛T,T)𝒛0]ϵT/Δtmin, where Δtmin=minn(tn+1tn) is the smallest step size. Hint: Apply the triangle inequality over the T/Δtmin steps.

  4. AudioLCM applies consistency distillation to an audio latent diffusion model (e.g., AudioLDM2). The key challenge is that the teacher's denoiser is a large U-Net operating on mel spectrogram latents. Discuss how the choice of distance function d affects the quality of the distilled model. Why might a perceptual distance (e.g., multi-scale STFT loss) be preferable to 2 distance for audio?

Exercise 28 (FAD as Squared Wasserstein Distance).

The Fréchet Audio Distance (FAD) between two sets of audio clips is defined analogously to FID: (FAD DEF)FAD=𝝁r𝝁g2+trace(𝚺r+𝚺g2(𝚺r𝚺g)1/2), where 𝝁r,𝚺r and 𝝁g,𝚺g are the means and covariances of features extracted from real and generated audio by a pretrained VGGish network.

  1. Recall from ch:sinkhorn that the squared 2-Wasserstein distance between two Gaussians p=Normal(𝝁p,𝚺p) and q=Normal(𝝁q,𝚺q) on d is given by the Dowson–Landau–Olkin–Pukelsheim formula: (W2 Gaussian)W22(p,q)=𝝁p𝝁q2+trace(𝚺p+𝚺q2(𝚺p1/2𝚺q𝚺p1/2)1/2). Show that trace((𝚺p1/2𝚺q𝚺p1/2)1/2)=trace((𝚺p𝚺q)1/2). Hint: Use the cyclic property of the trace and the fact that 𝚺p1/2𝚺q𝚺p1/2 and 𝚺p𝚺q have the same eigenvalues.

  2. Using part (a), conclude that FAD=W22(Normal(𝝁r,𝚺r),Normal(𝝁g,𝚺g)).

  3. Show that FAD=0 if and only if 𝝁r=𝝁g and 𝚺r=𝚺g. Hint: For the “if” direction, substitute directly. For the “only if” direction, note that FAD𝝁r𝝁g20, so FAD =0 implies 𝝁r=𝝁g. Then show that trace(𝚺r+𝚺g2(𝚺r𝚺g)1/2)=0 implies 𝚺r=𝚺g by considering the eigenvalues of 𝚺r1/2𝚺g𝚺r1/2.

  4. Compute the FAD between two univariate Gaussians Normal(μr,σr2) and Normal(μg,σg2). Verify that it simplifies to (μrμg)2+(σrσg)2.

Exercise 29 (Information-Theoretic Watermark Capacity).

Consider an audio watermarking system that embeds L bits into a host signal 𝒙N. The watermarked signal is 𝒙~=𝒙+𝒘, where 𝒘N is the watermark perturbation subject to a power constraint 1N𝒘2PW. The channel applies an attack 𝒜, producing 𝒚=𝒙~+𝒏, where 𝒏 is additive noise with power 1N𝔼[𝒏2]=PA.

  1. Modelling the watermarking system as a communication channel with input 𝒘 and output 𝒚𝒙=𝒘+𝒏 (assuming the host 𝒙 is known at both encoder and decoder as “side information”), show that the channel reduces to a standard additive white Gaussian noise (AWGN) channel with input power PW and noise power PA.

  2. Apply Shannon's channel coding theorem to bound the maximum payload: (Capacity)LN2log2(1+PWPA)bits.

  3. The imperceptibility constraint requires PWδ2 for a perceptual distortion budget δ. For δ=0.01 (watermark amplitude at most 1% of signal amplitude), PA=0.001 (SNR of 30 dB attack noise), and N=16,000 (one second at 16 kHz), compute the maximum payload L.

  4. In the rate-distortion framework, increasing the payload L requires either increasing PW (more distortion) or decreasing PA (weaker attack). Plot the Pareto frontier in the (PW,L) plane for fixed PA=0.001 and N=16,000. At what value of PW can we embed exactly L=128 bits?

  5. Discuss why practical watermarking systems achieve payloads well below the Shannon limit. Identify at least three factors that reduce the effective capacity (non-Gaussianity of the host, non-AWGN attacks, perceptual constraints that are more restrictive than a power constraint).

Exercise 30 (Bayesian Source Separation).

Consider an observed audio mixture 𝒙=i=1I𝒔i, where 𝒔iN are independent source signals (e.g., vocals, drums, bass, other). Each source has a prior distribution pi(𝒔i) learned by a diffusion model with score function 𝒔ilogpi(𝒔i).

  1. Write the posterior distribution over sources given the mixture: (Posterior)p(𝒔1,,𝒔I|𝒙)i=1Ipi(𝒔i)δ(𝒙i=1I𝒔i), where δ is the Dirac delta enforcing the mixture constraint.

  2. By introducing a relaxed mixture constraint p(𝒙|𝒔1,,𝒔I)=Normal(𝒙;i𝒔i,σ2𝐈), show that the score function for the relaxed posterior with respect to source j is (SEP Score)𝒔jlogp(𝒔1,,𝒔I|𝒙)=𝒔jlogpj(𝒔j)1σ2(i=1I𝒔i𝒙).

  3. Describe how to use Langevin dynamics with the score in part (b) to sample from the posterior, obtaining separated sources. Write the update rule for source j at iteration k: (Langevin SEP)𝒔j(k+1)=𝒔j(k)+η𝒔jlogp(𝒔1(k),,𝒔I(k)|𝒙)+2η𝝃j(k), where 𝝃j(k)Normal(0,𝐈).

  4. In the limit σ0, the relaxed constraint becomes exact. Explain why taking σ too small causes numerical instability, and discuss strategies for annealing σ during sampling.

  5. This approach requires running I diffusion models simultaneously (one per source). For I=4 sources and a diffusion model with T=50 denoising steps, compute the total number of neural network forward passes. Compare to a direct separation model (a single forward pass that outputs all I sources).

Exercise 31 (Beat-Synchronous Mixup).

Beat-synchronous mixup is a data augmentation strategy for music generation models. Given two music clips 𝒙A and 𝒙B with beat positions {b1A,b2A,} and {b1B,b2B,}, the augmented clip is formed by alternating segments from each clip at beat boundaries.

  1. Formalise beat-synchronous mixup. Define the augmented clip 𝒙mix by selecting, at each beat position, whether to use a segment from 𝒙A or 𝒙B according to independent Bernoulli coin flips zkBernoulli(p): (BEAT Mixup)𝒙mix[n]={𝒙A[n]if zk=0,𝒙B[n]if zk=1,for n[bk,bk+1).

  2. Show that the marginal distribution of beat positions in 𝒙mix is preserved. Specifically, let τk=bk+1bk denote the inter-beat interval. If both clips have the same tempo (so that τkA=τkB=τk for all k), show that the distribution of inter-beat intervals in 𝒙mix is identical to that in either original clip.

  3. The mixup operation introduces discontinuities at beat boundaries where the source switches. In the STFT domain, these discontinuities appear as broadband transients. Argue that crossfading over a window of W samples (typically W256 at 22 kHz, corresponding to 12 ms) eliminates perceptible clicks while preserving the beat structure, provided Wτk for all k.

  4. Show that for p=1/2, the expected fraction of 𝒙mix that comes from 𝒙A is 1/2, and the probability that any given beat segment comes from 𝒙A is independent of all other segments.

  5. Discuss the benefit of beat-synchronous mixup for training music generation models. How does it increase the effective diversity of the training set? What musical properties are preserved, and what properties are disrupted?

Exercise 32 (CFG for Audio Diffusion).

Let 𝝐θ(𝒛t,t,𝒄) be an audio diffusion denoiser conditioned on a text prompt embedding 𝒄, trained with classifier-free guidance by randomly dropping the conditioning with probability pdrop.

  1. Write the CFG-modified noise prediction: (CFG)𝝐~θ(𝒛t,t,𝒄)=(1+w)𝝐θ(𝒛t,t,𝒄)w𝝐θ(𝒛t,t,), where w0 is the guidance scale and denotes the null (unconditional) embedding. Show that this is equivalent to 𝝐~θ=𝝐θ(𝒛t,t,)+(1+w)[𝝐θ(𝒛t,t,𝒄)𝝐θ(𝒛t,t,)].

  2. Interpret CFG in terms of the score function. Recall that 𝝐θ(𝒛t,t,𝒄)σt𝒛logpt(𝒛t|𝒄). Show that the guided score is (CFG Score)𝝐~θσt𝒛logpt(𝒛t)+(1+w)𝒛logpt(𝒄|𝒛t). This is the score of the distribution pt(𝒛t)pt(𝒄|𝒛t)1+w, which sharpens the conditional distribution.

  3. For audio generation, the guidance scale w affects the spectral characteristics of the output. At high w (e.g., w>5), the generated audio tends to have exaggerated spectral peaks (louder harmonics, sharper formants) and reduced spectral diversity. Explain this phenomenon by analysing the effect of the (1+w) exponent on the conditional distribution: large w concentrates probability mass on the modes of p(𝒄|𝒛), producing outputs that are maximally “on-topic” but less diverse.

  4. In practice, w=3 to 5 is typical for text-to-audio generation. Derive the effect on the signal-to-noise ratio: if the conditional denoiser removes noise with efficiency η, show that CFG effectively increases the efficiency to η(1+w) for the conditional component, at the cost of amplifying the unconditional noise by a factor of w.

  5. Propose and justify a frequency-dependent guidance strategy: use a higher guidance scale wlow for low frequencies (below 1 kHz, where pitch and harmony are determined) and a lower scale whigh for high frequencies (above 4 kHz, where timbre and noise reside). Write the modified CFG formula and explain why this might produce more natural-sounding audio.

Exercise 33 (Timing Conditioning and Variable-Length Generation).

Stable Audio and similar systems condition the diffusion model on timing embeddings (tstart,tend) that specify the start and end times of the desired audio segment within a longer piece.

  1. The timing-conditioned diffusion model learns p(𝒙|𝒄,tstart,tend), where 𝒄 is the text prompt. Show that the marginal over all valid timing pairs gives the distribution over variable-length audio: (Timing Marginal)p(𝒙|𝒄)=p(𝒙|𝒄,ts,te)p(ts,te)dtsdte. This is a direct application of the law of total probability.

  2. In practice, at inference time one sets specific values ts=0,te=D for a desired duration D. Explain how this enables generating audio of any duration D at test time, even if the model was trained on clips of varying lengths.

  3. The timing embeddings ts,te are encoded as sinusoidal positional embeddings (analogous to those used in transformers). For a single timing value t, the embedding is γ(t)=[sin(2πf1t),cos(2πf1t),,sin(2πfDt),cos(2πfDt)]2D, where f1,,fD are geometrically spaced frequencies. Show that the inner product γ(t)𝖳γ(t) depends only on tt, making the embedding translation-equivariant.

  4. Derive the inner product explicitly: (Timing Inner)γ(t)𝖳γ(t)=j=1Dcos(2πfj(tt)). Hint: Use the identity sinαsinβ+cosαcosβ=cos(αβ).

  5. If D is much larger than the training set's maximum duration, the model is asked to extrapolate to unseen timing values. Discuss why sinusoidal embeddings enable moderate extrapolation (the inner product kernel is smooth and periodic) but fail for very large extrapolation (the model has never seen the corresponding timing values during training).

Exercise 34 (Snake Activation Function).

The Snake activation function, used in audio neural networks (BigVGAN, DAC), is defined as (Snake DEF)Snakeα(x)=x+1αsin2(αx), where α>0 is a learnable parameter controlling the frequency of the periodic component.

  1. Show that Snakeα is monotonically increasing for all α>0. To do this, compute the derivative: (Snake Deriv)ddxSnakeα(x)=1+sin(2αx). Then show that 1+sin(2αx)0 for all x, with equality only at isolated points.

  2. Since sin(2αx)1 with equality at 2αx=π/2+2kπ for k, the derivative satisfies Snakeα(x)0 for all x. Conclude that Snakeα is non-decreasing, and argue that it is strictly increasing (i.e., the zero-derivative points are isolated and do not form intervals).

  3. Show that as α0+, Snakeα(x)x+αx2+O(α3), so that Snake reduces to the identity plus a quadratic term. Hint: Use the Taylor expansion sin(αx)=αx(αx)36+O((αx)5).

  4. Show that as α, Snakeα(x)x+12α in the time-averaged sense: the oscillatory term 1αsin2(αx) averages to 12α over any interval of length π/α. Thus, for large α, Snake behaves approximately like the identity with a small positive bias.

  5. Explain why the periodic component of Snake is particularly well-suited for audio neural networks. The parameter α can learn to match the fundamental frequency of the audio being modelled, enabling the network to represent periodic waveforms with fewer parameters than non-periodic activations (ReLU, GELU). Compute the Fourier series of Snakeα(Asin(ωt)) for input Asin(ωt) and show that it contains harmonics at frequencies ω,2α±ω, and higher-order combinations, demonstrating that Snake naturally introduces harmonic content.

References

  1. BigVGAN: A Universal Neural Vocoder with Large-Scale Training

    Sang-gil Lee, Wei Ping, Boris Ginsburg, Bryan Catanzaro, Sungroh Yoon

    International Conference on Learning Representations · 2023

    BibTeX
    @inproceedings{lee2023bigvgan,
      title={{BigVGAN}: A Universal Neural Vocoder with Large-Scale Training},
      author={Lee, Sang-gil and Ping, Wei and Ginsburg, Boris and Catanzaro, Bryan and Yoon, Sungroh},
      booktitle={International Conference on Learning Representations},
      year={2023}
    }

    Conference paper

  2. Signal Estimation from Modified Short-Time Fourier Transform

    Daniel Griffin, Jae Lim

    IEEE Transactions on Acoustics, Speech, and Signal Processing, vol. 32, no. 2, pp. 236-243 · 1984

    BibTeX
    @article{griffin1984signal,
      title={Signal Estimation from Modified Short-Time {Fourier} Transform},
      author={Griffin, Daniel and Lim, Jae},
      journal={IEEE Transactions on Acoustics, Speech, and Signal Processing},
      volume={32},
      number={2},
      pages={236--243},
      year={1984},
      publisher={IEEE}
    }

    Journal article

  3. Online and Linear-Time Attention by Enforcing Monotonic Alignments

    Colin Raffel, Minh-Thang Luong, Peter J Liu, Ron J Weiss, Douglas Eck

    International Conference on Machine Learning, pp. 2837-2846 · 2017

    BibTeX
    @inproceedings{raffel2017online,
      title={Online and Linear-Time Attention by Enforcing Monotonic Alignments},
      author={Raffel, Colin and Luong, Minh-Thang and Liu, Peter J and Weiss, Ron J and Eck, Douglas},
      booktitle={International Conference on Machine Learning},
      pages={2837--2846},
      year={2017}
    }

    Conference paper

  4. RePaint: Inpainting using Denoising Diffusion Probabilistic Models

    Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, Luc Van Gool

    IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11461-11471 · 2022

    BibTeX
    @inproceedings{lugmayr2022repaint,
      title={{RePaint}: Inpainting using Denoising Diffusion Probabilistic Models},
      author={Lugmayr, Andreas and Danelljan, Martin and Romero, Andres and Yu, Fisher and Timofte, Radu and Van Gool, Luc},
      booktitle={IEEE/CVF Conference on Computer Vision and Pattern Recognition},
      pages={11461--11471},
      year={2022}
    }

    Conference paper

  5. Pop Music Transformer: Beat-based Modeling and Generation of Expressive Pop Piano Compositions

    Yu-Siang Huang, Yi-Hsuan Yang

    Proceedings of the 28th ACM International Conference on Multimedia, pp. 1180-1188 · 2020

    BibTeX
    @inproceedings{huang2020pop,
      title={Pop Music Transformer: Beat-based Modeling and Generation of Expressive Pop Piano Compositions},
      author={Huang, Yu-Siang and Yang, Yi-Hsuan},
      booktitle={Proceedings of the 28th ACM International Conference on Multimedia},
      pages={1180--1188},
      year={2020}
    }

    Conference paper

  6. Compound Word Transformer: Learning to Compose Full-Song Music over Dynamic Directed Hypergraphs

    Wen-Yi Hsiao, Jen-Yu Liu, Yin-Cheng Yeh, Yi-Hsuan Yang

    Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 1, pp. 178-186 · 2021

    BibTeX
    @inproceedings{hsiao2021compound,
      title={Compound Word Transformer: Learning to Compose Full-Song Music over Dynamic Directed Hypergraphs},
      author={Hsiao, Wen-Yi and Liu, Jen-Yu and Yeh, Yin-Cheng and Yang, Yi-Hsuan},
      booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
      volume={35},
      number={1},
      pages={178--186},
      year={2021}
    }

    Conference paper

  7. MusicBERT: Symbolic Music Understanding with Large-Scale Pre-Training

    Mingliang Zeng, Xu Tan, Rui Wang, Zeqian Ju, Tao Qin, Tie-Yan Liu

    Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 791-800 · 2021

    BibTeX
    @inproceedings{zeng2021musicbert,
      title={{MusicBERT}: Symbolic Music Understanding with Large-Scale Pre-Training},
      author={Zeng, Mingliang and Tan, Xu and Wang, Rui and Ju, Zeqian and Qin, Tao and Liu, Tie-Yan},
      booktitle={Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021},
      pages={791--800},
      year={2021}
    }

    Conference paper

  8. Structured denoising diffusion models in discrete state-spaces

    Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, Rianne van den Berg

    Advances in Neural Information Processing Systems, vol. 34, pp. 17981-17993 · 2021

    BibTeX
    @article{austin2021structured,
      title={Structured denoising diffusion models in discrete state-spaces},
      author={Austin, Jacob and Johnson, Daniel D and Ho, Jonathan and Tarlow, Daniel and van den Berg, Rianne},
      journal={Advances in Neural Information Processing Systems},
      volume={34},
      pages={17981--17993},
      year={2021}
    }

    Journal article

  9. MuLan: A Joint Embedding of Music Audio and Natural Language

    Qingqing Huang, Aren Jansen, Heiga Lee, Ravi Ganti, Judith Yue Li, Daniel P W Ellis

    arXiv preprint arXiv:2208.12415 · 2022

    BibTeX
    @article{huang2022mulan,
      title={{MuLan}: A Joint Embedding of Music Audio and Natural Language},
      author={Huang, Qingqing and Jansen, Aren and Lee, Heiga and Ganti, Ravi and Li, Judith Yue and Ellis, Daniel P W},
      journal={arXiv preprint arXiv:2208.12415},
      year={2022}
    }

    Journal article

  10. SoundStream: An End-to-End Neural Audio Codec

    Neil Zeghidour, Alejandro Luebs, Ahmed Omber, Marco Tagliasacchi, Aliaksei Severyn

    IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 30, pp. 495-507 · 2022

    BibTeX
    @article{zeghidour2022soundstream,
      title={{SoundStream}: An End-to-End Neural Audio Codec},
      author={Zeghidour, Neil and Luebs, Alejandro and Omber, Ahmed and Tagliasacchi, Marco and Severyn, Aliaksei},
      journal={IEEE/ACM Transactions on Audio, Speech, and Language Processing},
      volume={30},
      pages={495--507},
      year={2022},
      publisher={IEEE}
    }

    Journal article

  11. CNN Architectures for Large-Scale Audio Classification

    Shawn Hershey, Sourish Chaudhuri, Daniel P W Ellis, Jort F Gemmeke, Aren Jansen, R Channing Moore, Manoj Plakal, Devin Platt, et al.

    IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 131-135 · 2017

    BibTeX
    @inproceedings{hershey2017cnn,
      title={{CNN} Architectures for Large-Scale Audio Classification},
      author={Hershey, Shawn and Chaudhuri, Sourish and Ellis, Daniel P W and Gemmeke, Jort F and Jansen, Aren and Moore, R Channing and Plakal, Manoj and Platt, Devin and Sauber, Rif A and Seybold, Bryan and others},
      booktitle={IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
      pages={131--135},
      year={2017},
      organization={IEEE}
    }

    Conference paper

  12. ViSQOL: An Objective Speech Quality Model

    Andrew Hines, Jan Skoglund, Anil C. Kokaram, Naomi Harte

    EURASIP Journal on Audio, Speech, and Music Processing · 2015

    BibTeX
    @article{hines2015visqol,
      title={{ViSQOL}: An Objective Speech Quality Model},
      author={Hines, Andrew and Skoglund, Jan and Kokaram, Anil C. and Harte, Naomi},
      journal={EURASIP Journal on Audio, Speech, and Music Processing},
      year={2015}
    }

    Journal article

  13. WaveNet: A Generative Model for Raw Audio

    Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, et al.

    arXiv preprint arXiv:1609.03499 · 2016

    BibTeX
    @article{oord2016wavenet,
      author    = {van den Oord, Aaron and Dieleman, Sander and Zen, Heiga and Simonyan, Karen and Vinyals, Oriol and Graves, Alex and Kalchbrenner, Nal and Senior, Andrew and Kavukcuoglu, Koray},
      title     = {{WaveNet}: A Generative Model for Raw Audio},
      journal   = {arXiv preprint arXiv:1609.03499},
      year      = {2016}
    }

    Journal article

  14. Parallel WaveNet: Fast High-Fidelity Speech Synthesis

    Aaron van den Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu, George van den Driessche, Edward Lockhart, et al.

    arXiv preprint arXiv:1711.10433 · 2018

    BibTeX
    @article{oord2018parallel,
      title={Parallel {WaveNet}: Fast High-Fidelity Speech Synthesis},
      author={van den Oord, A{\"a}ron and Li, Yazhe and Babuschkin, Igor and Simonyan, Karen and Vinyals, Oriol and Kavukcuoglu, Koray and van den Driessche, George and Lockhart, Edward and Cobo, Luis C and Stimberg, Florian and others},
      journal={arXiv preprint arXiv:1711.10433},
      year={2018}
    }

    Journal article

  15. Efficient Neural Audio Synthesis

    Nal Kalchbrenner, Erich Elsen, Karen Simonyan, Seb Noury, Norman Casagrande, Edward Lockhart, Florian Stimberg, Aaron van den Oord, et al.

    arXiv preprint arXiv:1802.08435 · 2018

    BibTeX
    @article{kalchbrenner2018efficient,
      title={Efficient Neural Audio Synthesis},
      author={Kalchbrenner, Nal and Elsen, Erich and Simonyan, Karen and Noury, Seb and Casagrande, Norman and Lockhart, Edward and Stimberg, Florian and Oord, A{\"a}ron van den and Dieleman, Sander and Kavukcuoglu, Koray},
      journal={arXiv preprint arXiv:1802.08435},
      year={2018}
    }

    Journal article

  16. MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis

    Kundan Kumar, Rithesh Kumar, Thibault de Boissiere, Lucas Gestin, Wei Zhen Teoh, Jose Sotelo, Alexandre de Brebisson, Yoshua Bengio, et al.

    Advances in Neural Information Processing Systems · 2019

    BibTeX
    @inproceedings{kumar2019melgan,
      title={{MelGAN}: Generative Adversarial Networks for Conditional Waveform Synthesis},
      author={Kumar, Kundan and Kumar, Rithesh and de Boissiere, Thibault and Gestin, Lucas and Teoh, Wei Zhen and Sotelo, Jose and de Br{\'e}bisson, Alexandre and Bengio, Yoshua and Courville, Aaron C.},
      booktitle={Advances in Neural Information Processing Systems},
      year={2019}
    }

    Conference paper

  17. HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis

    Jungil Kong, Jaehyeon Kim, Jaekyoung Bae

    Advances in Neural Information Processing Systems, vol. 33, pp. 17022-17033 · 2020

    BibTeX
    @inproceedings{kong2020hifi,
      title={{HiFi-GAN}: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis},
      author={Kong, Jungil and Kim, Jaehyeon and Bae, Jaekyoung},
      booktitle={Advances in Neural Information Processing Systems},
      volume={33},
      pages={17022--17033},
      year={2020}
    }

    Conference paper

  18. UnivNet: A Neural Vocoder with Multi-Resolution Spectrogram Discriminators for High-Fidelity Waveform Generation

    Won Jang, Dan Lim, Jaesam Yoon, Bjoern Kim, Juntae Kim

    Interspeech, pp. 2207-2211 · 2021

    BibTeX
    @inproceedings{jang2021univnet,
      title={{UnivNet}: A Neural Vocoder with Multi-Resolution Spectrogram Discriminators for High-Fidelity Waveform Generation},
      author={Jang, Won and Lim, Dan and Yoon, Jaesam and Kim, Bjoern and Kim, Juntae},
      booktitle={Interspeech},
      pages={2207--2211},
      year={2021}
    }

    Conference paper

  19. DiffWave: A Versatile Diffusion Model for Audio Synthesis

    Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, Bryan Catanzaro

    International Conference on Learning Representations · 2021

    BibTeX
    @inproceedings{kong2021diffwave,
      title={{DiffWave}: A Versatile Diffusion Model for Audio Synthesis},
      author={Kong, Zhifeng and Ping, Wei and Huang, Jiaji and Zhao, Kexin and Catanzaro, Bryan},
      booktitle={International Conference on Learning Representations},
      year={2021}
    }

    Conference paper

  20. WaveGrad: Estimating Gradients for Waveform Generation

    Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, William Chan

    arXiv preprint arXiv:2009.00713 · 2020

    BibTeX
    @article{chen2020wavegrad,
      title={{WaveGrad}: Estimating Gradients for Waveform Generation},
      author={Chen, Nanxin and Zhang, Yu and Zen, Heiga and Weiss, Ron J and Norouzi, Mohammad and Chan, William},
      journal={arXiv preprint arXiv:2009.00713},
      year={2020}
    }

    Journal article

  21. AudioLDM: Text-to-Audio Generation with Latent Diffusion Models

    Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, Mark D. Plumbley

    International Conference on Machine Learning · 2023

    BibTeX
    @inproceedings{liu2023audioldm,
      title={{AudioLDM}: Text-to-Audio Generation with Latent Diffusion Models},
      author={Liu, Haohe and Chen, Zehua and Yuan, Yi and Mei, Xinhao and Liu, Xubo and Mandic, Danilo and Wang, Wenwu and Plumbley, Mark D.},
      booktitle={International Conference on Machine Learning},
      year={2023}
    }

    Conference paper

  22. Simple and Controllable Music Generation

    Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, Alexandre Defossez

    Advances in Neural Information Processing Systems, vol. 36 · 2024

    BibTeX
    @article{copet2024simple,
      title={Simple and Controllable Music Generation},
      author={Copet, Jade and Kreuk, Felix and Gat, Itai and Remez, Tal and Kant, David and Synnaeve, Gabriel and Adi, Yossi and D{\'e}fossez, Alexandre},
      journal={Advances in Neural Information Processing Systems},
      volume={36},
      year={2024}
    }

    Journal article

  23. Long-Form Music Generation with Latent Diffusion

    Zach Evans, Julian D. Parker, CJ Carr, Zack Zukowski, Josiah Taylor, Jordi Pons

    arXiv preprint arXiv:2404.10301 · 2024

    BibTeX
    @article{evans2024stable,
      title={Long-Form Music Generation with Latent Diffusion},
      author={Evans, Zach and Parker, Julian D. and Carr, CJ and Zukowski, Zack and Taylor, Josiah and Pons, Jordi},
      journal={arXiv preprint arXiv:2404.10301},
      year={2024}
    }

    Journal article

  24. SoundStream: An End-to-End Neural Audio Codec

    Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, Marco Tagliasacchi

    IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 30, pp. 495-507 · 2021

    BibTeX
    @article{zeghidour2021soundstream,
      title={{SoundStream}: An End-to-End Neural Audio Codec},
      author={Zeghidour, Neil and Luebs, Alejandro and Omran, Ahmed and Skoglund, Jan and Tagliasacchi, Marco},
      journal={IEEE/ACM Transactions on Audio, Speech, and Language Processing},
      volume={30},
      pages={495--507},
      year={2021}
    }

    Journal article

  25. High Fidelity Neural Audio Compression

    Alexandre Defossez, Jade Copet, Gabriel Synnaeve, Yossi Adi

    arXiv preprint arXiv:2210.13438 · 2022

    BibTeX
    @article{defossez2022high,
      title={High Fidelity Neural Audio Compression},
      author={D{\'e}fossez, Alexandre and Copet, Jade and Synnaeve, Gabriel and Adi, Yossi},
      journal={arXiv preprint arXiv:2210.13438},
      year={2022}
    }

    Journal article

  26. High-Fidelity Audio Compression with Improved RVQGAN

    Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, Kundan Kumar

    Advances in Neural Information Processing Systems, vol. 36 · 2024

    BibTeX
    @article{kumar2024high,
      title={High-Fidelity Audio Compression with Improved {RVQGAN}},
      author={Kumar, Rithesh and Seetharaman, Prem and Luebs, Alejandro and Kumar, Ishaan and Kumar, Kundan},
      journal={Advances in Neural Information Processing Systems},
      volume={36},
      year={2024}
    }

    Journal article

  27. Diff-BGM: A Diffusion Model for Video Background Music Generation

    Sizhe Li, Yiming Qin, Minghang Zheng, Xin Jin, Yang Liu

    Conference on Computer Vision and Pattern Recognition · 2024

    BibTeX
    @article{li2024diffbgm,
      title={{Diff-BGM}: A Diffusion Model for Video Background Music Generation},
      author={Li, Sizhe and Qin, Yiming and Zheng, Minghang and Jin, Xin and Liu, Yang},
      journal={Conference on Computer Vision and Pattern Recognition},
      year={2024}
    }

    Journal article

  28. VidMuse: A Simple Video-to-Music Generation Framework with Long-Short-Term Modeling

    Zeyue Tian, Zhaoyang Liu, Ruibin Yuan, Jiahao Pan, Qifeng Liu, Xu Tan, Qifeng Chen, Wei Xue, et al.

    Conference on Computer Vision and Pattern Recognition · 2025

    BibTeX
    @article{tian2024vidmuse,
      title={{VidMuse}: A Simple Video-to-Music Generation Framework with Long-Short-Term Modeling},
      author={Tian, Zeyue and Liu, Zhaoyang and Yuan, Ruibin and Pan, Jiahao and Liu, Qifeng and Tan, Xu and Chen, Qifeng and Xue, Wei and Guo, Yike},
      journal={Conference on Computer Vision and Pattern Recognition},
      year={2025}
    }

    Journal article

  29. Jukebox: A Generative Model for Music

    Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, Ilya Sutskever

    arXiv preprint arXiv:2005.00341 · 2020

    BibTeX
    @article{dhariwal2020jukebox,
      title={Jukebox: A Generative Model for Music},
      author={Dhariwal, Prafulla and Jun, Heewoo and Payne, Christine and Kim, Jong Wook and Radford, Alec and Sutskever, Ilya},
      journal={arXiv preprint arXiv:2005.00341},
      year={2020}
    }

    Journal article

  30. Music Transformer: Generating Music with Long-Term Structure

    Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Noam Shazeer, Ian Simon, Curtis Hawthorne, Andrew M Dai, Matthew D Hoffman, et al.

    International Conference on Learning Representations · 2019

    BibTeX
    @inproceedings{huang2019music,
      title={Music Transformer: Generating Music with Long-Term Structure},
      author={Huang, Cheng-Zhi Anna and Vaswani, Ashish and Uszkoreit, Jakob and Shazeer, Noam and Simon, Ian and Hawthorne, Curtis and Dai, Andrew M and Hoffman, Matthew D and Dinculescu, Monica and Eck, Douglas},
      booktitle={International Conference on Learning Representations},
      year={2019}
    }

    Conference paper

  31. MusicLM: Generating Music From Text

    Andrea Agostinelli, Timo I. Denk, Zalan Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, et al.

    arXiv preprint arXiv:2301.11325 · 2023

    BibTeX
    @article{agostinelli2023musiclm,
      title={{MusicLM}: Generating Music From Text},
      author={Agostinelli, Andrea and Denk, Timo I. and Borsos, Zal{\'a}n and Engel, Jesse and Verzetti, Mauro and Caillon, Antoine and Huang, Qingqing and Jansen, Aren and Roberts, Adam and Tagliasacchi, Marco and Sharifi, Matt and Zeghidour, Neil and Frank, Christian},
      journal={arXiv preprint arXiv:2301.11325},
      year={2023}
    }

    Journal article