32 Generative Models in Retrieval: From Memory to Meaning

The Philosophy of Retrieval – What Does It Mean to Remember?

You have lost your keys. You stand in the hallway, pockets empty, mind racing. Where did you last see them? You close your eyes. An image surfaces – the kitchen counter, the morning light, a coffee cup beside a set of keys. You walk to the kitchen. There they are.

What just happened? On the surface, a trivial act of memory. Beneath the surface, one of the most profound computational processes known to science. Your brain, without being told where to look, without scanning every object in the house, without any external index or catalogue, reconstructed a scene from partial cues, identified the relevant fragment, and translated that fragment into an action. It did not search for the keys in any mechanical sense. It remembered them – which is to say, it generated a plausible past from compressed traces of experience.

This chapter is about retrieval – the problem of finding relevant information given a need. But we begin with a confession: the word “retrieval” is misleading. It suggests a filing cabinet, a librarian pulling a book from a shelf, a database returning a row. These metaphors imply that the information exists somewhere in its original form, waiting to be fetched. The truth, as neuroscience and modern generative AI reveal, is far stranger. What we call retrieval is often closer to reconstruction, to generation, to the creative act of assembling meaning from fragments. Understanding this distinction – between retrieval as lookup and retrieval as generation – is the central theme of everything that follows.

Plato's Anamnesis: All Learning Is Recollection

Twenty-four centuries ago, Plato proposed a theory of knowledge that sounds, to modern ears, both mystical and eerily prescient. In the Meno dialogue, Socrates demonstrates that an uneducated slave boy can derive geometric truths when prompted with the right questions. Socrates' conclusion is startling: the boy did not learn these truths; he remembered them. The soul, Plato argues, has encountered all truths in a prior existence; what we call learning is merely anamnesis – recollection of what we already know.

Strip away the metaphysics, and Plato's insight contains a kernel that resonates with modern retrieval systems. The idea that knowledge is not acquired but recovered – that the answer to a question already exists, latent, in some compressed form, and the task is to surface it – is precisely what happens when a large language model answers a query. The model does not search an external database. It reaches into its parameters, into the compressed traces of billions of documents encountered during training, and reconstructs an answer. The parameters are the model's soul; inference is its anamnesis.

Aristotle's Associations: Similarity, Contiguity, Contrast

Where Plato looked upward to transcendent Forms, his student Aristotle looked around at the empirical world. In De Memoria et Reminiscentia (On Memory and Reminiscence), Aristotle proposed that memories are retrieved through three principles of association:

1. Similarity. We recall things that resemble the current stimulus. The smell of cinnamon recalls your grandmother's kitchen. In computational terms, this is nearest-neighbour search in a feature space.

2. Contiguity. We recall things that occurred together in space or time. The sight of a lecture hall recalls a specific class. This is co-occurrence statistics, the foundation of methods from TF-IDF to word embeddings [1].

3. Contrast. We recall things that are opposite to the current stimulus. The word “hot” triggers “cold.” This is negative sampling, the mechanism by which modern embedding methods learn discriminative representations.

It is remarkable how directly these twenty-three-century-old principles map onto the core mechanisms of modern information retrieval. Similarity retrieval is the foundation of vector search. Contiguity is exploited by language models that learn from co-occurrence patterns. Contrast is the training signal in contrastive learning and hard negative mining. Aristotle did not have matrices, but he had the right intuitions.

Borges' Library of Babel: The Retrieval Problem

In 1941, Jorge Luis Borges published a short story that is, arguably, the most profound meditation on the retrieval problem ever written. “The Library of Babel” describes a universe consisting of an enormous library containing every possible book of a fixed length – every possible arrangement of twenty-five characters across 410 pages. The library contains every truth ever uttered and every truth yet to be discovered. It contains the complete works of Shakespeare, the cure for cancer, the biography of every person who will ever live. It also contains every possible variation of these texts with a single character changed, and every text that is pure gibberish.

The inhabitants of the library are driven mad. Not because the knowledge is unavailable – it is all there, every last word of it – but because they cannot find it. The ratio of meaningful books to meaningless ones is vanishingly small; a librarian could wander the hexagonal galleries for a lifetime without finding a single coherent sentence. Storage is solved. Retrieval is impossible.

The modern internet is eerily close to Borges' vision. As of 2024, the indexed web contains hundreds of billions of pages. The vast majority are irrelevant to any given query. Google's PageRank algorithm, the neural retrieval models we shall study in later sections, and the retrieval-augmented generation pipelines that power modern AI assistants are all, in essence, solutions to the Library of Babel problem: mechanisms for finding the needle of relevance in the haystack of all possible texts.

Wittgenstein and the Contextuality of Meaning

There is a deeper problem that Borges' library does not address. Even if we could find the right book, would we know it was the right one? Relevance is not a property of documents in isolation; it depends on the query, the querier, the context, and the purpose.

Ludwig Wittgenstein, in his Philosophical Investigations (1953), argued that the meaning of a word is not some fixed, Platonic entity but its use in a language game – the social, contextual, practical circumstances in which it is employed. The word “bank” means one thing to a fisherman and another to a financier. The query “How to deal with depression” means one thing to a psychologist and another to a meteorologist studying atmospheric pressure systems.

Wittgenstein's insight cuts at the heart of a fundamental limitation of early retrieval systems. Term-matching approaches like TF-IDF [5] treat queries and documents as bags of tokens, ignoring the vast web of contextual meaning that determines relevance. A query for “jaguar speed” could seek information about the animal, the car, or the Atari game. The string is identical; the meaning is determined entirely by context.

This is why the transition from lexical retrieval (matching words) to semantic retrieval (matching meanings) represents one of the most important revolutions in the history of the field. Dense retrieval models – which embed queries and documents into a shared vector space where distance reflects semantic similarity rather than lexical overlap – are the computational realisation of Wittgenstein's philosophy. They encode not strings, but uses, not words, but meanings in context.

The Formal Retrieval Problem

Having gathered philosophical motivation, we now make the retrieval problem precise. Every retrieval system, from the ancient librarian of Alexandria to GPT-4's retrieval-augmented generation pipeline, can be understood as an instance of the following formalism.

The retrieval spectrum shown in fig:ret:retrieval_spectrum illustrates the central trajectory of this chapter. We begin with exact match – the simplest and oldest form of retrieval – and progress through sparse methods, dense methods, and finally generative retrieval, where the system does not merely find relevant documents but generates answers, identifiers, or entirely new text that synthesises information from multiple sources.

Retrieval as Reconstruction

We arrive at the central philosophical claim of this chapter, the thread that will weave through every section that follows.

This reconstruction view unifies several threads that have traditionally been studied in isolation. Autoregressive language models that answer questions from their parameters are performing retrieval by generation. Retrieval-augmented generation systems that condition on retrieved passages are performing generation by retrieval. The boundary between the two dissolves once we recognise that both are instances of the same fundamental operation: reconstructing relevant information from a combination of parametric memory (weights) and non-parametric memory (the corpus).

The Neuroscience of Memory and Retrieval

If we wish to build machines that retrieve information as fluidly as the human mind, we would do well to understand how the mind itself accomplishes this feat. The brain is, among many other things, the most sophisticated retrieval engine ever constructed – capable of recovering a specific childhood memory from a whiff of perfume, of completing a partial melody from its first three notes, of answering a novel question by synthesising information from decades of experience. How does it do this?

The answer, as we shall see, offers profound lessons for the design of artificial retrieval systems – and reveals that the brain's approach to retrieval is far closer to generation than to lookup.

The Hippocampus as a Retrieval Engine

In 1953, a young man known as patient H.M. underwent bilateral surgical removal of his medial temporal lobes, including most of his hippocampus, as a treatment for severe epilepsy. The seizures stopped, but something else was lost: H.M. could no longer form new long-term memories. He could remember his childhood, carry on a conversation, and solve puzzles, but five minutes after meeting someone, he had no recollection of the encounter [41].

H.M.'s tragedy transformed our understanding of memory. It revealed that the hippocampus is not a passive storage vault but an active indexing and retrieval system – a biological analogue of the very retrieval engines we build in software. The hippocampus performs two critical functions:

Rapid encoding.

The hippocampus can encode new associations in a single exposure – a property called one-shot learning in the machine learning literature. When you meet someone new, your hippocampus rapidly binds together the person's face, name, voice, and the context of the encounter into a unified memory trace. This is achieved through a form of sparse, pattern-separated encoding: the hippocampal dentate gyrus uses highly sparse activity patterns to represent distinct memories, ensuring that similar experiences are stored in non-overlapping representations [35].

Pattern completion.

When a partial cue arrives – a familiar face, a fragment of a song, the smell of rain – the hippocampus performs pattern completion: it takes the partial input, matches it against stored memory traces, and reactivates the full memory, including components that were not present in the cue. You see the face and recall the name, the voice, the restaurant where you met. This is not lookup; it is reconstruction. The hippocampus does not store memories as complete records. It stores indices – compressed pointers that, when activated, trigger the reconstruction of the full memory from distributed cortical representations.

The parallels to modern retrieval systems are striking. The hippocampus is the index; the neocortex is the document store. Pattern completion is approximate nearest-neighbour search – finding the stored representation most similar to the query cue and then expanding it into a full response.

Consider what happens when you try to remember a colleague's name. The cue – perhaps the person's face – arrives at the hippocampus. The hippocampus does not search through every memory trace sequentially (that would be $O(N)$ and the brain has billions of traces). Instead, the sparse activity pattern generated by the cue directly activates a small number of candidate memory traces through content-addressable lookup – the biological equivalent of hash-based or graph-based approximate nearest-neighbour search. The best-matching trace “wins” through a competitive process (lateral inhibition in CA3), and the full memory – name, voice, context – is reconstructed from the index pointer.

The computational cost of this biological retrieval is remarkable. The hippocampus achieves sublinear query time through three mechanisms that have direct analogues in computer science: locality-sensitive hashing (through sparse, random projections in the dentate gyrus), winner-take-all competition (through lateral inhibition in CA3), and iterative refinement (through recurrent dynamics that converge to an attractor state). Each of these mechanisms has inspired a family of approximate nearest-neighbour algorithms.

Episodic and Semantic Memory

In 1972, the cognitive psychologist Endel Tulving proposed a distinction that has profoundly shaped our understanding of memory and, by extension, of retrieval [36]. He argued that long-term memory comprises at least two qualitatively different systems:

Episodic memory.

Episodic memory stores specific, personally experienced events, bound to a particular time and place. You remember eating breakfast this morning, your first day of school, the moment you heard a particular piece of news. These memories are rich in contextual detail and are associated with a subjective sense of “re-experiencing” – what Tulving called autonoetic consciousness.

Semantic memory.

Semantic memory stores general knowledge about the world, abstracted from any particular experience. You know that Paris is the capital of France, that water freezes at $0^{\circ }$C, and that mammals are warm-blooded. You do not remember when or where you learned these facts; the knowledge has been separated from its experiential context.

The distinction maps elegantly onto two modes of artificial retrieval:

• Passage retrieval is analogous to episodic memory: the system returns specific documents or passages, each tied to a particular source and context. The user “re-experiences” the original text.

• Parametric knowledge in a language model is analogous to semantic memory: the model “knows” facts but has lost the connection to the specific training documents from which they were learned. It cannot cite its sources, just as you cannot remember where you learned that $2+2=4$.

Engrams: The Physical Trace of Memory

Where does a memory live? The question haunted neuroscience for a century. In the early 1900s, Richard Semon coined the term engram to denote the physical trace that a memory leaves in the brain – the material substrate of recollection. Karl Lashley spent decades searching for the engram, systematically lesioning rat brains and testing for memory loss. His conclusion, formulated as the “law of mass action,” was that memories are not localised to specific regions but are distributed across the cortex.

The mystery was partially resolved by Susumu Tonegawa's Nobel Prize-winning work, which demonstrated that specific populations of neurons – engram cells – are activated during memory encoding and reactivated during recall [34]. By using optogenetic techniques to artificially activate engram cells in mice, Tonegawa's group could implant false memories – compelling evidence that these cells are indeed the physical substrate of memory.

The engram concept connects directly to the representations learned by neural retrieval models. When a dense retrieval model encodes a document into a vector $𝒅\in {\mathrm{ℝ}}^d$, that vector is the document's engram – a compressed trace that captures the essential semantic content. When a query vector $𝒒$ activates the most similar document vectors through approximate nearest-neighbour search, the system is performing artificial pattern completion, reactivating the “memory” most compatible with the current cue.

Memory Consolidation: From Encoding to Retrieval

A memory is not created in a single instant. The process of transforming an experience into a retrievable memory involves three distinct stages, each with a computational analogue:

1. Encoding. The experience is perceived and processed by sensory cortices, then rapidly bound into a coherent memory trace by the hippocampus. Computational analogue: the document encoder that maps raw text to a dense embedding vector.

2. Storage (Consolidation). Over hours to weeks, the hippocampal memory trace is gradually transferred to the neocortex through a process of replay – during sleep, the hippocampus reactivates recent memories, and the neocortex slowly integrates them into its long-term representations. Computational analogue: the indexing phase, where document embeddings are organised into data structures (inverted indices, HNSW graphs, product quantisation codebooks) that support efficient retrieval.

3. Retrieval. A cue activates the hippocampal index, which triggers pattern completion and reactivates the relevant cortical representations. Computational analogue: the query processing phase, where the query embedding is compared against indexed document embeddings to produce a ranked list of results.

The consolidation stage is particularly illuminating. The brain does not simply dump memories into long-term storage; it replays them, extracting regularities, building abstractions, and gradually transforming episodic memories into semantic knowledge. During slow-wave sleep, the hippocampus replays the day's experiences at compressed timescales (up to 20 times faster than real time), broadcasting them to the neocortex, which gradually integrates the new information into its existing knowledge structures. Over weeks and months, a memory that was once hippocampal-dependent – vivid, contextual, episodic – becomes cortical – abstract, decontextualised, semantic.

This process bears a remarkable resemblance to the distillation and fine-tuning procedures used to train generative retrieval models, where a large corpus is gradually compressed into the parameters of a neural network through repeated exposure during training. The training loop of a neural retrieval model is, computationally speaking, a form of artificial memory consolidation: raw data (experiences) are replayed (iterated over in training epochs) and gradually compressed into the model's parameters (long-term cortical representations).

Hebbian Learning and Associative Retrieval

In 1949, Donald Hebb proposed the principle that has become the most famous sentence in neuroscience: “When an axon of cell A is near enough to excite cell B and repeatedly or persistently takes part in firing it, some growth process or metabolic change takes place in one or both cells such that A's efficiency, as one of the cells firing B, is increased.” Or, as it is colloquially summarised: neurons that fire together wire together.

This principle – that synaptic connections strengthen between neurons that are co-activated – is the biological foundation of associative retrieval. If you repeatedly experience coffee and mornings together, the neural representations of “coffee” and “morning” become linked by strengthened synapses. Activating one pattern (the smell of coffee) automatically activates the other (the sense of morning).

The Hebbian rule can be formalised as a simple outer product update: (Hebbian Update)$$\begin{array}{ccc}& \mathrm{\Delta }𝐖=\eta 𝒙{𝒙}^{\top },& \text{<span class="tml-eqn"></span>}\end{array}$$ where $𝐖\in {\mathrm{ℝ}}^{d\times d}$ is the weight (synapse) matrix, $𝒙\in {\mathrm{ℝ}}^d$ is the activity pattern being stored, and $\eta$ is the learning rate. After storing $M$ patterns ${𝒙}_1,\dots ,{𝒙}_M$, the weight matrix becomes: (Hebbian Weight)$$\begin{array}{ccc}& 𝐖=\eta \sum _{m=1}^M{𝒙}_m{𝒙}_m^{\top }.& \text{<span class="tml-eqn"></span>}\end{array}$$

Given a partial cue $\widetilde{𝒙}$ that is close to one of the stored patterns, the network performs retrieval by computing: (Hebbian Retrieval)$$\begin{array}{ccc}& \widehat{𝒙}=𝐖\widetilde{𝒙}=\eta \sum _{m=1}^M{𝒙}_m({𝒙}_m^{\top }\widetilde{𝒙})=\eta \sum _{m=1}^M⟨{𝒙}_m,\widetilde{𝒙}⟩{𝒙}_m.& \text{<span class="tml-eqn"></span>}\end{array}$$

This is nothing other than a weighted sum of stored patterns, where the weight on each pattern is its inner product with the query. If the stored patterns are orthogonal, the cue $\widetilde{𝒙}\approx {𝒙}_k$ will produce $\widehat{𝒙}\approx \eta {\Vert {𝒙}_k\Vert }^2{𝒙}_k$ – perfect retrieval. When the patterns are not orthogonal (which is the typical case when $M>d$), the retrieved pattern is contaminated by crosstalk from other stored patterns: (Hebbian Crosstalk)$$\begin{array}{ccc}& \widehat{𝒙}=\eta ⟨{𝒙}_k,\widetilde{𝒙}⟩{𝒙}_k+\eta \sum _{m\ne k}⟨{𝒙}_m,\widetilde{𝒙}⟩{𝒙}_m.& \text{<span class="tml-eqn"></span>}\end{array}$$ The first term is the signal (the desired memory); the second is the interference (crosstalk from other memories). Successful retrieval requires that the signal dominates the interference, which imposes an upper bound on the number of patterns that can be stored – the capacity of the associative memory.

This is the simplest form of content-addressable memory, and it is the mathematical ancestor of modern attention mechanisms. The progression from Hebb's rule to Hopfield networks to the transformer's attention mechanism is one of the most remarkable intellectual lineages in the history of computing – a direct line from neuroscience to the systems that now power web search and question answering.

Sparse Distributed Representations in the Brain

The brain faces the same challenge as any large-scale retrieval system: how to store and retrieve a vast number of patterns without catastrophic interference. If every memory activated a large fraction of neurons, memories would blur together hopelessly – a phenomenon called crosstalk in the associative memory literature.

The brain's solution is sparse distributed representations (SDRs): each memory is encoded by a small, randomly chosen subset of neurons from a large population. In the hippocampal dentate gyrus, only about 2–4% of neurons are active for any given memory, ensuring that different memories have minimal overlap.

The capacity advantage of sparse coding is dramatic. For a Hopfield network with $d$ neurons storing dense patterns, the storage capacity is approximately $0.14d$ patterns before retrieval errors occur. With sparse patterns of sparsity $k/d$ (i.e., $k$ active neurons out of $d$), the capacity scales as $d^2/(k\log d)$ – a quadratic improvement.

This result has a direct lesson for retrieval system design: sparse representations enable higher-capacity memories. This principle underlies the success of sparse retrieval methods (where document representations have most entries equal to zero) and explains why hybrid systems that combine sparse and dense representations often outperform either approach alone.

Formal Foundations of Information Retrieval

We now turn from philosophy and neuroscience to the mathematical foundations upon which all modern retrieval systems are built. The preceding sections established that the brain solves the retrieval problem through content-addressable memory, sparse encoding, and pattern completion – mechanisms that are associative, parallel, and approximate. The formal methods we study now represent humanity's attempt to solve the same problem with a fundamentally different substrate: discrete symbols, sequential algorithms, and exact data structures. The tension between these two approaches – the brain's fuzzy, generative reconstruction versus the computer's precise, lookup-based retrieval – is not merely historical. It is the very tension that generative retrieval models seek to resolve by bringing the brain's strategy into the digital realm.

The story of information retrieval is a story of successive abstractions: from manual catalogues to Boolean queries, from Boolean queries to ranked retrieval, from hand-crafted scoring functions to learned representations. Each step brought us closer to the semantic ideal – retrieval based on meaning rather than surface form – and each left behind a mathematical framework that remains in use today.

From Library Catalogues to Automated Retrieval

The dream of automated information retrieval is older than the electronic computer. In 1945, Vannevar Bush – then Director of the U.S. Office of Scientific Research and Development – published a visionary essay in The Atlantic Monthly entitled “As We May Think.” Bush described a hypothetical device called the Memex: a mechanised desk that would store an individual's books, records, and communications on microfilm, with the ability to retrieve any item instantly via associative trails – chains of links between related items that could be created, shared, and traversed.

The first practical automated retrieval systems emerged in the 1960s, driven by the rapid growth of scientific literature. Gerard Salton's SMART system at Cornell, developed between 1961 and the 1990s, introduced many of the foundational concepts that remain central to the field: the vector space model, term weighting, relevance feedback, and automatic indexing [5].

The Cranfield experiments, conducted by Cyril Cleverdon at the College of Aeronautics in Cranfield, England, established the methodology of laboratory-based evaluation that persists in the TREC (Text REtrieval Conference) tradition to this day [2]. By constructing standardised test collections – a fixed corpus, a set of queries, and human relevance judgments – Cleverdon showed that retrieval systems could be compared objectively. This seemingly obvious insight was revolutionary: it transformed information retrieval from an art into an experimental science.

The Vector Space Model

The conceptual breakthrough that enabled quantitative retrieval was deceptively simple: represent both queries and documents as vectors in a high-dimensional space, and measure their similarity geometrically.

Let $V=\{w_1,w_2,\dots ,w_{|V|}\}$ be the vocabulary – the set of all distinct terms that appear in the corpus. Each document $d$ is represented as a vector $𝒅\in {\mathrm{ℝ}}^{|V|}$, where the $i$-th component $d_i$ encodes the importance of term $w_i$ in document $d$. Similarly, a query $q$ is represented as a vector $𝒒\in {\mathrm{ℝ}}^{|V|}$.

The relevance of document $d$ to query $q$ is then measured by the cosine similarity: (Cosine)$$\begin{array}{ccc}& \text{cos}(𝒒,𝒅)=\frac{⟨𝒒,𝒅⟩}{\Vert 𝒒\Vert \cdot \Vert 𝒅\Vert }=\frac{{\sum }_{i=1}^{|V|}{q}_i{d}_i}{\sqrt{{\sum }_{i=1}^{|V|}{q}_i^2}\sqrt{{\sum }_{i=1}^{|V|}{d}_i^2}}.& \text{<span class="tml-eqn"></span>}\end{array}$$

The critical question is: how should we set the weights $d_i$ and $q_i$? The simplest choice – binary indicators of term presence – is too crude. The term “the” appears in nearly every English document and provides no discrimination. The term “hippocampus” is rare and highly informative. We need a weighting scheme that captures both the local importance of a term within a document and its global discrimination power across the corpus.

TF-IDF: A First-Principles Derivation

The term frequency–inverse document frequency (TF-IDF) weighting scheme is perhaps the most important single formula in information retrieval. We derive it from first principles, motivating each component.

Term frequency (TF).

The more often a term $w$ appears in a document $d$, the more likely the document is about a topic related to $w$. The raw term frequency is simply the count: (RAW TF)$$\begin{array}{ccc}& \text{tf}(w,d)=f_{w,d},& \text{<span class="tml-eqn"></span>}\end{array}$$ where $f_{w,d}$ is the number of times $w$ appears in $d$. However, raw counts are problematic. A document that mentions “retrieval” 100 times is not 100 times more relevant than one that mentions it once; the relationship between count and relevance is sublinear. Common choices for dampening include the logarithmic transform: (LOG TF)$$\begin{array}{ccc}& {\text{tf}}_{\log }(w,d)=\{\begin{array}{cc}1+\log f_{w,d}& \text{if }{f}_{w,d}>0,\\ 0& \text{otherwise},\end{array}& \text{<span class="tml-eqn"></span>}\end{array}$$ and the sublinear normalisation used in BM25, which we derive below.

Document frequency and inverse document frequency (IDF).

A term that appears in many documents is a poor discriminator. The word “the” may have high term frequency in a relevant document, but it also has high term frequency in every document. We penalise such terms using the inverse document frequency: (IDF)$$\begin{array}{ccc}& \text{idf}(w)=\log \frac{N}{\text{df}(w)},& \text{<span class="tml-eqn"></span>}\end{array}$$ where $N$ is the total number of documents in the corpus and $\text{df}(w)$ is the number of documents containing term $w$. Terms that appear in every document get $\text{idf}=0$; terms that appear in a single document get $\text{idf}=\log N$.

The TF-IDF product.

The weight assigned to term $w$ in document $d$ is the product: (Tfidf)$$\begin{array}{ccc}& \text{tfidf}(w,d)=\text{tf}(w,d)\cdot \text{idf}(w).& \text{<span class="tml-eqn"></span>}\end{array}$$

BM25: From Probabilistic Principles to Practice

TF-IDF, for all its elegance, is an ad hoc weighting scheme. Can we derive a principled scoring function from probability theory? The answer is yes, and the result is BM25 (“Best Match 25”), the most widely used retrieval scoring function in the world, powering everything from Elasticsearch to the first stage of Google's ranking [6].

The derivation begins with the Probabilistic Relevance Framework (PRF), developed by Stephen Robertson and Karen Spärck Jones in the 1970s.

The Binary Independence Model

Assume each document is represented as a binary vector $𝒅\in {\{0,1\}}^{|V|}$, where $d_i=1$ if term $w_i$ appears in the document and $d_i=0$ otherwise. Given a query $q$, we wish to estimate the probability that document $d$ is relevant: $P(R=1|𝒅,q)$, where $R$ is a binary relevance variable.

By Bayes' theorem: (BIM Bayes)$$\begin{array}{ccc}& P(R=1|𝒅,q)=\frac{P(𝒅|R=1,q)P(R=1|q)}{P(𝒅|q)}.& \text{<span class="tml-eqn"></span>}\end{array}$$

The binary independence assumption states that, conditioned on relevance (or non-relevance), the terms in the document are independent: (BIM Independence)$$\begin{array}{ccc}& P(𝒅|R,q)=\prod _{i=1}^{|V|}P(d_i|R,q).& \text{<span class="tml-eqn"></span>}\end{array}$$

Taking the log-odds ratio of relevance to non-relevance and applying the independence assumption yields the Retrieval Status Value (RSV): (BIM RSV)$$\begin{array}{cccc}& \text{RSV}(d,q)& =\log \frac{P(R=1|𝒅,q)}{P(R=0|𝒅,q)}& \\ & & =\sum _{i:d_i=1,q_i=1}\log \frac{p_i(1-r_i)}{r_i(1-p_i)}+C,& \text{<span class="tml-eqn"></span>}\end{array}$$ where $p_i=P(d_i=1|R=1,q)$ is the probability that a relevant document contains term $w_i$, $r_i=P(d_i=1|R=0,q)$ is the probability that a non-relevant document contains term $w_i$, and $C$ is a query-dependent constant that does not affect the ranking.

The 2-Poisson Model and BM25 Derivation

The binary independence model ignores term frequency: it knows only whether a term is present or absent. To incorporate term frequency, Robertson and Walker proposed the 2-Poisson model [42], which assumes that term occurrences in relevant documents follow a mixture of two Poisson distributions – one for “elite” terms (terms central to the document's topic) and one for non-elite terms: (2poisson)$$\begin{array}{ccc}& P(f_{w,d}=k|R)=\lambda \cdot \frac{e^{-{\mu }_1}{\mu }_1^k}{k!}+(1-\lambda )\cdot \frac{e^{-{\mu }_2}{\mu }_2^k}{k!},& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${\mu }_1>{\mu }_2$ and $\lambda$ is the mixing weight.

The full derivation of BM25 from the 2-Poisson model involves substantial algebra. The key insight is that the log-odds contribution of term frequency can be approximated by a saturating function that increases with $f_{w,d}$ but approaches an asymptote. Robertson and Walker showed that the following formula provides an excellent approximation [6]:

The BM25 formula has a beautiful structure. The numerator $f_{w,d}\cdot (k_1+1)$ ensures that the contribution grows with term frequency. The denominator introduces saturation: as $f_{w,d}\to \infty$, the TF component approaches $k_1+1$, preventing any single term from dominating the score. The document length normalisation factor $1-b+b\cdot \text{dl}(d)/\text{avgdl}$ ensures that longer documents are not unfairly rewarded simply for containing more terms.

Probabilistic Language Models for Retrieval

An elegant alternative to the vector space approach is to view retrieval through the lens of language modelling. The core idea, introduced by Ponte and Croft (1998), is disarmingly simple: rank documents by the probability that the query was generated by the document's language model.

For each document $d$, we estimate a unigram language model $p(\cdot |d)$ that defines a probability distribution over terms. The probability of generating query $q=(w_1,w_2,\dots ,w_{|q|})$ from this model is: (Query Likelihood)$$\begin{array}{ccc}& p(q|d)=\prod _{i=1}^{|q|}p(w_i|d).& \text{<span class="tml-eqn"></span>}\end{array}$$

The maximum likelihood estimate of $p(w|d)$ is simply the relative frequency: $\hat{p}(w|d)=f_{w,d}/|d|$, where $|d|$ is the length of document $d$. But this assigns zero probability to query terms not in the document, making any such document irretrievable. The solution is smoothing:

Jelinek-Mercer smoothing.

(JM Smoothing)$$\begin{array}{ccc}& p_{\text{JM}}(w|d)=(1-\lambda )\frac{f_{w,d}}{|d|}+\lambda p(w|𝒞),& \text{<span class="tml-eqn"></span>}\end{array}$$ where $\lambda \in (0,1)$ is a smoothing parameter and $p(w|𝒞)$ is the background language model estimated from the entire corpus.

Dirichlet smoothing.

(Dirichlet Smoothing)$$\begin{array}{ccc}& p_{\mu }(w|d)=\frac{f_{w,d}+\mu p(w|𝒞)}{|d|+\mu },& \text{<span class="tml-eqn"></span>}\end{array}$$ where $\mu >0$ is the Dirichlet prior parameter. This has the appealing property that shorter documents are smoothed more heavily (they have less evidence), while longer documents are smoothed less.

Taking the log of the query likelihood and dropping query-independent constants, the ranking function becomes: (LM Ranking)$$\begin{array}{ccc}& \log p(q|d)=\sum _{w\in q}\log p(w|d)\propto \sum _{w\in q\cap d}\log \frac{p(w|d)}{p(w|𝒞)}+C(q),& \text{<span class="tml-eqn"></span>}\end{array}$$ where $C(q)$ depends only on the query. The log-ratio $\log (p(w|d)/p(w|𝒞))$ is positive when the term is more common in the document than in the corpus – a natural notion of term importance that arises automatically from the probabilistic framework, without the need to hand-design a weighting scheme like TF-IDF.

The language model approach to retrieval is philosophically significant because it inverts the traditional direction of the retrieval problem. Instead of asking “how similar is this document to the query?” (a matching perspective), it asks “how likely is it that this document generated the query?” (a generative perspective). This inversion – from discriminative matching to generative modelling – is a precursor to the full generative retrieval paradigm that we develop in later sections. When we later study models that generate document identifiers autoregressively, we are taking the language model view of retrieval to its logical conclusion: the entire retrieval process becomes an act of conditional generation.

Latent Semantic Indexing

The term-document matrix is vast and sparse. A typical corpus with $N={10}^6$ documents and vocabulary $|V|={10}^5$ yields a matrix with ${10}^{11}$ entries, the overwhelming majority of which are zero. Worse, the raw term space suffers from two plagues:

• Synonymy: different terms with the same meaning (“car” and “automobile”) have orthogonal representations, so a query for “car” will miss documents about “automobiles.”

• Polysemy: the same term with different meanings (“bank”) conflates unrelated concepts into a single dimension.

Latent Semantic Indexing (LSI), introduced by Deerwester et al. [7], addresses both problems by projecting the term-document matrix onto a low-dimensional subspace using the truncated singular value decomposition (SVD).

Let $𝐀\in {\mathrm{ℝ}}^{|V|\times N}$ be the TF-IDF-weighted term-document matrix. The rank-$k$ truncated SVD is: (SVD)$$\begin{array}{ccc}& 𝐀\approx {𝐀}_k={𝐔}_k{𝚺}_k{𝐕}_k^{\top },& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${𝐔}_k\in {\mathrm{ℝ}}^{|V|\times k}$ contains the top $k$ left singular vectors (a basis for the “topic” space), ${𝚺}_k\in {\mathrm{ℝ}}^{k\times k}$ is diagonal with the top $k$ singular values, and ${𝐕}_k\in {\mathrm{ℝ}}^{N\times k}$ contains the top $k$ right singular vectors.

Each document $d_j$ is now represented by the $j$-th row of ${𝐕}_k{𝚺}_k$ – a $k$-dimensional vector in the latent topic space. A query $𝒒$ is projected into the same space via: (LSI Query)$$\begin{array}{ccc}& \widehat{𝒒}={𝒒}^{\top }{𝐔}_k{𝚺}_k^{-1}.& \text{<span class="tml-eqn"></span>}\end{array}$$

Retrieval is performed by computing cosine similarity between the projected query and the projected documents.

Latent Dirichlet Allocation

Where LSI finds latent topics via matrix factorisation, Latent Dirichlet Allocation (LDA) discovers them via probabilistic generative modelling [8]. LDA posits that documents are generated by the following process:

1. For each topic $k\in \{1,\dots ,K\}$, draw a distribution over words: ${𝝓}_k\sim \text{Dir}(𝜷)$.

2. For each document $d$: enumerate

3. Draw a distribution over topics: ${𝜽}_d\sim \text{Dir}(𝜶)$.

4. For each word position $n$ in document $d$: enumerate

5. Draw a topic assignment: $z_{d,n}\sim \mathrm{Categorical}({𝜽}_d)$.

6. Draw the word: $w_{d,n}\sim \mathrm{Categorical}({𝝓}_{z_{d,n}})$. enumerate enumerate

For retrieval, LDA provides a principled way to represent documents as probability distributions over topics. The KL-divergence (or Jensen–Shannon divergence) between topic distributions serves as a similarity measure that captures thematic overlap regardless of lexical overlap: (LDA Retrieval)$$\begin{array}{ccc}& s_{\text{LDA}}(q,d)=-{𝖣}_{\text{KL}}({𝜽}_q\Vert {𝜽}_d)=-\sum _{k=1}^K{\theta }_{q,k}\log \frac{{\theta }_{q,k}}{{\theta }_{d,k}}.& \text{<span class="tml-eqn"></span>}\end{array}$$

LDA also reveals a deep connection between generative modelling and retrieval that foreshadows the central theme of this chapter. In LDA, each document is “explained” by a generative story: a mixture of topics, each of which generates words. Retrieval is then the inverse of generation – given the observed words (the query), infer the latent topics, and find documents with similar topic distributions. This pattern – train a generative model, then use its learned representations for retrieval – will recur throughout this chapter at increasing levels of sophistication, from topic models to variational autoencoders to autoregressive language models that generate document identifiers directly.

Evaluation Metrics for Information Retrieval

A retrieval system is only as good as our ability to measure its quality. The field of information retrieval has developed a rich vocabulary of evaluation metrics, each capturing a different aspect of retrieval quality. We present the major metrics here, as they will be used throughout the chapter.

Let $\pi =(d_{\pi (1)},\dots ,d_{\pi (N)})$ be a ranked list of documents produced by a retrieval system for a given query, and let $\text{rel}(d)\in \{0,1\}$ (or $\{0,1,2,\dots \}$ for graded relevance) denote the relevance of document $d$.

Inverted Indices: The Data Structure of Retrieval

All the scoring functions discussed above – TF-IDF, BM25, query likelihood – share a common computational pattern: they sum contributions from query terms that appear in a document. The naive approach – iterating over all $N$ documents for each query – is prohibitively expensive for web-scale corpora. The solution is the inverted index, the fundamental data structure of information retrieval.

The efficiency of the inverted index comes from a simple observation: most documents do not contain most query terms. For a query with $m$ terms, only documents that appear in at least one posting list are scored. If the average posting list length is $L$, the total work is $O(m\cdot L+k\log k)$ – typically far less than $O(N)$ for sparse queries on large corpora.

Comparison of Classical Methods

Method	Representation	Query Time	Handles Synonymy?	Interpretable?
TF-IDF [5]	Sparse, $|V|$-dim	$O(m\cdot L)$	No	Yes
BM25 [6]	Sparse, $|V|$-dim	$O(m\cdot L)$	No	Yes
LSI [7]	Dense, $k$-dim	$O(N\cdot k)$	Yes	No
LDA [8]	Dense, $K$-dim	$O(N\cdot K)$	Yes	Yes

Table 1 summarises the strengths and weaknesses of the classical methods. Sparse methods (TF-IDF, BM25) are fast and interpretable but cannot handle synonym mismatch. Dense methods (LSI, LDA) address synonymy through dimensionality reduction but sacrifice the efficiency of inverted indices – computing similarity in a dense space requires either scanning all documents or building specialised approximate nearest-neighbour indices, which we discuss in later sections.

The resolution of this tension – achieving the semantic power of dense representations with the efficiency of sparse retrieval – is one of the driving forces behind the dense retrieval revolution and the generative retrieval models that are the ultimate subject of this chapter.

It is worth noting that the methods surveyed so far represent only the scoring component of a retrieval system. The broader discipline of learning to rank [43] studies how to learn the scoring function from data, combining multiple signals – BM25 scores, PageRank, click-through data, document freshness, query-document feature vectors – into a single relevance prediction. Learning-to-rank methods (pointwise, pairwise, and listwise) laid the intellectual groundwork for the neural ranking models that would follow, establishing the principle that retrieval scoring should be learned, not hand-designed.

We have now laid the three pillars upon which the rest of this chapter stands. The philosophical framework of The Philosophy of Retrieval – What Does It Mean to Remember? established that retrieval is not mere lookup but reconstruction of meaning. The neuroscience of The Neuroscience of Memory and Retrieval showed that the brain implements retrieval through content-addressable memory, pattern completion, and generative reconstruction – mechanisms that neural networks are now learning to emulate. The formal foundations of Formal Foundations of Information Retrieval gave us the mathematical vocabulary – scoring functions, evaluation metrics, indexing structures – that we need to reason precisely about retrieval quality.

In the sections that follow, we will see how deep learning transforms every component of the retrieval pipeline. We begin with the dense retrieval revolution: the replacement of sparse bag-of-words representations with learned dense embeddings that capture semantic meaning. From there, we will progress to the most radical idea in modern retrieval: generative retrieval, where the index, the scoring function, and even the documents themselves are absorbed into the parameters of a single neural network. The journey from TF-IDF to generative retrieval is, in a sense, the journey from Aristotle's associations to Plato's anamnesis – from matching surface patterns to recovering meaning from compressed memory.

Dense Retrieval Revolution

The previous section laid the mathematical foundations of information retrieval in the language of probability, relevance, and term statistics. Those foundations are elegant, but they rest on an assumption so deeply embedded that it is easy to overlook: that documents and queries are best represented by the words they contain. The bag-of-words assumption – that the meaning of a text is captured by the multiset of its tokens – served the field for half a century. It powered TF-IDF, BM25, and the probabilistic models that built the modern web. But it harbours a fatal weakness: it cannot recognise that “automobile” and “car” refer to the same concept, that “heart attack” and “myocardial infarction” describe the same event, that a question about “the capital of France” should match a document that says “Paris is where the French government sits” even though the two texts share almost no words.

This is the vocabulary mismatch problem, and it is not a minor annoyance. Studies have repeatedly shown that users and authors use different words for the same concepts roughly 70–80% of the time [44]. Techniques like query expansion, stemming, and pseudo-relevance feedback provide partial remedies, but they are fundamentally patches on a representational framework that was never designed to capture meaning.

The dense retrieval revolution replaces this framework entirely. Instead of representing documents as sparse vectors over a vocabulary of hundreds of thousands of tokens, dense retrieval maps both queries and documents into a shared continuous vector space of a few hundred dimensions, where semantic similarity is captured by geometric proximity. A query about automobiles and a document about cars end up as nearby points – not because they share tokens, but because a neural network has learned that they share meaning.

The shift from sparse to dense is not merely a change of technique. It is a change of philosophy. Sparse retrieval asks: “Does this document contain the right words?” Dense retrieval asks: “Does this document mean the right thing?” The former is a syntactic question; the latter is a semantic one. And as we shall see, the answer to the semantic question requires us to learn representations – which in turn requires data, computation, and the entire apparatus of modern deep learning.

From Sparse to Dense: The Paradigm Shift

Let us make the contrast precise. In sparse retrieval, a document $d$ with vocabulary $V$ is represented as a vector ${𝒅}_{\mathrm{sparse}}\in {\mathrm{ℝ}}^{|V|}$ where most entries are zero. For BM25, the non-zero entries are the BM25 term weights. The dimensionality is enormous – typically $|V|\ge {10}^5$ – but the vectors are extremely sparse, with only a few hundred non-zero entries per document. This sparsity enables efficient inverted-index lookups, but it means that two documents can have zero overlap in their representations even when they are semantically identical.

In dense retrieval, we learn an encoder $f_{𝜽}$ that maps a text $x$ (query or document) to a dense vector $𝒙=f_{𝜽}(x)\in {\mathrm{ℝ}}^d$ where $d$ is typically 128–1024. Every dimension carries information; there are no structural zeros. Retrieval reduces to finding the documents whose dense vectors are closest to the query vector: (Dense Retrieval)$$\begin{array}{ccc}& d^{\ast }={\mathrm{arg\hspace{0.17em}max}}_{d\in 𝒟}\mathrm{sim}(f_{𝜽}(q),f_{𝜽}(d)),& \text{<span class="tml-eqn"></span>}\end{array}$$ where $\mathrm{sim}$ is typically the inner product or cosine similarity.

DPR: Dense Passage Retrieval

The work that crystallised the dense retrieval paradigm for open-domain question answering was Dense Passage Retrieval (DPR) [45], published by Karpukhin et al. in 2020. DPR demonstrated that a simple dual-encoder architecture, trained with the right contrastive objective, could outperform BM25 on multiple open-domain QA benchmarks – a result that many in the community found surprising, given the decades of engineering that had gone into sparse retrieval systems.

Architecture.

DPR uses two independent BERT encoders: one for queries (${\mathrm{BERT}}_Q$) and one for passages (${\mathrm{BERT}}_P$). Each encoder takes a text as input and produces a $d$-dimensional vector by extracting the [CLS] token representation from the final layer: (DPR Query)$$\begin{array}{cccc}& 𝒒& ={\mathrm{BERT}}_Q(q)[\text{𝙲𝙻𝚂}]\in {\mathrm{ℝ}}^d,& \text{<span class="tml-eqn"></span>}\\ & 𝒑& ={\mathrm{BERT}}_P(p)[\text{𝙲𝙻𝚂}]\in {\mathrm{ℝ}}^d.& \text{<span class="tml-eqn"></span>}\end{array}$$ The similarity between a query and a passage is the inner product: (DPR SIM)$$\begin{array}{ccc}& s(q,p)=⟨𝒒,𝒑⟩={𝒒}^{\top }𝒑.& \text{<span class="tml-eqn"></span>}\end{array}$$

Training objective.

Given a query $q_i$ with a positive passage $p_i^{+}$ and a set of negative passages $\{p_{i,1}^{-},\dots ,p_{i,m}^{-}\}$, DPR minimises the negative log-likelihood of the positive passage: (DPR LOSS)$$\begin{array}{ccc}& {ℒ}_{\mathrm{DPR}}=-\log \frac{\exp \left(⟨{𝒒}_i,{𝒑}_i^{+}⟩\right)}{\exp \left(⟨{𝒒}_i,{𝒑}_i^{+}⟩\right)+{\sum }_{j=1}^m\exp \left(⟨{𝒒}_i,{𝒑}_{i,j}^{-}⟩\right)}.& \text{<span class="tml-eqn"></span>}\end{array}$$ This is an instance of the InfoNCE loss [40]. The critical engineering insight in DPR is the use of in-batch negatives: within a mini-batch of $B$ query–passage pairs, the positive passage for query $q_i$ serves as a negative for all other queries $q_j$ ($j\ne i$), yielding $B-1$ negatives per query at no additional computational cost.

Hard negative mining.

Not all negatives are equally informative. Random negatives are easily distinguished from positives and provide weak gradient signal – their similarity to the query is so low that the softmax in (DPR LOSS) assigns them negligible weight, producing near-zero gradients. DPR addresses this by mining hard negatives – passages that are similar to the query according to BM25 but are not the gold positive. These negatives force the model to learn fine-grained distinctions that go beyond lexical overlap.

Concretely, for each training query, DPR retrieves the top-100 BM25 results and selects passages that do not contain the answer string as hard negatives. The original DPR paper found that using one BM25 hard negative per query, combined with in-batch negatives, yielded the best results. Adding a gold passage from a different query as a “gold negative” provided a further small improvement.

ColBERT: Late Interaction for Token-Level Matching

The dual-encoder architecture of DPR is efficient – query and document encodings are computed independently and can be cached – but it compresses the entire semantics of a passage into a single vector. This is a severe bottleneck. A passage about “the bank of the river” and a passage about “the bank approved the loan” might end up with similar single-vector representations because the word “bank” dominates both, even though the passages are about entirely different topics.

ColBERT (Contextualised Late Interaction over BERT) [46] addresses this by retaining token-level representations and deferring the matching computation to query time. Instead of a single vector per document, ColBERT produces a matrix: (Colbert Query)$$\begin{array}{cccc}& {𝐄}_q& ={\mathrm{BERT}}_Q(q)\in {\mathrm{ℝ}}^{n_q\times d},& \text{<span class="tml-eqn"></span>}\\ & {𝐄}_d& ={\mathrm{BERT}}_D(d)\in {\mathrm{ℝ}}^{n_d\times d},& \text{<span class="tml-eqn"></span>}\end{array}$$ where $n_q$ and $n_d$ are the token lengths of the query and document, respectively. The similarity is computed via the MaxSim operator: (Maxsim)$$\begin{array}{ccc}& s_{\mathrm{ColBERT}}(q,d)=\sum _{i=1}^{n_q}\underset{j\in \{1,\dots ,n_d\}}{\max }⟨{𝒒}_i,{𝒅}_j⟩,& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${𝒒}_i$ is the $i$-th row of ${𝐄}_q$ and ${𝒅}_j$ is the $j$-th row of ${𝐄}_d$. Each query token finds its best-matching document token, and these maximum similarities are summed.

The MaxSim operator has an intuitive interpretation: it performs a soft word-by-word alignment between query and document. If the query is “When was the Eiffel Tower built?”, the token “Eiffel” will align with the document token “Eiffel” (or “tower” or “Tour”), “built” will align with “constructed” or “completed”, and “when” will align with a date or year token. Each alignment contributes its similarity to the total score, so documents that match on more query tokens receive higher scores. This is far more expressive than a single inner product, which cannot distinguish which aspects of the query are matched.

Contrastive Learning for Retrieval

The DPR loss in (DPR LOSS) is a specific instance of the InfoNCE objective. Let us state the general form and analyse the role of temperature.

Given an anchor $𝒒$, a positive ${𝒑}^{+}$, and $K$ negatives $\{{𝒑}_1^{-},\dots ,{𝒑}_K^{-}\}$, the InfoNCE loss with temperature $\tau >0$ is: (Infonce)$$\begin{array}{ccc}& {ℒ}_{\mathrm{InfoNCE}}=-\log \frac{\exp \left(⟨𝒒,{𝒑}^{+}⟩/\tau \right)}{\exp \left(⟨𝒒,{𝒑}^{+}⟩/\tau \right)+{\sum }_{k=1}^K\exp \left(⟨𝒒,{𝒑}_k^{-}⟩/\tau \right)}.& \text{<span class="tml-eqn"></span>}\end{array}$$

The temperature $\tau$ controls the sharpness of the softmax distribution over candidates. Small $\tau$ makes the distribution peaky, assigning nearly all probability to the most similar candidate and creating strong gradients for hard negatives. Large $\tau$ makes the distribution uniform, treating all negatives more equally.

SPLADE: Learned Sparse Representations

Dense retrieval captures semantics but sacrifices the efficiency of inverted indices. Sparse retrieval is efficient but misses semantics. SPLADE (SParse Lexical AnD Expansion model) [47] bridges the two worlds by using a neural network to produce sparse representations – vectors in ${\mathrm{ℝ}}^{|V|}$ with mostly zero entries – but where the non-zero entries are learned rather than derived from term frequency statistics.

The key idea is to use a BERT-based masked language model head to predict a distribution over the entire vocabulary for each token position, then aggregate across positions: (Splade)$$\begin{array}{ccc}& w_j(x)=\sum _{i=1}^n\log (1+\mathrm{ReLU}({𝐖}_{\mathrm{MLM}}{𝒉}_i+𝒃{)}_j),& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${𝒉}_i$ is the contextual representation of the $i$-th token, ${𝐖}_{\mathrm{MLM}}$ is the MLM projection matrix, and $j$ indexes the vocabulary. The $\log (1+\mathrm{ReLU}(\cdot ))$ transformation ensures non-negative weights with a saturating growth. SPLADE adds a regularisation term to encourage sparsity: (Splade REG)$$\begin{array}{ccc}& {ℒ}_{\mathrm{SPLADE}}={ℒ}_{\mathrm{InfoNCE}}+{\lambda }_q{\Vert 𝒘(q)\Vert }_1+{\lambda }_d{\Vert 𝒘(d)\Vert }_1.& \text{<span class="tml-eqn"></span>}\end{array}$$

Because the resulting representations are sparse and non-negative, they can be stored and searched using standard inverted indices – the same infrastructure that powers BM25. Yet SPLADE performs term expansion: a document about “heart attack” will have non-zero weights for “cardiac”, “myocardial”, and “infarction” even if those words never appear in the text.

The SPLADE family has evolved through several versions. SPLADE v2 introduced document-side distillation from a cross-encoder, which significantly improved the quality of learned term weights. SPLADE++ further explored the design space of self-distillation and ensemble distillation, achieving results competitive with dense retrieval on the BEIR benchmark while retaining the efficiency of inverted-index retrieval.

Hybrid Retrieval

In practice, the best retrieval systems combine sparse and dense signals. The intuition is straightforward: sparse retrieval excels at exact lexical matching (entity names, rare terms, identifiers), while dense retrieval excels at semantic matching (paraphrases, synonyms, implicit relevance). The two failure modes are largely complementary.

The simplest hybrid approach is a linear interpolation of scores: (Hybrid Score)$$\begin{array}{ccc}& s_{\mathrm{hybrid}}(q,d)=\alpha \cdot s_{\mathrm{sparse}}(q,d)+(1-\alpha )\cdot s_{\mathrm{dense}}(q,d),& \text{<span class="tml-eqn"></span>}\end{array}$$ where $\alpha \in [0,1]$ is a mixing weight. Because the two score distributions may have different scales, it is common to normalise each score to $[0,1]$ before interpolation, for instance via min-max normalisation within the retrieved set: (Hybrid NORM)$$\begin{array}{ccc}& \hat{s}(q,d)=\frac{s(q,d)-{\min }_{d^{\prime }}s(q,d^{\prime })}{{\max }_{d^{\prime }}s(q,d^{\prime })-{\min }_{d^{\prime }}s(q,d^{\prime })}.& \text{<span class="tml-eqn"></span>}\end{array}$$

A more principled approach is Reciprocal Rank Fusion (RRF), which combines ranked lists rather than scores: (RRF)$$\begin{array}{ccc}& s_{\mathrm{RRF}}(q,d)=\sum _{r\in ℛ}\frac{1}{k+{\mathrm{rank}}_r(d)},& \text{<span class="tml-eqn"></span>}\end{array}$$ where $ℛ$ is the set of retrieval systems and $k$ is a smoothing constant (typically $k=60$).

RRF has the advantage of being score-agnostic: it requires only ranked lists, not calibrated scores, making it robust to the different score distributions produced by sparse and dense systems. Empirically, RRF with $k=60$ is remarkably difficult to beat with more sophisticated learned combination methods, a fact that has surprised many researchers.

Multi-stage retrieval.

Production search systems often use a multi-stage pipeline rather than a single fusion step:

1. First stage (recall). Run BM25 and/or a dual-encoder to retrieve a large candidate set ($\sim$1000 documents). The goal is high recall, not precision.

2. Second stage (reranking). Apply a cross-encoder reranker to score each candidate in the context of the query. The cross-encoder sees the concatenation $[q;d]$ and can model fine-grained query-document interactions.

3. Third stage (optional). Apply a generative model to rewrite or summarise the top candidates, further improving relevance.

This pipeline exploits the complementary strengths of each approach: fast but coarse first-stage retrieval followed by slow but precise reranking.

Architecture Comparison

The three dominant neural retrieval architectures – dual-encoder, cross-encoder, and late interaction – represent different points on the efficiency-effectiveness trade-off.

Dual-encoder.

The dual-encoder computes query and document representations independently. Document encodings can be precomputed and cached, so online latency scales only with the ANN search cost, not with the number of documents. However, the single-vector bottleneck limits the model's ability to capture fine-grained query-document interactions.

Cross-encoder.

The cross-encoder processes the concatenation $[q;d]$ through a single transformer, allowing full cross-attention between query and document tokens. This is the most expressive architecture – the model can learn arbitrary interactions between query and document words. But it is also the slowest: each query-document pair requires a separate forward pass, making it impractical as a first-stage retriever for large corpora. Cross-encoders are typically used as rerankers for a small set of candidates retrieved by a dual-encoder or BM25.

Late interaction.

ColBERT's late interaction occupies the middle ground: encodings are independent (enabling precomputation), but scoring involves token-level matching (enabling expressiveness). The storage cost is higher than a dual-encoder ($n_d\cdot d$ floats per document instead of $d$), but ColBERTv2 addresses this with residual compression, reducing storage to approximately 2 bytes per token.

fig:retrieval-architectures illustrates the key structural differences.

Method	Top-5	Top-20	Top-100
BM25	43.8	62.9	78.3
DPR	68.3	80.1	86.1
DPR + BM25 hard negs	72.1	82.6	87.3
ColBERT	73.2	83.9	88.7
SPLADE v2	72.8	83.5	88.1
Hybrid (BM25 + DPR)	74.1	84.3	89.0

Exercises

Approximate Nearest Neighbour Search

Dense retrieval transforms the problem of finding relevant documents into the problem of finding nearby vectors. At training time, we learn to map semantically similar texts to nearby points in ${\mathrm{ℝ}}^d$. At inference time, we must find, among billions of document vectors, the $k$ vectors closest to the query vector. This is the nearest neighbour search problem, and in its exact form it is one of the oldest and most studied problems in computational geometry.

Unfortunately, exact nearest neighbour search in high dimensions is intractable at the scales required by modern retrieval. This is a consequence of the curse of dimensionality: in spaces with more than about 10–20 dimensions, all known exact search algorithms degrade to brute-force linear scan. The intuition is that in high dimensions, distances become increasingly uniform – the ratio between the nearest and farthest neighbour approaches 1 – and geometric partitioning structures (trees, grids, Voronoi cells) become ineffective because query balls intersect exponentially many cells.

For a corpus of $N={10}^9$ vectors in $d=768$ dimensions, a brute-force scan requires $N\cdot d$ floating-point multiplications per query – roughly $7.7\times {10}^{11}$ operations, or about 100 milliseconds on a modern GPU. For an interactive search engine handling thousands of queries per second, this is unacceptable.

More formally, the difficulty is captured by the following observation. Let ${𝒙}_1,\dots ,{𝒙}_N$ be $N$ points drawn uniformly from ${[\mathrm{0,1}]}^d$. For a query $𝒒$, let $\mathrm{NN}(𝒒)$ be the nearest neighbour and $\mathrm{FN}(𝒒)$ the farthest neighbour. Then for large $d$: (Curse OF DIM)$$\begin{array}{ccc}& \frac{𝔼[\mathrm{dist}(𝒒,\mathrm{FN}(𝒒))]}{𝔼[\mathrm{dist}(𝒒,\mathrm{NN}(𝒒))]}\to 1\text{as }d\to \infty .& \text{<span class="tml-eqn"></span>}\end{array}$$ When all distances are approximately equal, the very concept of “nearest neighbour” becomes ill-defined, and no partitioning scheme can prune the search space effectively.

The solution is to accept approximate answers. Approximate nearest neighbour (ANN) methods trade a small amount of accuracy (typically 95–99% of exact recall) for orders-of-magnitude speedups, reducing query latency from seconds to microseconds. The key insight is that for retrieval applications, we do not need the exact nearest neighbour – we need a set of candidates that is good enough to contain the relevant documents, which a subsequent reranking stage can then sort precisely.

Locality-Sensitive Hashing

Locality-sensitive hashing (LSH) was among the first theoretically grounded approaches to ANN search. The key idea is to design hash functions that map similar items to the same hash bucket with high probability and dissimilar items to different buckets.

For cosine similarity, the standard construction uses random hyperplane hashing. Each hash function is defined by a random vector $𝒓\sim 𝒩(0,{𝐈}_d)$: (LSH Hyperplane)$$\begin{array}{ccc}& h_{𝒓}(𝒙)=\{\begin{array}{cc}1& \text{if }{𝒓}^{\top }𝒙\ge 0,\\ 0& \text{otherwise}.\end{array}& \text{<span class="tml-eqn"></span>}\end{array}$$

A single random hyperplane provides only weak discrimination. If two vectors have angle $\theta =60°$, the collision probability is $1-60/180=2/3$ – better than chance, but not enough to build a reliable index. To amplify the gap between collision probabilities, we use AND-OR composition: concatenate $b$ hash functions to form a band (AND amplification), then use $L$ independent bands (OR amplification). Two items collide if they agree on all $b$ hashes within at least one band.

The AND step makes the collision probability drop rapidly for dissimilar items (since all $b$ hashes must agree), while the OR step ensures that similar items collide in at least one of the $L$ bands. The overall collision probability for items at angle $\theta$ is: (LSH Amplified)$$\begin{array}{ccc}& P(b,L,\theta )=1-{(1-{(1-\frac{\theta }{\pi })}^b)}^L.& \text{<span class="tml-eqn"></span>}\end{array}$$

Tree-Based Methods

KD-trees partition space by recursively splitting along coordinate axes. At each node, the dimension with the greatest spread is chosen, and the split is placed at the median value, yielding a balanced binary tree of depth $𝒪(\log N)$. For query processing, the tree is traversed by following the branch that contains the query, then backtracking to check sibling branches that could contain closer neighbours.

In low dimensions ($d\lesssim 20$), KD-trees achieve query times of $𝒪(\log N)$. In high dimensions, however, the backtracking becomes so frequent that performance degrades to $𝒪(N)$ – no better than brute force. Ball trees improve on KD-trees by using hyperspheres rather than axis-aligned splits, which better captures the geometry of clustered data, but they suffer from the same fundamental curse of dimensionality.

The failure of tree-based methods in high dimensions can be quantified precisely. For a KD-tree with $N$ points in ${\mathrm{ℝ}}^d$, the expected number of cells visited during a nearest-neighbour query is: (Kdtree Cells)$$\begin{array}{ccc}& 𝔼[\text{cells visited}]=\mathrm{\Omega }\left(N^{1-1/d}\right),& \text{<span class="tml-eqn"></span>}\end{array}$$ which for $d=768$ is essentially $\mathrm{\Omega }(N)$ – brute force.

Product Quantization

Product Quantization (PQ) [48] takes a fundamentally different approach: rather than building search structures, it compresses the vectors themselves so that distance computation becomes fast.

The idea is to decompose each $d$-dimensional vector into $M$ disjoint subvectors of dimension $d/M$, quantise each subvector independently using a codebook of $K$ centroids, and represent the entire vector by $M$ codebook indices. This reduces memory from $d\cdot 32$ bits (for float32) to $M\cdot \lceil {\log }_{2}K\rceil$ bits – a compression ratio of roughly $50\times$ to $380\times$ depending on $M$ (for $d=768$, $K=256$: from $48\times$ at $M=64$ to $384\times$ at $M=8$).

Codebook learning.

The codebooks ${𝒞}^1,\dots ,{𝒞}^M$ are learned by running $k$-means independently in each subspace. Specifically, the database vectors are split into subvectors, and $k$-means with $K$ clusters is run on each set of subvectors. The total training cost is $𝒪(M\cdot N\cdot K\cdot d/M)=𝒪(NKd)$, which is linear in the database size. The choice of $M$ and $K$ determines the trade-off between compression ratio and quantisation error. Typical values are $M=8$–$64$ and $K=256$ (so each index fits in one byte), giving PQ codes of 8–64 bytes per vector.

Asymmetric distance computation.

The key efficiency trick is asymmetric distance computation (ADC): we do not quantise the query. Instead, we precompute a lookup table of distances ${\Vert {𝒒}^m-c_k^m\Vert }^2$ for all $M$ subspaces and all $K$ centroids – a table of size $M\times K$. Then the distance to any database vector requires only $M$ table lookups and additions, regardless of $d$. The lookup table computation costs $𝒪(MKd/M)=𝒪(Kd)$, which is amortised over all $N$ database vectors. The total query cost is therefore $𝒪(Kd+NM)$, which for $K=256$, $d=768$, $M=64$ is roughly $2\times {10}^5+64N$ – dominated by the $64N$ term, which is $64/768\approx 8\%$ of the brute-force cost.

HNSW: Hierarchical Navigable Small World Graphs

Hierarchical Navigable Small World (HNSW) [49] is currently the most widely used ANN index, powering systems from Faiss to Pinecone to Weaviate. It combines two ideas: the navigable small world property of proximity graphs and a hierarchical structure inspired by skip lists.

Navigable small world graphs.

A navigable small world (NSW) graph connects each vector to a set of neighbours. Crucially, each node maintains both short-range links (to nearby vectors) and long-range links (to distant vectors). This combination enables greedy search – always moving to the neighbour closest to the query – to find near-optimal results in polylogarithmic time, analogous to the “six degrees of separation” phenomenon in social networks. The theoretical foundation is Kleinberg's result on navigable networks: if long-range links are distributed according to an inverse-power-law of distance, greedy routing finds near-optimal paths in $𝒪({\log }^{2}N)$ steps.

Hierarchical structure.

HNSW organises the NSW graph into $L$ layers, numbered $0$ (bottom, containing all vectors) through $L-1$ (top, containing very few vectors). Each vector is assigned a maximum layer $\ell \sim \lfloor -\ln (\mathrm{Uniform}(\mathrm{0,1}))\cdot m_L\rfloor$ where $m_L$ is a normalisation constant. This gives an exponentially decaying probability of appearing in higher layers – exactly like a skip list.

Search proceeds top-down: starting from the entry point at the top layer, greedy search finds the closest node in that layer, then descends to the next layer using that node as the starting point, and repeats. The upper layers provide long-range navigation, while the bottom layer provides fine-grained local search.

Construction.

Index construction inserts vectors one at a time. For each new vector $𝒙$, a random maximum layer ${\ell }_x$ is drawn from the geometric distribution. The algorithm then searches the current graph (top-down, as in the query algorithm) to find the $M$ closest neighbours of $𝒙$ at each layer $\ell \le {\ell }_x$, and creates bidirectional edges to those neighbours. If a neighbour's edge count exceeds a maximum $M_{\max }$, the weakest edges are pruned using a heuristic that favours diverse connections – neighbours are kept only if they are closer to $𝒙$ than to any already-selected neighbour. This diversity-preserving heuristic is crucial: without it, the graph degenerates into local clusters with poor long-range connectivity.

The construction cost per vector is $𝒪(M\cdot {\mathrm{𝑒𝑓}}_{\mathrm{construction}}\cdot \log N\cdot d)$, where ${\mathrm{𝑒𝑓}}_{\mathrm{construction}}$ is the candidate list size used during insertion (typically much larger than the search $\mathrm{𝑒𝑓}$). Total index construction is therefore $𝒪(NM\cdot {\mathrm{𝑒𝑓}}_{\mathrm{construction}}\cdot \log N\cdot d)$.

Complexity analysis.

The number of layers is $L=𝒪(\log N)$, and greedy search at each layer visits $𝒪(\log N)$ nodes (under mild distributional assumptions). The total search complexity is therefore $𝒪({\log }^{2}N)$ distance computations, each costing $𝒪(d)$. The $\mathrm{𝑒𝑓}$ parameter (candidate list size at the bottom layer) provides a tunable knob: larger $\mathrm{𝑒𝑓}$ improves recall at the cost of more distance computations, scaling the bottom-layer cost to $𝒪(\mathrm{𝑒𝑓}\cdot \log N)$.

Memory overhead.

Each vector stores $M$ edges per layer, and the expected number of layers per vector is $1/(1-1/M_L)\approx 1+1/\ln M$. The total memory overhead for edges is approximately $N\cdot M\cdot (1+1/\ln M)\cdot 4$ bytes (storing neighbour indices as 32-bit integers). For $N={10}^9$ and $M=32$, this is roughly 160 GB for edges alone – a significant cost that must be added to the storage of the vectors themselves.

IVF: Inverted File Index

The Inverted File Index (IVF) partitions the vector space into $C$ Voronoi cells using $k$-means clustering. Each database vector is assigned to its nearest centroid. At query time, only the vectors in the $n_{\mathrm{probe}}$ closest cells are searched:

1. Offline. Run $k$-means on the database vectors to obtain $C$ centroids $\{{𝝁}_1,\dots ,{𝝁}_C\}$. Assign each vector to its nearest centroid and store in the corresponding inverted list.

2. Online. Compute distances from the query $𝒒$ to all $C$ centroids. Select the $n_{\mathrm{probe}}$ nearest centroids. Exhaustively search the vectors in those $n_{\mathrm{probe}}$ inverted lists.

The expected number of distance computations per query is $n_{\mathrm{probe}}\cdot N/C$, which for typical values ($C=\sqrt{N}$, $n_{\mathrm{probe}}=16$) is roughly $16\sqrt{N}$ – a dramatic reduction from $N$.

Residual encoding.

A refinement of IVF stores not the original vectors but the residuals – the difference between each vector and its assigned centroid: $${𝒓}_i={𝒙}_i-{𝝁}_{c(i)},$$ where $c(i)$ is the cell assignment of ${𝒙}_i$. At query time, the query residual $𝒒-{𝝁}_c$ is computed for each probed cell $c$, and distances are measured in residual space. Because residuals have smaller variance than the original vectors (the centroid has “explained” the coarse location), they can be quantised more accurately with the same number of bits.

IVF-PQ.

The IVF index can be combined with Product Quantization: the inverted lists store PQ-encoded residual vectors instead of full float32 vectors. This combination – IVF-PQ – is the workhorse of billion-scale retrieval systems, offering both fast search (via cell pruning) and low memory (via vector compression). Faiss's implementation of IVF-PQ can index one billion 768-dimensional vectors in approximately 64 GB of RAM (with $M=64$ and $K=256$), compared to roughly 3 TB for storing the raw float32 vectors.

Multi-probe IVF.

The parameter $n_{\mathrm{probe}}$ creates a direct trade-off between recall and latency. With $n_{\mathrm{probe}}=1$, only the nearest cell is searched – fast but inaccurate if the query lies near a cell boundary. With $n_{\mathrm{probe}}=C$, all cells are searched – exact but slow. Empirically, $n_{\mathrm{probe}}\approx \sqrt{C}$ provides a good balance, achieving 90–95% recall at a fraction of the brute-force cost.

ScaNN: Anisotropic Quantization for MIPS

Most ANN methods are designed for Euclidean distance or cosine similarity, but retrieval models often optimise the inner product $⟨𝒒,𝒅⟩$, which does not correspond to a metric (it can be negative and does not satisfy the triangle inequality). ScaNN [50] introduces anisotropic vector quantization, which recognises that when optimising inner products, errors in the direction of the query matter more than errors in orthogonal directions.

Standard PQ minimises the isotropic reconstruction error ${\Vert 𝒙-\widehat{𝒙}\Vert }^2$. ScaNN instead minimises a weighted error that penalises the component parallel to the query direction more heavily: (Scann LOSS)$$\begin{array}{ccc}& {ℒ}_{\mathrm{ScaNN}}={𝔼}_{𝒒}\left[{(⟨𝒒,𝒙⟩-⟨𝒒,\widehat{𝒙}⟩)}^2\right]={𝔼}_{𝒒}\left[{⟨𝒒,𝒙-\widehat{𝒙}⟩}^2\right].& \text{<span class="tml-eqn"></span>}\end{array}$$ This is not equivalent to minimising ${\Vert 𝒙-\widehat{𝒙}\Vert }^2$ unless the query distribution is isotropic.

In more detail, ScaNN decomposes the quantisation error $𝒙-\widehat{𝒙}$ into a component parallel to $𝒙$ and a component perpendicular to $𝒙$: (Scann Decomp)$$\begin{array}{ccc}& 𝒙-\widehat{𝒙}=\underset{\text{parallel error}}{\underbrace{\frac{⟨𝒙-\widehat{𝒙},𝒙⟩}{{\Vert 𝒙\Vert }^2}𝒙}}+\underset{\text{perpendicular error}}{\underbrace{(𝒙-\widehat{𝒙}-\frac{⟨𝒙-\widehat{𝒙},𝒙⟩}{{\Vert 𝒙\Vert }^2}𝒙)}}.& \text{<span class="tml-eqn"></span>}\end{array}$$ The parallel error directly affects the inner product estimate, while the perpendicular error has zero expected effect (assuming isotropic query distribution). ScaNN assigns a weight $\eta >1$ to the parallel component during codebook optimisation, focusing the quantiser's limited capacity on the dimension that matters most for MIPS.

Recall-Latency Trade-Off

Method	Build	Query	Memory	Recall@10	Updatable
LSH	$𝒪(NLbd)$	$𝒪(NLb/2^b\cdot d)$	High	85–92%	Yes
PQ ($M=64$)	$𝒪(NMK)$	$𝒪(NM)$	64 B	70–85%	No
HNSW	$𝒪(N\log N\cdot d)$	$𝒪(\mathrm{𝑒𝑓}\log N\cdot d)$	3 KB + links	95–99%	Yes
IVF-PQ	$𝒪(NMK+NC)$	$𝒪(n_{p}NM/C)$	64 B	80–92%	Partial
ScaNN	$𝒪(NMK)$	$𝒪(n_{p}NM/C)$	64 B	85–95%	No

Exercises

Embedding Models and Training

The dense retrieval methods of Dense Retrieval Revolution depend entirely on the quality of the underlying embeddings. A dual-encoder is only as good as the vectors it produces – if semantically similar texts are not mapped to nearby points, no amount of ANN engineering will salvage retrieval quality. This section traces the evolution of embedding models from early Siamese networks to modern universal embedding systems, and examines the training techniques that make them effective.

Sentence-BERT: The Foundation

Before Sentence-BERT [51], using BERT for sentence similarity required passing every pair through a cross-encoder – $𝒪(N^2)$ forward passes for $N$ sentences. Sentence-BERT (SBERT) introduced the idea of fine-tuning BERT in a Siamese architecture to produce fixed-size sentence embeddings that could be compared with cosine similarity.

Architecture.

SBERT passes each sentence independently through a shared BERT encoder, applies a pooling operation (typically mean pooling over token embeddings) to obtain a sentence vector, and optimises a contrastive or regression objective: (Sbert Embedding)$$\begin{array}{ccc}& {𝒆}_s=\mathrm{MeanPool}(\mathrm{BERT}(s))=\frac{1}{n}\sum _{i=1}^n{𝒉}_i,& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${𝒉}_i$ is the contextual embedding of the $i$-th token and $n$ is the sequence length. For natural language inference (NLI) training, SBERT uses a softmax classifier over the concatenation $[{𝒆}_u;{𝒆}_v;|{𝒆}_u-{𝒆}_v|]$, while for semantic textual similarity (STS) it uses cosine similarity with a regression objective.

Training objectives.

SBERT explores several training configurations depending on the available supervision:

• Classification objective (NLI). Given sentence pairs labelled as entailment, contradiction, or neutral, the model concatenates $[{𝒆}_u;{𝒆}_v;|{𝒆}_u-{𝒆}_v|]\in {\mathrm{ℝ}}^{3d}$ and passes it through a softmax classifier. This trains the embeddings to be informative about semantic relationships.

• Regression objective (STS). Given sentence pairs with continuous similarity scores $y\in [0,5]$, the model minimises the mean squared error between $\cos ({𝒆}_u,{𝒆}_v)$ and $y/5$.

• Contrastive objective (retrieval). Given query–positive pairs, the model uses the multiple negatives ranking loss with in-batch negatives.

The key finding of the original SBERT paper was that NLI pre-training followed by STS fine-tuning produced the best general-purpose embeddings – a recipe that influenced all subsequent work.

Instruction-Conditioned Embeddings

A limitation of early embedding models is that they produce a single embedding per text, regardless of the downstream task. But the “meaning” of a sentence depends on what you intend to do with it. For clustering, you want to capture the topic. For retrieval, you want to capture the information need. For classification, you want to capture the label-relevant features.

Instructor [52] and E5 [53] address this by prepending a task instruction to the input text: (Instructor)$$\begin{array}{ccc}& 𝒆=f_{𝜽}(\text{“Represent the query for retrieving documents: ”}\oplus ︎s),& \text{<span class="tml-eqn"></span>}\end{array}$$ where $\oplus ︎$ denotes string concatenation. Different instructions steer the encoder towards different aspects of the text's meaning, enabling a single model to serve multiple tasks.

E5 (EmbEddings from bidirEctional Encoder rEpresentations) further refined this approach by training on massive curated datasets of text pairs with task-specific prefixes (“query:” and “passage:”), achieving strong zero-shot performance across diverse benchmarks.

The importance of instructions can be understood geometrically. Without instructions, the model must map a sentence to a single point in ${\mathrm{ℝ}}^d$ that is simultaneously close to all “similar” sentences – but similarity is task-dependent. For clustering, “The stock market rose today” should be near other finance sentences; for sentiment analysis, it should be near other positive sentences. Instruction conditioning effectively gives the model access to a different projection for each task, expanding the effective capacity of the embedding space without increasing dimensionality.

Modern Open Embedding Models

The period from 2023 onward saw rapid progress in open-source embedding models, driven by improved training data, multi-stage training pipelines, and architectural innovations.

GTE (General Text Embeddings).

Developed by Alibaba, GTE uses a multi-stage training pipeline: (1) unsupervised pre-training with contrastive learning on web-scale text pairs, (2) supervised fine-tuning on curated retrieval datasets, and (3) task-specific adaptation. GTE models range from 100M to 1.5B parameters and achieve state-of-the-art results on the MTEB benchmark.

BGE (BAAI General Embedding).

Developed by the Beijing Academy of Artificial Intelligence, BGE similarly employs multi-stage training with an emphasis on hard-negative mining across stages. A notable innovation is RetroMAE pre-training, which uses masked autoencoding with an asymmetric architecture to learn better representations before contrastive fine-tuning.

Training at scale.

The key insight behind both GTE and BGE is that data quality and quantity matter more than architectural innovation for embeddings. Both groups invested heavily in curating large-scale training datasets:

• Web-mined pairs. Title-body pairs from web pages, question-answer pairs from forums, and anchor-text-to-page mappings provide hundreds of millions of weakly supervised pairs.

• Synthetic data. Large language models generate query-passage pairs for domains where natural pairs are scarce (e.g., scientific and legal texts).

• Cross-lingual alignment. Parallel translations and bilingual dictionaries enable multilingual embedding models that map semantically equivalent texts in different languages to nearby vectors.

Matryoshka Representation Learning

A practical challenge with embedding models is that different applications have different computational budgets. A mobile application might afford only 64-dimensional embeddings, while a server-side system can handle 1024 dimensions. Traditionally, this required training and maintaining separate models for each dimensionality.

Matryoshka Representation Learning (MRL) [54] trains a single model whose embeddings are useful at any prefix dimensionality. Named after the Russian nesting dolls, MRL ensures that the first $d^{\prime }$ dimensions of a $d$-dimensional embedding form a good $d^{\prime }$-dimensional embedding.

The training loss is a weighted sum over multiple truncation levels: (Matryoshka LOSS)$$\begin{array}{ccc}& {ℒ}_{\mathrm{MRL}}=\sum _{d^{\prime }\in 𝒟}{w}_{d^{\prime }}\cdot ℒ\left({\left\{{𝒆}_s^{1:d^{\prime }}\right\}}_s\right),& \text{<span class="tml-eqn"></span>}\end{array}$$ where $𝒟=\{8,16,32,64,128,256,512,768\}$ is a set of target dimensionalities, ${𝒆}_s^{1:d^{\prime }}$ denotes the first $d^{\prime }$ dimensions of the embedding, $w_{d^{\prime }}$ are weights (often uniform), and $ℒ$ is the standard contrastive loss applied at each scale.

The key property that makes MRL work is that the information content of the embedding is front-loaded: the first few dimensions capture the coarsest, most important distinctions (broad topic, language, domain), while later dimensions encode progressively finer details (specific entities, subtle sentiment, stylistic nuances). Without the Matryoshka loss, a standard contrastive training procedure distributes information more uniformly across dimensions, so truncation causes a sharp quality drop.

Long-Context Embeddings

Standard BERT-based embeddings are limited to 512 tokens – roughly a paragraph. Many retrieval applications require embedding entire documents, legal filings, or codebases that span thousands of tokens.

Nomic Embed.

Nomic Embed extends the context window to 8,192 tokens using ALiBi (Attention with Linear Biases) positional encoding, which replaces learned absolute position embeddings with a linear bias term that decays with token distance. This allows the model to generalise to sequence lengths not seen during training.

jina-embeddings.

The jina-embeddings family uses a modified BERT architecture with RoPE (Rotary Position Embeddings) to support context lengths up to 8,192 tokens. jina-embeddings-v3 further introduces Late Chunking: rather than embedding each chunk independently, it processes the full document through the transformer and then extracts chunk embeddings from the contextual representations, preserving cross-chunk coherence.

The chunking dilemma.

Long-context embeddings address a fundamental tension in document retrieval. Short chunks (128–256 tokens) are precise – each chunk covers a single topic – but they lose context. A chunk saying “He won the award in 1995” is useless without knowing who “he” is. Long chunks (2048+ tokens) preserve context but dilute the embedding: a 2000-token passage about multiple topics produces a vector that is an imprecise average of several distinct meanings.

The ideal solution would embed each passage with full awareness of its surrounding context – which is exactly what Late Chunking achieves, and what long-context models make feasible by processing entire documents through the transformer before extracting per-chunk embeddings from the contextualised representations.

Gemini Embedding: Universal Multimodal Embeddings

The trajectory from single-task text embeddings to multi-task instruction-conditioned embeddings finds its logical continuation in universal multimodal embedding models that map text, images, code, and other modalities into a shared vector space.

Gemini Embedding is Google's entry in this space. Built on the Gemini multimodal architecture, it produces embeddings that can be used for text retrieval, code search, image-text matching, and cross-lingual transfer within a single model. The architecture leverages Gemini's native multimodal understanding to produce embeddings without modality-specific encoder heads. Key design choices include:

• Instruction-conditioned. Like Instructor and E5, Gemini Embedding accepts task instructions, but extends this to multimodal queries (e.g., “Find images similar to this description”).

• Shared latent space. All modalities are projected into the same $d$-dimensional space, enabling zero-shot cross-modal retrieval.

• Matryoshka-compatible. The model supports flexible dimensionality via MRL-style training.

The move towards universal multimodal embeddings raises fundamental questions about the geometry of the shared space. When text, images, and code are all mapped to ${\mathrm{ℝ}}^d$, the model must learn to organise semantically related items of different modalities nearby while keeping unrelated items of the same modality apart. This creates a tension: the model must simultaneously optimise intra-modal discrimination and cross-modal alignment. Early multimodal embedding models like CLIP resolved this by training exclusively on cross-modal pairs (image-text), but this left intra-modal performance suboptimal. Modern models like Gemini Embedding train on a mixture of within-modal and cross-modal contrastive objectives, with careful balancing of the loss weights.

The Embedding Training Pipeline

State-of-the-art embedding models are not trained in a single pass. They follow a multi-stage pipeline that progressively refines representations:

Stage 1: Pre-training.

The base language model (typically a BERT variant or decoder model) is pre-trained on a large unlabelled corpus with masked language modelling (MLM) or causal language modelling (CLM). This stage provides general linguistic knowledge but does not produce good sentence-level representations.

Stage 2: Contrastive pre-training.

The model is further pre-trained with a contrastive objective on weakly supervised text pairs – typically title-body pairs, question-answer pairs, and parallel translations harvested from the web. This stage – sometimes called unsupervised contrastive learning or weakly-supervised pre-training – teaches the model to recognise semantic similarity at the sentence level. The scale of data at this stage is massive: GTE uses over 800 million text pairs, and E5 uses a curated dataset of over 300 million pairs spanning 93 languages.

Stage 3: Supervised fine-tuning.

The model is fine-tuned on curated, high-quality labelled datasets with hard negatives. This is where most of the retrieval-specific learning happens. The loss is typically InfoNCE with in-batch negatives and mined hard negatives.

Stage 4: Distillation (optional).

A smaller student model is trained to mimic the similarity judgements of a larger teacher model (often a cross-encoder). The distillation loss is typically a KL divergence between the student's and teacher's softmax distributions over candidates: (Embedding Distill)$$\begin{array}{ccc}& {ℒ}_{\mathrm{distill}}=D_{\mathrm{KL}}\left(\sigma ({𝒔}^{\mathrm{teacher}}/{\tau }_d)\Vert \sigma ({𝒔}^{\mathrm{student}}/{\tau }_d)\right),& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${𝒔}^{\mathrm{teacher}}$ and ${𝒔}^{\mathrm{student}}$ are the vectors of scores assigned by teacher and student to a set of candidates, $\sigma$ is the softmax function, and ${\tau }_d$ is a distillation temperature. This allows production deployment of efficient models (100M–300M parameters) that approach the quality of expensive cross-encoders (which may have billions of parameters but cannot precompute embeddings).

The distillation pipeline typically works as follows: (1) retrieve the top 100 candidates for each training query using a first-stage retriever; (2) score all 100 candidates with the cross-encoder teacher; (3) train the student embedding model to reproduce the teacher's ranking, using the KL divergence loss in (Embedding Distill). The student never sees the teacher's internal representations – only its output scores. This score distillation approach is simpler and more effective than feature-level distillation for embedding tasks, because what matters for retrieval is the relative ordering of candidates, not the internal features used to produce that ordering.

Training Objectives: Contrastive vs. Generative

The dominant training paradigm for embeddings is contrastive learning, but an alternative generative approach has gained traction.

Contrastive objectives.

The standard approach uses InfoNCE ((Infonce)) with temperature scaling. Variants include:

• Triplet loss: ${ℒ}_{\mathrm{triplet}}=\max (0,\mathrm{dist}(𝒒,{𝒑}^{+})-\mathrm{dist}(𝒒,{𝒑}^{-})+m)$ where $m>0$ is a margin.

• Multiple negatives ranking loss: The in-batch softmax cross-entropy, which is equivalent to InfoNCE with $\tau =1$.

• Cosine similarity loss: Direct regression of cosine similarity to human-annotated similarity scores.

Generative objectives.

Recent work has shown that decoder-based language models can produce strong embeddings by encoding the text as a prompt and extracting embeddings from the last token representation. The key insight is that the last token in a causal model has attended to all preceding tokens, making its representation a natural summary of the input.

Several approaches have emerged for extracting embeddings from LLMs:

• LLM2Vec: Converts any decoder LLM into a bidirectional encoder by enabling bidirectional attention (removing the causal mask) and training with a contrastive objective. This preserves the LLM's world knowledge while enabling symmetric encoding.

• E5-Mistral: Fine-tunes Mistral-7B with instruction-following and contrastive learning, using the last token embedding. Achieves state-of-the-art MTEB scores while operating with the original causal attention.

• Gecko: Google's approach of distilling knowledge from a large LLM into a compact embedding model by using the LLM to generate diverse query-passage training pairs with fine-grained relevance labels.

The advantage of LLM-based embeddings is leveraging the massive scale and world knowledge of pre-trained models (7B+ parameters) for embedding tasks. The disadvantage is computational: encoding a single text with a 7B-parameter model is roughly 20$\times$ slower than with a 335M-parameter BERT model, which can be prohibitive for applications that require real-time indexing of new documents.

Hard Negative Mining Strategies

The quality of negative examples is arguably the most important factor in embedding training. We describe three strategies in order of increasing effectiveness.

BM25 negatives.

The simplest hard negatives are passages retrieved by BM25 for the query that are not in the gold positive set. These negatives share lexical overlap with the query, forcing the model to learn that surface similarity does not imply relevance. This is the strategy used by DPR.

Cross-encoder negatives.

A stronger approach uses a cross-encoder reranker to score a large candidate set retrieved by a first-stage model. Passages that the cross-encoder scores highly but that are not in the gold set become hard negatives. These negatives require the embedding model to match the fine-grained discriminative ability of the cross-encoder.

Self-negatives.

In iterative training, the embedding model itself is used to retrieve negatives. After each training epoch, the model re-indexes the corpus, retrieves the hardest non-positive passages for each query, and uses them as negatives for the next epoch. This creates an adversarial curriculum that progressively challenges the model. The process is analogous to self-play in reinforcement learning: the model generates its own training signal by identifying its current failure modes.

The practical challenge with self-negatives is computational: re-indexing a large corpus after each epoch requires encoding all documents with the updated model. A common approximation is to re-index every $k$ epochs or to use an exponential moving average of the model weights for indexing, which reduces but does not eliminate the staleness of the negative set. The ANCE (Approximate Nearest Neighbour Negative Contrastive Learning) method formalised this approach, showing significant gains over static negative mining on multiple benchmarks.

Embedding Model Comparison

Model	Params	Dims	Context	MTEB Avg	Key Feature
SBERT (2019)	110M	768	512	56.3	Siamese BERT
E5-base (2022)	110M	768	512	59.9	Task prefixes
BGE-large (2023)	335M	1024	512	64.2	RetroMAE pre-training
GTE-large (2023)	335M	1024	512	65.4	Multi-stage pipeline
Nomic-embed (2024)	137M	768	8192	62.4	Long context (ALiBi)
jina-v3 (2024)	570M	1024	8192	66.1	Late Chunking
E5-mistral-7B (2024)	7.1B	4096	32768	66.6	LLM-based
Gemini Embed (2025)	-	3072	8192	67.5	Multimodal universal

The trend is clear: models have grown in parameter count, embedding dimensionality, context length, and benchmark performance. But perhaps the most important trend is the convergence towards universal models that handle multiple tasks, languages, and modalities within a single architecture.

Several observations emerge from this comparison. First, the jump from SBERT to E5 (3.6 points) came primarily from better training data, not architectural changes – both use similar BERT-based architectures. Second, the LLM-based models (E5-mistral-7B) achieve the highest scores but at 20$\times$ the computational cost, raising the question of whether the quality gain justifies the expense for latency-sensitive applications. Third, long-context models (Nomic, jina) sacrifice a few MTEB points for dramatically expanded context windows – a trade-off that is worthwhile for document-level retrieval but unnecessary for short-query tasks.

Exercises

Classical Vision Retrieval

Imagine you are sitting in a small café in Lisbon. On the wall hangs a painting: a turbulent sky rendered in thick impasto strokes, deep blues swirling above a sleeping village. You do not know the artist or the title, but you want to find it online. You could describe it in words, but language is an impoverished medium for capturing visual texture, colour palette, and compositional geometry. What you really want is to take a photograph and ask a system: find me this painting. This is the problem of image retrieval: given a query image, find the most visually similar images in a large database.

The story of how computer vision solved this problem is one of the most elegant intellectual arcs in the field. It begins with a deceptively simple insight: images, like documents, can be represented as collections of discrete visual features. This insight connects the rich mathematical machinery of text retrieval, which we developed in the preceding sections, to the entirely different modality of visual perception. The bridge between these two worlds was built by local feature descriptors, most notably the Scale-Invariant Feature Transform.

SIFT: Scale-Invariant Feature Transform

The fundamental challenge of image matching is that the same object looks different under different viewing conditions. A photograph of a cathedral taken from ten metres away at noon and another taken from thirty metres away at sunset depict the same structure, yet nearly every pixel value differs. The genius of David Lowe's Scale-Invariant Feature Transform (SIFT) [55] is to extract local image features that remain stable under changes in scale, rotation, and illumination, precisely the transformations that make naïve pixel-level comparison fail.

The SIFT algorithm proceeds in four stages, each building carefully on the last. We describe each in turn.

Stage 1: Scale-space extrema detection.

The first stage identifies candidate keypoint locations that are stable across scales. SIFT constructs a scale space by convolving the image $I(x,y)$ with Gaussian kernels of increasing width: (Scale Space)$$\begin{array}{ccc}& L(x,y,\sigma )=G(x,y,\sigma )\ast I(x,y),G(x,y,\sigma )=\frac{1}{2\pi {\sigma }^2}\exp (-\frac{x^2+y^2}{2{\sigma }^2}).& \text{<span class="tml-eqn"></span>}\end{array}$$ Keypoint candidates are found at extrema of the Difference of Gaussians (DoG): (DOG)$$\begin{array}{ccc}& D(x,y,\sigma )=L(x,y,k\sigma )-L(x,y,\sigma ),& \text{<span class="tml-eqn"></span>}\end{array}$$ where $k$ is a constant multiplicative factor separating adjacent scales. A point $(x,y,\sigma )$ is a candidate keypoint if $D(x,y,\sigma )$ is a local extremum in both spatial coordinates and scale, compared to its $3\times 3\times 3$ neighbourhood of 26 adjacent samples.

Stage 2: Keypoint localisation.

Each candidate is refined by fitting a three-dimensional quadratic to the DoG function via a Taylor expansion: (Keypoint Refine)$$\begin{array}{ccc}& D(𝒙)\approx D({𝒙}_0)+{\frac{\partial D}{\partial 𝒙}}^{\top }(𝒙-{𝒙}_0)+\frac{1}{2}{(𝒙-{𝒙}_0)}^{\top }\frac{{\partial }^{2}D}{\partial {𝒙}^2}(𝒙-{𝒙}_0),& \text{<span class="tml-eqn"></span>}\end{array}$$ where $𝒙={(x,y,\sigma )}^{\top }$. The extremum is located at $\widehat{𝒙}={𝒙}_0-(\frac{{\partial }^{2}D}{\partial {𝒙}^2}{)}^{-1}\frac{\partial D}{\partial 𝒙}$, and candidates with low contrast $|D(\widehat{𝒙})|<0.03$ or high edge response (ratio of principal curvatures above a threshold) are discarded.

Stage 3: Orientation assignment.

Each surviving keypoint is assigned one or more dominant orientations based on a histogram of gradient orientations in a neighbourhood weighted by the keypoint's scale. Let $m(x,y)=\sqrt{L_x^2+L_y^2}$ and $\theta (x,y)=\arctan (L_y/L_x)$ be the gradient magnitude and orientation. A 36-bin orientation histogram is computed, and any peak within 80% of the maximum defines a keypoint orientation. This step ensures rotational invariance: all subsequent computations are performed relative to the assigned orientation.

Stage 4: Descriptor construction.

The descriptor is computed in a $16\times 16$ patch around the keypoint, rotated to align with the assigned orientation. This patch is divided into a $4\times 4$ grid of sub-regions, and within each sub-region an 8-bin orientation histogram is computed, weighted by gradient magnitude and a Gaussian centred on the keypoint. The result is a $4\times 4\times 8=128$-dimensional vector $𝒅\in {\mathrm{ℝ}}^{128}$, which is normalised to unit length to achieve illumination invariance: (SIFT Descriptor)$$\begin{array}{ccc}& {𝒅}_{\text{SIFT}}=\frac{𝒅}{{\Vert 𝒅\Vert }_2+ϵ}.& \text{<span class="tml-eqn"></span>}\end{array}$$

Bag of Visual Words

SIFT gives us a powerful way to describe individual keypoints, but an image contains thousands of them. How do we compare two images when each is represented by a variable-size set of 128-dimensional vectors? The answer came from an analogy so natural it seems inevitable in hindsight: treat images exactly like text documents.

In text retrieval, a document is represented by the frequencies of words it contains, discarding word order. The bag of visual words model, introduced by Csurka et al. [56], applies the same idea to images. Local feature descriptors play the role of words, and a visual vocabulary is learned by clustering the descriptor space.

With a visual vocabulary in hand, the Bag of Visual Words (BoVW) representation of an image is constructed in exact analogy with the bag-of-words model for text:

1. Extract local features $\{{𝒅}_1,\dots ,{𝒅}_n\}$ from the image.

2. Quantise each descriptor: $w_i=q({𝒅}_i)$ for $i=1,\dots ,n$.

3. Construct a histogram $𝒉\in {\mathrm{ℝ}}^K$ where $h_k=|\{i:w_i=k\}|$.

4. Apply TF-IDF weighting, exactly as in text retrieval: (Visual Tfidf)$$\begin{array}{ccc}& {\tilde{h}}_k=\frac{h_k}{n}\cdot \log \frac{N}{|\{j:h_k^{(j)}>0\}|},& \text{<span class="tml-eqn"></span>}\end{array}$$ where $N$ is the number of images in the database and the denominator counts images containing visual word $k$.

5. Normalise: $\widehat{𝒉}=\widetilde{𝒉}/{\Vert \widetilde{𝒉}\Vert }_2$.

Image similarity is measured by the cosine similarity between normalised BoVW histograms, and an inverted index enables sub-linear retrieval over millions of images.

Video Google: Text Retrieval Meets Vision

The full power of the visual-words analogy was demonstrated by Sivic and Zisserman [57] in their landmark Video Google system. Their insight was breathtakingly direct: if images are documents and local features are words, then the entire arsenal of text retrieval, including inverted files, TF-IDF weighting, stop-word removal, and query expansion, transfers directly to visual search.

The Video Google pipeline introduced several innovations that became standard in the field:

• Visual stop words: visual words that appear in nearly every image (e.g. uniform gradient regions) carry little discriminative information and are removed, exactly as “the” and “a” are removed in text retrieval.

• Spatial verification: after an initial ranking based on BoVW similarity, candidate matches are verified by estimating an affine transformation between query and candidate keypoint locations using RANSAC. Matches that are not geometrically consistent are discarded.

• Query expansion: once a set of verified matches is found, their features are added to the query and retrieval is repeated, improving recall.

VLAD: Vector of Locally Aggregated Descriptors

The Bag of Visual Words model, for all its elegance, discards information. When a descriptor $𝒅$ is assigned to visual word ${𝒘}_k$, we record only which cluster it belongs to, not where within the cluster it lies. The residual $𝒅-{𝒘}_k$ encodes precisely this lost information. Jégou et al. [58] proposed aggregating these residuals into a single, compact vector called the Vector of Locally Aggregated Descriptors (VLAD).

The key insight is that VLAD captures first-order statistics of the residual distribution within each cluster, whereas BoVW captures only zeroth-order statistics (counts). With $K=64$ visual words and $d=128$ (SIFT), VLAD produces a $64\times 128=8192$-dimensional vector, which can be compressed to 128 bytes via PCA and product quantisation while retaining excellent retrieval accuracy.

Fisher Vectors: The Probabilistic Generalisation

If VLAD captures first-order residual statistics, a natural question arises: can we capture higher-order statistics as well? The Fisher vector provides an affirmative answer by casting the aggregation problem in a probabilistic framework.

Deep Learning for Image Retrieval

The classical pipeline of keypoint detection, handcrafted description, and aggregation served the vision community well for over a decade. But it was always a pipeline, with each stage designed independently, and every design choice (Gaussian scale space, gradient histograms, $k$-means quantisation) reflected a human engineer's intuition about what makes a good visual representation. Deep learning replaces this multi-stage pipeline with a single, end-to-end learned mapping from pixels to a compact embedding vector, and in doing so achieves representations that are not merely incrementally better but qualitatively different in their semantic richness.

CNN Features for Image Retrieval

The earliest approach to deep image retrieval was disarmingly simple: take a CNN pre-trained on ImageNet classification, remove the final classification layer, and use the penultimate layer's activations as an image descriptor. A $224\times 224$ image passed through a ResNet-101 produces a 2048-dimensional vector that captures semantic content far richer than any SIFT-based representation.

Generalised Mean (GeM) pooling.

The standard approach pools the final convolutional feature map $𝐗\in {\mathrm{ℝ}}^{C\times H\times W}$ into a single vector $𝒇\in {\mathrm{ℝ}}^C$. Global average pooling (GAP) computes the spatial mean of each channel, but this treats all spatial locations equally. Generalised Mean (GeM) pooling [38] interpolates between average and max pooling via a learnable parameter $p\ge 1$: (GEM Pooling)$$\begin{array}{ccc}& f_c={\left(\frac{1}{HW}\sum _{h=1}^H\sum _{w=1}^W{X}_{c,h,w}^p\right)}^{1/p},c=1,\dots ,C.& \text{<span class="tml-eqn"></span>}\end{array}$$ When $p=1$, GeM reduces to average pooling; as $p\to \infty$, it approaches max pooling. Learning $p$ on a retrieval task typically yields $p\approx 3$, emphasising the most activated spatial regions without the noise sensitivity of pure max pooling.

Whitening and dimensionality reduction.

After pooling, the raw feature vector $𝒇$ is whitened by a learned linear transformation ${𝒇}^{\prime }=𝐖(𝒇-𝝁)$ where $𝐖$ is obtained from PCA on a set of training features. This decorrelates the feature dimensions, suppresses co-occurring patterns (which tend to be uninformative for discrimination), and reduces the dimension from 2048 to typically 256 or 512, enabling efficient large-scale search.

CLIP: Contrastive Language-Image Pre-training

The methods described so far learn visual representations from images alone, using class labels or visual similarity as supervision. In 2021, Radford et al. [9] introduced a paradigm shift: learning visual concepts not from a fixed taxonomy of labels but from the free-form natural language that humans already use to describe images. The resulting model, Contrastive Language-Image Pre-training (CLIP), demonstrated that a simple contrastive objective applied to 400 million image-text pairs produces visual representations that transfer to a remarkable range of downstream tasks without any task-specific fine-tuning.

Architecture: dual encoders.

CLIP consists of two encoders trained jointly:

• An image encoder $f_{𝜽}:ℐ\to {\mathrm{ℝ}}^d$ that maps images to $d$-dimensional embeddings. The original CLIP uses either a ResNet or a Vision Transformer (ViT). The ViT variant divides an image into $16\times 16$ patches, linearly projects each patch, prepends a [CLS] token, and processes the sequence through transformer layers. The [CLS] token's final representation, after a learned linear projection, serves as the image embedding.

• A text encoder $g_{𝝓}:𝒯\to {\mathrm{ℝ}}^d$ that maps text sequences to the same $d$-dimensional space. CLIP uses a 12-layer transformer with a modified tokeniser. The representation of the [EOS] token, after a linear projection, serves as the text embedding.

Both encoders project into the same $d$-dimensional space ($d=512$ or $d=768$), enabling direct comparison of images and texts via inner products.

Training: InfoNCE with temperature.

Given a mini-batch of $N$ image-text pairs ${\{(I_i,T_i)\}}_{i=1}^N$, CLIP computes normalised embeddings: (CLIP Normalise)$$\begin{array}{ccc}& {𝒊}_i=\frac{f_{𝜽}(I_i)}{{\Vert f_{𝜽}(I_i)\Vert }_2},{𝒕}_j=\frac{g_{𝝓}(T_j)}{{\Vert g_{𝝓}(T_j)\Vert }_2}.& \text{<span class="tml-eqn"></span>}\end{array}$$ The similarity between image $i$ and text $j$ is the scaled cosine similarity $s_{ij}=⟨{𝒊}_i,{𝒕}_j⟩/\tau$, where $\tau >0$ is a learned temperature parameter. The CLIP loss is a symmetric InfoNCE objective:

Zero-shot transfer: why it works.

CLIP's most striking capability is zero-shot image classification. Given a set of class names $\{c_1,\dots ,c_M\}$, one constructs text prompts such as “a photo of a $\{c_k\}$” and encodes them with the text encoder. The predicted class for a query image $I$ is: (CLIP Zeroshot)$$\begin{array}{ccc}& \hat{y}={\mathrm{arg\hspace{0.17em}max}}_{k\in \{1,\dots ,M\}}⟨f_{𝜽}(I),g_{𝝓}(\text{“a photo of a }{c}_k\text{”})⟩.& \text{<span class="tml-eqn"></span>}\end{array}$$ This works because CLIP's training objective forces the model to learn a shared semantic space where images and their natural language descriptions are aligned. The 400 million training pairs effectively provide supervision for millions of visual concepts, far exceeding any manually curated label set.

Scaling and Simplifying: ALIGN and SigLIP

CLIP demonstrated the power of contrastive vision-language learning, but its 400 million training pairs were carefully curated (filtered from a larger web crawl). Two subsequent works explored opposite ends of a fundamental question: what matters more, data quality or data quantity?

ALIGN: scaling with noisy data.

Jia et al. [19] introduced ALIGN (A Large-scale ImaGe and Noisy-text embedding), trained on 1.8 billion image-text pairs harvested from alt-text on the web with only minimal frequency-based filtering. The key finding was striking: simply scaling the data by $4.5\times$, even with substantially noisier supervision, matched or exceeded CLIP's performance. ALIGN used an EfficientNet image encoder and a BERT text encoder, both larger than CLIP's, and trained with the same contrastive loss.

SigLIP: sigmoid pairwise loss.

Zhai et al. [20] identified a subtle but important limitation of the softmax-based InfoNCE loss used by CLIP and ALIGN: it requires all-to-all communication within a batch. The denominator ${\sum }_j\exp (s_{ij})$ in (CLIP LOSS) couples every sample to every other sample in the batch, making distributed training across multiple devices difficult.

SigLIP replaces the softmax with independent sigmoid losses for each pair:

BLIP and BLIP-2: Bootstrapping Captions for Better Retrieval

While ALIGN showed that noisy data can work, the noise still hurts: many alt-text descriptions on the web are irrelevant, truncated, or misleading. Li et al. [21] proposed BLIP (Bootstrapping Language-Image Pre-training), which addresses this problem by generating and filtering captions as part of the training loop.

BLIP: captioning as denoising.

BLIP trains three components simultaneously: (1) a contrastive image-text encoder (like CLIP), (2) an image-grounded text decoder that generates captions, and (3) a filter network that scores image-text pairs for relevance. After an initial round of training, the decoder generates synthetic captions for all web images, and the filter removes noisy web captions, replacing them with the higher-quality synthetic ones. This bootstrapping procedure produces a cleaner training set that improves both retrieval and captioning performance.

The elegance of BLIP lies in this self-improving loop: the model generates better captions, which produce a cleaner dataset, which trains a better model, which generates even better captions. In practice, a single round of bootstrapping suffices to achieve significant improvements.

BLIP-2: bridging frozen models with Q-Former.

Li et al. [59] took the bootstrapping idea further with BLIP-2, which addresses a practical challenge: the best image encoders and the best language models are both extremely expensive to train, so how can we combine them without retraining either? The answer is the Querying Transformer (Q-Former), a lightweight transformer that bridges a frozen image encoder and a frozen large language model.

Diffusion Features for Retrieval

The models discussed so far were explicitly trained for representation learning, whether through contrastive objectives or caption generation. A surprising recent discovery is that diffusion models, trained solely to generate images by iteratively denoising Gaussian noise, learn intermediate representations that are excellent features for retrieval and correspondence tasks. This finding, introduced by Tang et al. [60] as Diffusion Features (DIFT), challenges the assumption that good representations require discriminative or contrastive training.

Why diffusion models learn good features.

The denoising process at the heart of diffusion models forces the network to understand visual structure at multiple levels of abstraction. At early denoising timesteps (high noise), the model must understand global scene layout and object categories. At late timesteps (low noise), the model must understand fine-grained texture and local geometry. This multi-scale understanding is precisely what a good retrieval feature needs.

PDM: personalised retrieval with diffusion features.

Samuel et al. [61] extended diffusion features to the problem of personalised image retrieval: given a few reference images of a specific object (e.g. “my dog”), retrieve all images of that object from a large gallery. Their Personalised Diffusion Model (PDM) approach fine-tunes a diffusion model on the reference images using textual inversion, then extracts DIFT features that are now attuned to the specific object's visual characteristics. The personalised features significantly outperform generic CLIP embeddings on fine-grained instance retrieval tasks.

Sketch-based retrieval with diffusion priors.

Koley et al. [62] demonstrated that diffusion models can bridge the domain gap between sketches and photographs, a notoriously difficult cross-modal retrieval task. The key insight is that the diffusion model's latent space provides a shared representation where sketches and photos of the same object are naturally aligned, because the model has learned that both are valid renderings of the same underlying visual concept.

Benchmarks and Model Comparison

We now compare the models discussed in this section on standard image-text retrieval benchmarks.

	MSCOCO		Flickr30k
(lr)2-3 (lr)4-5 Model	I$\to$T	T$\to$I	I$\to$T	T$\to$I
BoVW + TF-IDF	8.2	5.4	12.1	7.8
Fisher Vector	14.7	10.2	21.3	14.6
CNN (ResNet-101)	42.3	31.5	61.7	44.2
CLIP (ViT-B/32)	58.4	37.8	88.0	68.7
CLIP (ViT-L/14)	64.2	43.6	91.3	75.1
ALIGN	64.8	45.2	91.8	76.8
SigLIP (ViT-L)	65.7	44.9	92.1	77.2
BLIP	65.1	44.7	94.2	82.4
BLIP-2 (ViT-g)	69.7	50.1	96.6	87.3

Video Retrieval

Images are frozen moments; video is the flowing river of visual experience. The transition from image retrieval to video retrieval introduces a dimension that fundamentally changes the nature of the problem: time. A video is not merely a bag of frames but a structured temporal sequence in which actions unfold, objects interact, and narratives develop. Finding the right video in a large corpus is hard enough; finding the right moment within the right video is harder still.

This section develops the theory and practice of video retrieval, from adapting image-text models to the temporal domain through the challenging problems of moment retrieval and temporal grounding.

The Temporal Challenge

What makes this fundamentally different from image retrieval? Consider the query “a person opens a gift and looks surprised.” No single frame captures this event; it unfolds across time. The act of opening involves hand motion and changing spatial relationships, while the surprise is expressed through a temporal sequence of facial expressions. A retrieval system must understand this temporal structure, which requires going beyond the frame-by-frame application of image features.

Formally, a video can be viewed as a function $V:[0,T]\to ℐ$ from time to the space of images, and a video retrieval system must compute a similarity function $s(q,V)$ that integrates information over this continuous (or discretely sampled) trajectory. The curse of dimensionality is severe: if a single image embedding has $d$ dimensions, then naively representing $T$ frames requires $T\cdot d$ dimensions, and the space of possible temporal patterns grows combinatorially.

The key technical challenge is how to aggregate information across frames. Three dominant strategies have emerged:

1. Mean pooling: compute a frame-level feature for each frame and average them: $𝒗=\frac{1}{T}{\sum }_{t=1}^T{f}_{𝜽}(F_t)$. Simple but discards temporal order entirely.

2. Temporal attention: use a transformer to attend across frame features, allowing the model to weight frames differently based on the query: $𝒗=\text{Attn}(𝒒,[f_{𝜽}(F_1),\dots ,f_{𝜽}(F_T)])$.

3. Temporal convolution / 3D CNN: process the video with spatiotemporal convolutions that jointly encode spatial and temporal patterns.

CLIP4Clip and ClipBERT: Adapting CLIP to Video

Given CLIP's remarkable success on image-text tasks, a natural question is: how do we extend it to video? The answer is not obvious, because CLIP was trained on individual images, not temporal sequences. Two influential approaches explored different strategies.

CLIP4Clip: temporal aggregation post-hoc.

Luo et al. [63] proposed CLIP4Clip, which uses CLIP's pre-trained image encoder to extract frame-level features and then aggregates them temporally. Three aggregation strategies are compared:

1. Mean pooling: $𝒗=\frac{1}{T}{\sum }_{t=1}^T{𝒊}_t$ where ${𝒊}_t=f_{𝜽}^{\text{CLIP}}(F_t)$.

2. Temporal transformer: a lightweight transformer processes the sequence $[{𝒊}_1,\dots ,{𝒊}_T]$ and outputs a single video embedding via a [CLS] token.

3. Tight coupling: late interaction between frame tokens and text tokens via cross-attention.

ClipBERT: sparse sampling for efficiency.

Lei et al. [64] took a different approach with ClipBERT. Instead of processing all frames, ClipBERT randomly samples a small number of frames (typically 1–2) during training and applies a multimodal transformer for joint image-text reasoning. The key insight is that for many text-video retrieval queries, a single well-chosen frame contains sufficient information. This dramatically reduces computational cost and enables end-to-end fine-tuning of both visual and textual components.

Video Corpus Moment Retrieval

Standard video retrieval returns whole videos, but users often seek a specific moment: “the scene where the dog jumps into the lake” or “the moment the speaker mentions climate change.” This motivates video corpus moment retrieval (VCMR): given a natural language query $q$ and a corpus of untrimmed videos $𝒱$, find both the correct video $V^{\ast }$ and the temporal segment $[{\tau }_s,{\tau }_e]$ within it.

VCMR is typically decomposed into two stages:

1. Video retrieval: select a shortlist of candidate videos using a fast retrieval model.

2. Temporal grounding: within each candidate video, localise the relevant temporal segment.

Temporal grounding.

The temporal grounding problem, also known as natural language video localisation [11][65], takes a single untrimmed video and a text query and predicts the start and end times of the described event. Three families of approaches exist:

1. Proposal-based: generate a set of temporal proposals (candidate segments) and rank them by query relevance. This is analogous to region proposal networks in object detection.

2. Proposal-free: directly regress the start and end times from a joint video-text representation. Given frame-level features ${𝒉}_t$ attended by the query, predict: (Grounding Regression)$$\begin{array}{ccc}& {\hat{\tau }}_s=\sigma ({𝒘}_s^{\top }𝒉+b_s),{\hat{\tau }}_e=\sigma ({𝒘}_e^{\top }𝒉+b_e),& \text{<span class="tml-eqn"></span>}\end{array}$$ where $𝒉$ is a query-conditioned video representation and $\sigma$ maps to $[0,1]$ (normalised time).

3. Span-based: predict a score for every possible span $(i,j)$ in a discretised timeline and select the highest-scoring span. The scoring function $s(i,j)={𝒘}^{\top }[{𝒉}_i;{𝒉}_j;{𝒉}_i\odot {𝒉}_j]+b$ captures interactions between start and end frame representations.

Multimodal Fusion for Video Understanding

Video retrieval often benefits from multiple modalities beyond visual frames: audio, speech transcripts, optical character recognition (OCR), and metadata. How these modalities are combined determines both the quality and efficiency of the retrieval system.

Recent models such as Flamingo [66] have shown that cross-attention fusion between a frozen visual encoder and a frozen language model, mediated by learned gating mechanisms, can achieve strong video understanding performance with minimal trainable parameters.

Speech Retrieval

Imagine trying to find a specific conversation in 10,000 hours of recorded meetings. You remember a colleague mentioning a particular budget figure, perhaps in the context of a quarterly review, but you cannot recall when the meeting took place, who else was present, or even the exact words that were used. Scrolling through recordings at real-time speed would take more than a year of continuous listening. This is the problem of spoken document retrieval: searching through vast archives of speech to find relevant passages, a problem that sits at the intersection of speech processing, information retrieval, and, increasingly, self-supervised representation learning.

The challenge is fundamentally harder than text retrieval. Speech is temporally extended, ambiguous (homophones, disfluencies, accents), and lacks the convenient discrete token structure of written language. A single word in text occupies a handful of bytes; the same word in speech occupies hundreds of milliseconds of high-dimensional audio signal. Moreover, speech carries information absent from text, including speaker identity, emotion, emphasis, and prosody, all of which may be relevant to the retrieval task but are difficult to index.

This section traces the evolution of speech retrieval from classical template-matching techniques through statistical models to the modern revolution brought about by self-supervised speech representations and large-scale weakly supervised ASR systems. Along the way, we will see a recurring theme: the tension between approaches that convert speech to text (sacrificing paralinguistic information for the sake of mature text retrieval infrastructure) and approaches that work directly with audio representations (preserving the full acoustic signal at the cost of less interpretable matching). Understanding this tension is essential for designing effective speech retrieval systems, as the optimal choice depends heavily on the application domain, language resources, and query types.

Dynamic Time Warping for Template Matching

The earliest approaches to speech retrieval were based on template matching: given a spoken query (a recording of the word or phrase to be found), slide it along the audio archive and measure similarity at each position. The fundamental obstacle is that the same word spoken twice, even by the same speaker, will have different durations. The vowel in “cat” might last 80,ms in one utterance and 120,ms in another. Simple Euclidean distance between the two feature sequences will fail catastrophically because corresponding frames are misaligned.

Dynamic Time Warping (DTW), introduced by Sakoe and Chiba [67], solves this problem by finding an optimal nonlinear alignment between two sequences that minimises the total matching cost.

The DTW distance can be computed efficiently by dynamic programming. Define the cumulative cost matrix $D\in {\mathrm{ℝ}}^{N\times M}$ by the recurrence (DTW Recurrence)$$\begin{array}{ccc}& D(i,j)=c({𝒙}_i,{𝒚}_j)+\min \{\begin{array}{cc}D(i-1,j)& \text{(insertion)}\\ D(i,j-1)& \text{(deletion)}\\ D(i-1,j-1)& \text{(match)}\end{array},& \text{<span class="tml-eqn"></span>}\end{array}$$ with boundary conditions $D(\mathrm{1,1})=c({𝒙}_1,{𝒚}_1)$, and $D(i,0)=D(0,j)=\infty$ for all $i,j\ge 1$. The DTW distance is then $\mathrm{DTW}(𝒙,𝒚)=D(N,M)$, and the optimal warping path can be recovered by backtracking through the matrix.

For spoken term detection, DTW is applied in a subsequence variant: rather than aligning two complete sequences, we search for the query pattern within a longer audio stream. This requires modifying the boundary conditions so that the alignment can begin at any position in the reference sequence, converting the problem from global alignment to local alignment.

HMMs for Keyword Spotting

The statistical revolution in speech processing, beginning in the late 1970s and reaching maturity in the 1990s, replaced template matching with probabilistic models. The Hidden Markov Model (HMM) [68] became the workhorse of speech recognition and, by extension, keyword spotting and spoken term detection.

The key idea is to model each word (or phoneme) as a Markov chain over hidden states, where each state emits acoustic feature vectors according to a probability distribution, typically a Gaussian Mixture Model (GMM). A keyword spotter [69] trains HMMs for the keywords of interest and a filler model that captures all non-keyword speech, then runs a Viterbi decoder to find the most likely state sequence through the combined keyword-filler model.

DTW-Based Spoken Term Detection

Despite the dominance of HMM-based approaches, DTW experienced a renaissance in the 2000s for query-by-example spoken term detection (QbE-STD), where the query is itself a spoken utterance rather than a text string. This setting is particularly relevant for low-resource languages where training HMMs or ASR systems is infeasible due to lack of transcribed data.

Algorithm 6 uses subsequence DTW, where the boundary condition on the reference sequence is relaxed: $D(0,j)=0$ for all $j$, allowing the alignment to begin at any position in the archive. The normalisation by query length $N$ ensures that scores are comparable across queries of different durations.

Self-Supervised Speech Representations

The most transformative development in speech retrieval over the past five years has been the emergence of self-supervised speech representations. These models learn rich, general-purpose representations from unlabelled speech, eliminating the need for expensive transcription and enabling high-quality retrieval even in languages with minimal labelled resources.

The core insight is borrowed from the success of masked language modelling in NLP: if a model can learn to predict missing parts of a speech signal from context, it must develop internal representations that capture phonetic, linguistic, and even semantic structure. Three models stand out as landmarks in this progression.

wav2vec 2.0: contrastive learning on speech.

The wav2vec 2.0 model [70] introduced a two-stage architecture that combines convolutional feature extraction with transformer-based contextual modelling and a contrastive learning objective. The architecture operates as follows.

A multi-layer CNN feature encoder $f_{\text{enc}}:{\mathrm{ℝ}}^T\to {\mathrm{ℝ}}^{L\times d}$ processes raw audio waveform $𝒙\in {\mathrm{ℝ}}^T$ (sampled at 16,kHz) and produces a sequence of latent representations ${𝒛}_1,\dots ,{𝒛}_L$, where each ${𝒛}_{\ell }$ corresponds to approximately 25,ms of audio with a stride of 20,ms. A quantisation module $q:{\mathrm{ℝ}}^d\to 𝒬$ discretises each latent vector into one of $|𝒬|$ codebook entries, producing quantised targets ${\widehat{𝒛}}_{\ell }=q({𝒛}_{\ell })$. A transformer encoder $g_{𝜽}:{\mathrm{ℝ}}^{L\times d}\to {\mathrm{ℝ}}^{L\times d}$ processes the (partially masked) latent sequence and produces contextualised representations ${𝒄}_1,\dots ,{𝒄}_L$.

The training objective combines a contrastive loss with a diversity penalty: (Wav2vec2 LOSS)$$\begin{array}{ccc}& {ℒ}_{\text{w2v2}}=-\sum _{\ell \in ℳ}\log \frac{\exp (\mathrm{sim}({𝒄}_{\ell },{\widehat{𝒛}}_{\ell })/\kappa )}{{\sum }_{j=1}^{|{𝒩}_{\ell }|}\exp (\mathrm{sim}({𝒄}_{\ell },{\widetilde{𝒛}}_j)/\kappa )}+\alpha {ℒ}_{\text{div}},& \text{<span class="tml-eqn"></span>}\end{array}$$ where $ℳ$ is the set of masked positions, $\mathrm{sim}(𝒂,𝒃)={𝒂}^{\top }𝒃/(\Vert 𝒂\Vert \Vert 𝒃\Vert )$ is cosine similarity, $\kappa$ is a temperature parameter, ${𝒩}_{\ell }$ contains the true quantised target ${\widehat{𝒛}}_{\ell }$ plus $K$ distractors sampled from other masked positions, and ${ℒ}_{\text{div}}$ encourages uniform use of the codebook entries.

The representations learned by wav2vec 2.0 are remarkably effective for speech retrieval. By extracting frame-level embeddings ${𝒄}_1,\dots ,{𝒄}_L$ from the transformer layers and pooling them (via mean pooling or attention-based aggregation), one obtains utterance-level representations that capture both acoustic and linguistic content. These representations can then be indexed and searched using the approximate nearest-neighbour techniques developed in earlier sections of this chapter.

HuBERT: masked prediction of pseudo-labels.

While wav2vec 2.0 uses a contrastive objective with learned quantisation targets, HuBERT [71] (Hidden-Unit BERT) takes a different approach: it performs masked prediction against offline-discovered pseudo-labels. The training proceeds in iterations:

1. Cluster: Apply $k$-means clustering to MFCC features (first iteration) or to representations from the previous model (subsequent iterations) to produce frame-level pseudo-labels $z_{\ell }\in \{1,\dots ,C\}$.

2. Predict: Train a transformer model to predict the pseudo-label $z_{\ell }$ at masked positions from the unmasked context, using a cross-entropy loss: (Hubert LOSS)$$\begin{array}{ccc}& {ℒ}_{\text{HuBERT}}=-\sum _{\ell \in ℳ}\log P(z_{\ell }|{𝒙}_{\backslash ℳ};𝜽),& \text{<span class="tml-eqn"></span>}\end{array}$$ where $ℳ$ is the set of masked positions and ${𝒙}_{\backslash ℳ}$ denotes the unmasked input.

3. Iterate: Use the trained model's representations to generate better pseudo-labels for the next iteration.

The iterative refinement is key: initial pseudo-labels from MFCC clustering are crude, capturing only basic acoustic properties. But as the model improves, its representations yield increasingly linguistically meaningful clusters, which in turn provide better training targets. This bootstrapping process converges to representations that rival or exceed those of wav2vec 2.0 on downstream tasks.

WavLM: speech processing across the full stack.

WavLM [72] builds on HuBERT's masked prediction framework but adds denoising objectives that make the learned representations robust to noise, reverberation, and overlapping speakers. During training, the input audio is augmented by mixing with noise and other utterances, while the prediction targets remain the clean pseudo-labels. This forces the model to learn representations that are invariant to acoustic degradation, a critical property for retrieval in real-world conditions where recordings are rarely studio-quality.

WavLM additionally benefits from a gated relative position bias in its transformer architecture, which provides more flexible modelling of temporal dependencies than absolute position embeddings.

This noise robustness is critical for speech retrieval in real-world settings: meeting recordings contain air conditioning hum, keyboard clicks, and cross-talk; field recordings suffer from wind noise and equipment handling; phone calls are degraded by codec compression and network jitter. Models trained with denoising objectives produce stable embeddings under all these conditions, enabling retrieval systems that work reliably outside the laboratory.

Model	Parameters	Pre-training Data	Objective	Key Innovation
wav2vec 2.0	317M	960h (LS) / 60kh (LV)	Contrastive	Learned quantisation
HuBERT	316M	960h (LS) / 60kh (LV)	Masked prediction	Iterative pseudo-labels
WavLM	316M	960h (LS) / 94kh (mix)	Masked prediction	Denoising augmentation

Whisper: Universal ASR via Weak Supervision

A fundamentally different approach to speech retrieval is to transcribe the speech into text and then apply standard text retrieval techniques. The viability of this approach depends entirely on the quality of the automatic speech recognition (ASR) system. For decades, ASR errors propagated catastrophically into the retrieval pipeline, degrading recall by 20–40% relative to retrieval on manual transcriptions. Whisper [73] changed this calculus.

Whisper is a sequence-to-sequence transformer trained on 680,000 hours of weakly supervised audio-transcript pairs collected from the internet. The key insight is scale over curation: rather than training on a small, carefully transcribed dataset, Whisper trains on a massive, noisy dataset and relies on the model's capacity to learn robust transcription despite label noise.

The architecture follows a standard encoder-decoder transformer. The audio encoder processes log-mel spectrograms (80 channels, computed from 25,ms windows with a 10,ms stride) through a stack of transformer blocks with sinusoidal position embeddings. The text decoder autoregressively generates tokens conditioned on the encoder outputs and previously generated tokens. Special tokens control language identification, timestamp prediction, and translation.

The scale of Whisper's training is worth emphasising. The 680,000 hours of audio span 99 languages, collected by filtering internet audio-transcript pairs for quality using heuristic alignment checks. The largest model configuration (Whisper Large V3) uses 1.55 billion parameters with a 32-layer encoder and 32-layer decoder, each with 1,280-dimensional hidden states and 20 attention heads. Training was performed on a cluster of GPUs for multiple weeks, a brute-force investment in scale that, combined with the massive and diverse dataset, produced a model with remarkable robustness to accents, background noise, and domain shifts. For retrieval purposes, this robustness translates directly to transcript quality: fewer ASR errors mean less information loss in the transcribe-then-retrieve pipeline.

The effectiveness of Whisper-based retrieval stems from the model's remarkable robustness. Whisper achieves near-human word error rates across a wide range of accents, noise conditions, and domains, making the transcribe-then-retrieve pipeline competitive with (and often superior to) direct acoustic retrieval for languages where Whisper performs well.

Spoken Document Retrieval: Architectures Compared

With all these components in hand, we can now compare the two main architectural paradigms for spoken document retrieval.

The ASR-based pipeline transcribes audio into text and applies text retrieval. Its strengths are modularity (any ASR system can be combined with any text retrieval system), interpretability (users can see the transcripts and understand why a document was retrieved), and the ability to leverage decades of progress in text retrieval. Its weakness is error propagation: ASR errors, particularly on named entities, technical terms, and code-switched speech, directly degrade retrieval performance.

The embedding-based pipeline maps both audio and queries directly into a shared embedding space, bypassing transcription entirely. Self-supervised models like wav2vec 2.0, HuBERT, or WavLM produce fixed-dimensional embeddings that capture acoustic and linguistic content. Retrieval is performed by nearest-neighbour search in embedding space. This approach is robust to ASR errors (there is no ASR step to produce errors), works for any language (no language-specific training required), and preserves paralinguistic information. Its weakness is the loss of fine-grained lexical information: embedding-based retrieval may miss exact keyword matches that BM25 would catch.

In practice, the best systems often combine both paradigms. A hybrid retrieval system might use BM25 on Whisper transcripts for initial candidate generation (leveraging exact keyword matching), then re-rank using embedding similarity from a self-supervised speech model (incorporating acoustic and semantic information). This two-stage architecture captures the strengths of both approaches while mitigating their individual weaknesses.

Music Information Retrieval

You are in a coffee shop when a song begins playing through the speakers, something you have heard before but cannot quite place. You pull out your phone, open an app, hold it up for a few seconds, and within moments the screen displays the song title, artist, and album art. This act, so casual that it barely registers as remarkable, depends on one of the most elegant algorithms in all of computer science: audio fingerprinting. More broadly, it exemplifies the field of Music Information Retrieval (MIR), which encompasses the computational analysis of, search through, and recommendation of music.

This section explores three facets of music retrieval: exact identification (“What song is this?”), similarity-based search (“Find songs that sound like this one”), and the modern integration of music with language models.

Audio Fingerprinting: The Shazam Algorithm

The Shazam algorithm, published by Wang [74], solves a needle-in-a-haystack problem: given a short, noisy recording captured through a phone's microphone in a crowded environment, identify the exact song from a database of millions of tracks. The solution is based on three key ideas: spectral peak extraction, combinatorial hashing, and hash-table lookup.

The Shazam algorithm constructs fingerprints through a process that is best understood as building a constellation map of the audio signal's time-frequency structure.

Step 1: Spectrogram computation.

Compute the short-time Fourier transform (STFT) of the audio signal, producing a spectrogram $S(t,f)=|X(t,f){|}^2$, where $X(t,f)$ is the STFT coefficient at time frame $t$ and frequency bin $f$.

Step 2: Peak extraction.

Identify spectral peaks: time-frequency points $(t_i,f_i)$ where the spectrogram energy is a local maximum, exceeding the energy of all neighbouring points within a region of size $\mathrm{\Delta }t\times \mathrm{\Delta }f$. These peaks are remarkably robust to noise and distortion because they correspond to the dominant resonances in the audio signal, which persist even when the signal is degraded.

Step 3: Constellation map.

The set of all spectral peaks $𝒞=\{(t_1,f_1),(t_2,f_2),\dots ,(t_P,f_P)\}$ forms the constellation map. This map is a sparse representation of the audio signal that discards most of the spectrogram energy and retains only the structural skeleton.

Step 4: Combinatorial hashing.

For each anchor point $(t_a,f_a)$ in the constellation map, select nearby points $(t_b,f_b)$ within a target zone $\{(t,f):t_a+t_{\min }\le t\le t_a+t_{\max },f_{\min }\le f\le f_{\max }\}$ and form a hash: (Shazam HASH)$$\begin{array}{ccc}& h=\text{hash}(f_a,f_b,t_b-t_a).& \text{<span class="tml-eqn"></span>}\end{array}$$ The hash encodes the frequencies of the anchor and target peaks and their time difference. Each hash is stored in a hash table along with the song identifier and the anchor time $t_a$: $$\text{table}[h]\leftarrow \text{table}[h]\cup \{(\text{song\_id},t_a)\}.$$

Step 5: Matching and verification.

Given a query recording, the same process is applied to extract query hashes $\mathrm{\Phi }(q)$. Each query hash is looked up in the hash table to retrieve candidate matches. A true match produces a coherent time offset: if the query was captured starting at time $t_0$ in the original song, then all matching hashes should satisfy $t_a^{\text{song}}-t_a^{\text{query}}\approx t_0$. The song with the most coherent matches (the largest peak in the histogram of time offsets) is returned as the identification.

The elegance of Algorithm 7 lies in its $𝒪(1)$ hash-table lookups: for each query hash, the lookup returns a small set of candidate matches, and the time-offset histogram aggregates evidence across all query hashes to identify the correct song. The entire process typically completes in under one second for a database of millions of songs.

Music Similarity and Recommendation

While fingerprinting answers “What song is this?”, a different question drives much of music information retrieval [75][76]: “What songs sound like this one?” Answering this question requires extracting features that capture the perceptual qualities of music and defining meaningful similarity measures over these features.

Music features are traditionally organised into a hierarchy spanning three levels of abstraction.

Music similarity is then computed by representing each song as a feature vector (or sequence of feature vectors) and measuring distance in feature space. A common approach represents a song by the mean and covariance of its frame-level MFCC vectors [77], forming a Gaussian model per song, and measures similarity using the Kullback–Leibler divergence between these Gaussians: (Music Similarity KL)$$\begin{array}{ccc}& d_{\mathrm{KL}}(p\Vert q)=\frac{1}{2}(\mathrm{tr}({𝚺}_q^{-1}{𝚺}_p)+{({𝝁}_q-{𝝁}_p)}^{\top }{𝚺}_q^{-1}({𝝁}_q-{𝝁}_p)-d+\log \frac{|{𝚺}_q|}{|{𝚺}_p|}),& \text{<span class="tml-eqn"></span>}\end{array}$$ where $({𝝁}_p,{𝚺}_p)$ and $({𝝁}_q,{𝚺}_q)$ are the mean and covariance of the MFCC distributions for songs $p$ and $q$ respectively, and $d$ is the feature dimensionality.

Query-by-Humming

One of the most user-friendly forms of music retrieval is query-by-humming (QBH): a user hums, sings, or whistles a melody, and the system identifies the song. This is a fundamentally different problem from fingerprinting because the query is a highly distorted version of the original: the rhythm may be approximate, the pitch may be shifted, and the timbre is entirely different (a human voice rather than instruments).

The QBH problem reduces to melody matching: comparing the pitch contour of the hummed query against a database of reference melodies. The pitch contour is extracted from the query using a fundamental frequency ($F_0$) estimation algorithm, producing a sequence of pitch values ${𝒑}^q=(p_1^q,\dots ,p_N^q)$. Reference melodies can be extracted from MIDI files, audio recordings, or music score databases.

Matching is performed using a variant of DTW that operates on pitch intervals (differences between consecutive pitch values) rather than absolute pitch values, making the system transposition-invariant: (QBH DTW)$$\begin{array}{ccc}& c(p_i^q,p_j^r)=|(p_i^q-p_{i-1}^q)-(p_j^r-p_{j-1}^r)|,& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${𝒑}^q$ and ${𝒑}^r$ are the query and reference pitch sequences respectively. This formulation ensures that a query hummed in a different key still matches the correct song.

Modern Approaches: Neural Music Representations

The feature engineering approach to music similarity is being supplanted by deep learning models that learn representations end-to-end. Two recent models illustrate this trend.

MusicGen.

MusicGen [78] is a language model for music generation that operates on discrete audio tokens produced by EnCodec, a neural audio codec. While designed for generation, MusicGen's internal representations (the hidden states of its transformer layers) serve as powerful feature extractors for music retrieval. These representations encode harmonic, rhythmic, and timbral information at multiple levels of abstraction, learned implicitly through the generation objective.

MusicLM.

MusicLM [79] generates high-fidelity music from text descriptions by combining MuLan (a music-text embedding model) with AudioLM (an audio language model). The MuLan embedding space, where music and text descriptions are aligned, provides a natural basis for text-to-music retrieval: given a text query like “upbeat jazz piano with brushed drums,” retrieve songs whose MuLan embeddings are closest to the text embedding.

These models represent a paradigm shift: instead of hand-engineering features for each aspect of music (timbre, rhythm, harmony), a single model learns a unified representation that captures all relevant dimensions simultaneously. For retrieval, this means that a single embedding can support diverse query types, from acoustic similarity to semantic text-based search.

Music-Text Retrieval

The connection between music and natural language has become a central focus of modern MIR. Music-text retrieval enables two complementary tasks: text-to-music retrieval (finding songs that match a text description) and music-to-text retrieval (generating or retrieving text descriptions for a given song, often called music captioning).

The dominant approach follows the contrastive dual-encoder paradigm familiar from CLIP and its variants: a music encoder and a text encoder are trained jointly to produce aligned embeddings, using a contrastive loss that pushes matching music-text pairs together and non-matching pairs apart. The CLAP framework [22], discussed in detail in the next section, provides the foundation for this approach.

Content-based vs. collaborative filtering.

Music recommendation systems traditionally use one of two approaches. Content-based filtering recommends songs based on acoustic similarity to songs the user has enjoyed, using the features and similarity measures described above. Collaborative filtering recommends songs based on the listening patterns of similar users, without analysing the audio content at all. Modern systems combine both approaches in hybrid recommenders that use content-based features to address the cold-start problem (recommending new songs with no listening history) while leveraging collaborative signals for personalisation.

The emergence of music-text models adds a third dimension: users can now describe what they want in natural language, and the system can retrieve music that matches the description even if the user has never heard similar music before. This language-mediated retrieval is particularly powerful because natural language can express complex, compositional musical concepts (“a slow, melancholic cello piece reminiscent of Elgar”) that are difficult to capture with acoustic features alone.

Audio-Language Retrieval

The previous two sections focused on speech and music, two structured forms of audio with well-developed retrieval traditions. But the auditory world extends far beyond speech and music: the creak of a door, the rumble of thunder, a car horn in traffic, the chirping of crickets at dusk. General audio retrieval, the ability to search for arbitrary sounds using natural language descriptions, has emerged as a frontier where audio understanding meets language understanding, producing models that can find sounds they have never been explicitly trained to recognise.

The key enabler is Contrastive Language-Audio Pre-training (CLAP) [22], which extends the CLIP paradigm from images to audio, learning a shared embedding space where audio recordings and text descriptions are aligned.

CLAP: Contrastive Language-Audio Pre-training

CLAP follows the dual-encoder contrastive framework that has proven so effective for vision-language models, adapted for the audio domain. The architecture consists of two components.

Audio encoder.

The audio encoder $f_{\text{audio}}:{\mathrm{ℝ}}^{F\times T}\to {\mathrm{ℝ}}^d$ takes a log-mel spectrogram as input and produces a $d$-dimensional embedding. Common choices include:

• HTSAT (Hierarchical Token-Semantic Audio Transformer): a Swin Transformer adapted for audio spectrograms, which captures both local spectral patterns and global temporal structure.

• PANN (Pre-trained Audio Neural Network): CNN-based architectures pre-trained on AudioSet for audio tagging, fine-tuned within the CLAP framework.

• BEATs: an audio pre-training framework using iterative audio tokenisation and masked prediction.

Text encoder.

The text encoder $f_{\text{text}}:{𝒱}^{\ast }\to {\mathrm{ℝ}}^d$ maps a text description to the same $d$-dimensional space. CLAP typically uses a pre-trained language model (RoBERTa, BERT, or GPT-2) as the backbone, with a projection head that maps the [CLS] token representation to the shared embedding space.

Training objective.

The model is trained with the symmetric contrastive loss familiar from CLIP: (CLAP LOSS)$$\begin{array}{ccc}{ℒ}_{\text{CLAP}}=-\frac{1}{2B}\sum _{i=1}^B& (\log \frac{\exp \left(⟨f_{\text{audio}}({𝒂}_i),f_{\text{text}}({𝒕}_i)⟩/\kappa \right)}{{\sum }_{j=1}^B\exp \left(⟨f_{\text{audio}}({𝒂}_i),f_{\text{text}}({𝒕}_j)⟩/\kappa \right)}& +\log \frac{\exp \left(⟨f_{\text{text}}({𝒕}_i),f_{\text{audio}}({𝒂}_i)⟩/\kappa \right)}{{\sum }_{j=1}^B\exp \left(⟨f_{\text{text}}({𝒕}_i),f_{\text{audio}}({𝒂}_j)⟩/\kappa \right)}),\end{array}$$ where ${\{({𝒂}_i,{𝒕}_i)\}}_{i=1}^B$ is a batch of audio-text pairs, $⟨\cdot ,\cdot ⟩$ denotes the inner product in the shared embedding space, and $\kappa >0$ is a learnable temperature parameter.

Training data.

CLAP models are trained on audio-text pairs drawn from several sources:

• AudioSet [13]: over 2 million 10-second YouTube clips with human-annotated labels, which can be converted to text descriptions via templates (e.g., label “Dog bark” becomes “The sound of a dog barking”).

• AudioCaps: a subset of AudioSet with human-written free-form captions describing the audio content in natural language.

• LAION-Audio-630K [14]: a large-scale dataset of 633,526 audio-text pairs collected from the web, covering music, environmental sounds, and speech.

• FreeSound: a collaborative database of Creative Commons–licensed sounds with user-provided descriptions.

LAION-Audio: Scaling Audio-Text Pre-training

The effectiveness of contrastive audio-language models scales with the size and diversity of the training data. LAION-Audio [14] addresses this by providing a large-scale, openly available dataset of audio-text pairs harvested from the web. The collection pipeline mirrors the approach used for LAION-5B in the image domain:

1. Crawl web pages containing audio content (YouTube, FreeSound, BBC Sound Effects, etc.).

2. Extract audio files and their associated text metadata (titles, descriptions, tags, captions).

3. Filter for quality: remove very short clips ($<1$,s), very long clips ($>30$,s), and clips with uninformative descriptions.

4. De-duplicate by audio fingerprinting to avoid training on near-identical clips.

The resulting LAION-Audio-630K dataset contains 633,526 audio-text pairs spanning diverse categories: environmental sounds (rain, traffic, machinery), music (instruments, genres, moods), speech-related sounds (laughter, applause, crowd noise), and animal vocalisations. Models trained on LAION-Audio consistently outperform those trained on AudioSet alone, particularly on out-of-domain retrieval tasks, because the web-sourced data provides greater diversity in both audio content and text descriptions.

ImageBind: Binding Audio to Other Modalities

CLAP binds audio to text. CLIP binds images to text. A natural question arises: can we bind all modalities, including audio, images, text, video, depth, and thermal, into a single shared embedding space? ImageBind [15] provides an affirmative answer through a remarkably elegant mechanism.

The key insight is that images can serve as a universal binding modality. Because high-quality paired data exists between images and each other modality (image-text pairs, image-audio pairs from video, image-depth pairs from RGB-D sensors), training separate encoders to align each modality with images implicitly aligns all modalities with each other. If audio embedding $𝒂$ is close to image embedding $𝒗$ in one space, and text embedding $𝒕$ is close to image embedding $𝒗$ in another space, then $𝒂$ and $𝒕$ should be close to each other, even though they were never directly paired during training.

The practical implication is striking: by training $M$ modality-specific encoders against a single image encoder, ImageBind obtains $\left(\genfrac{}{}{0px}{}{M}{2}\right)=M(M-1)/2$ pairwise retrieval capabilities, of which only $M-1$ are directly trained. For $M=6$ modalities, this means 15 pairwise tasks from only 5 trained alignments, a 3$\times$ amplification of retrieval capabilities.

For audio retrieval, ImageBind's contribution is the ability to perform cross-modal audio retrieval without direct audio-text or audio-image paired training data. Given a text query, one can retrieve audio clips by comparing text embeddings to audio embeddings in the shared space, even if the model was trained only on image-text and image-audio pairs. This emergent capability dramatically expands the scope of audio retrieval.

Sound Event Retrieval

Sound event retrieval focuses on finding specific sounds within audio recordings based on textual descriptions. This capability is essential for applications including sound effect production (finding the right door creak for a film), environmental monitoring (detecting specific bird calls in ecological surveys), and accessibility (alerting hearing-impaired users to relevant environmental sounds).

The retrieval task can be formalised as follows.

The challenge in sound event retrieval lies in the compositionality of natural language descriptions. A query like “a dog barking in the distance with traffic noise in the background” requires the model to understand multiple concurrent sound sources, their spatial relationships, and their relative prominence. Current CLAP models handle simple compositional queries reasonably well but struggle with complex spatial or temporal relationships.

Environmental sound classification provides an important testing ground for audio retrieval models. The ESC-50 benchmark, containing 2,000 five-second recordings across 50 classes of environmental sounds (from “dog bark” to “chainsaw” to “pouring water”), has become the standard evaluation for zero-shot audio understanding. A CLAP model achieves zero-shot classification on ESC-50 by computing the similarity between the audio embedding and text embeddings for each of the 50 class labels, selecting the label with the highest similarity. The progression from 72% accuracy (early CLAP models) to over 93% (current state of the art) reflects improvements in both model architecture and training data scale.

Model	AudioCaps	Clotho	ESC-50 (ZS)	Training Data
CLAP (HTSAT)	57.8	27.4	82.6	AudioSet + AudioCaps
LAION-CLAP	64.2	33.1	90.4	LAION-Audio-630K
ImageBind	51.3	22.8	77.2	Image-paired data
MS-CLAP (2023)	68.7	36.2	93.1	AudioSet + LAION + FSD50K

Retrieval-Augmented Audio Generation

A natural application of audio retrieval is to enhance audio generation models by grounding them in retrieved examples. Just as retrieval-augmented generation (RAG) improves text generation by conditioning on relevant documents, retrieval-augmented audio generation improves the fidelity and controllability of audio synthesis by conditioning on retrieved audio clips.

Re-AudioLDM.

Re-AudioLDM [32] augments the AudioLDM [80] latent diffusion model with a retrieval module. Given a text prompt describing the desired audio (e.g., “rain falling on a metal roof with occasional thunder”), the system first retrieves the $k$ most relevant audio clips from a database using CLAP embeddings, then conditions the diffusion process on both the text prompt and the retrieved audio features: (Reaudioldm)$$\begin{array}{ccc}& p_{𝜽}({𝒙}_0|𝒕,ℛ)=\int p_{𝜽}({𝒙}_0|{𝒙}_T,𝒕,ℛ)p({𝒙}_T)d{𝒙}_T,& \text{<span class="tml-eqn"></span>}\end{array}$$ where $𝒕$ is the text conditioning, $ℛ=\{r_1,\dots ,r_k\}$ is the set of retrieved audio clips, and ${𝒙}_T\to {𝒙}_0$ denotes the reverse diffusion process. The retrieved clips provide concrete acoustic examples that guide the diffusion process toward generating audio with the right timbral and spectral characteristics.

RECAP.

RECAP [33] takes a complementary approach, focusing on the retrieval of audio captions rather than audio clips. Given a text prompt, RECAP retrieves similar captions from a large database and uses them to enrich the text conditioning for the generation model. This is particularly effective for underspecified prompts: a user who writes “ocean” gets the benefit of retrieved captions like “gentle waves lapping on a sandy beach with distant seagulls,” which provide the acoustic detail needed for high-quality generation.

Multimodal Embedding Spaces

Imagine a single coordinate system in which a photograph of a thunderstorm, the sound of rolling thunder, the sentence “dark clouds gathering before a downpour,” a video clip of lightning, and even the thermal signature of a storm front all live as nearby points. Not merely nearby within their own modality, but nearby to each other, so that the sound of thunder sits closer to the image of lightning than to the sound of birdsong, and the sentence sits closer to the photograph than to an unrelated paragraph about tax policy. This is the grand vision of multimodal embedding spaces: a single, shared geometric arena in which every form of human experience, text, vision, audio, touch, motion, can be compared, combined, and retrieved with the same inner product.

The vision is revolutionary because it collapses the tower of modality-specific systems that has dominated machine learning for decades. Instead of separate text search, image search, audio search, and video search engines, each with bespoke architectures, training pipelines, and index structures, one obtains a single embedding model and a single vector index. A query in any modality retrieves results in every modality, with relevance measured by a single score.

The vision is also incredibly challenging. Each modality lives on a fundamentally different data manifold. Text is discrete and sequential; images are dense and spatial; audio is temporal and frequency-structured; video is spatiotemporal; depth maps are geometric. Aligning these manifolds into a shared space requires not just large paired datasets but a principled framework for defining what “alignment” even means when the modalities carry different information at different levels of abstraction.

The difficulty scales superlinearly with the number of modalities. For $M$ modalities, there are $\left(\genfrac{}{}{0px}{}{M}{2}\right)$ pairwise alignment tasks, but paired data is typically available for only a small subset of these pairs. Collecting audio-depth pairs, or text-thermal pairs, or IMU-video pairs at scale is prohibitively expensive. This data scarcity problem is what makes the naive approach of training $\left(\genfrac{}{}{0px}{}{M}{2}\right)$ separate contrastive models impractical, and it is what motivates the elegant solutions we study next.

ImageBind: Binding Six Modalities Through Images

The key insight behind ImageBind [15] is deceptively simple: images are nature's universal pairing mechanism. Consider the modalities we care about: text, audio, depth, thermal imagery, and inertial measurement unit (IMU) data. In the natural world, images are routinely paired with each of these modalities. Web data provides billions of image-text pairs. Videos provide image-audio pairs (every video frame is paired with its soundtrack). RGB-D cameras provide image-depth pairs. Thermal cameras provide image-thermal pairs. Wearable devices with cameras provide image-IMU pairs.

None of these pairings are exotic or expensive; they arise organically from the way sensors capture the world. The genius of ImageBind is to exploit this star topology of natural pairings, with images at the hub, to bind all six modalities into a single embedding space without ever needing direct pairings between non-image modalities.

The architecture of ImageBind uses a Vision Transformer (ViT-H) as the image encoder $f_1$, initialized from a pretrained model. Each non-image modality $m$ uses a separate Transformer encoder $f_m$, with modality-specific tokenization: audio spectrograms are treated as 2D patches (following the Audio Spectrogram Transformer approach), depth and thermal maps are treated as single-channel images, IMU signals are treated as 1D sequences, and text uses the standard tokenizer from CLIP [9].

The emergent cross-modal retrieval capabilities are perhaps ImageBind's most striking contribution. Consider the following: train the model exclusively on (image, audio), (image, text), (image, depth), (image, thermal), and (image, IMU) pairs. Now, at inference time, compute embeddings for an audio clip and a text description. Despite never having been trained on audio-text correspondence, the model can retrieve the correct text given an audio query, and vice versa. This is zero-shot emergent cross-modal retrieval, and it arises purely from the geometry of the shared embedding space.

The practical implication is that image-anchored alignment makes distant modalities align, but the guarantee is weaker than a naive additive estimate suggests. If each pairwise cosine similarity exceeds $0.8$, the emergent similarity is only guaranteed to exceed $0.8\cdot 0.8-0.6\cdot 0.6=0.28$ (not the additive $0.6$), still positive and useful, but a reminder that transitivity of cosine similarity on the sphere is sublinear.

Gemini Embedding 2: Natively Multimodal Embeddings

While ImageBind demonstrated that multimodal alignment could emerge from image-anchored contrastive learning, a natural question followed: what if a model were trained from the ground up to be natively multimodal, understanding text, images, video, audio, and structured documents within a single architecture? Google's Gemini Embedding [16][17] represents exactly this next step, leveraging the full power of the Gemini multimodal foundation model to produce embeddings that are natively multimodal rather than post-hoc aligned.

The key differences from approaches like ImageBind are architectural and philosophical. ImageBind uses separate encoders per modality, aligned through a shared loss. Gemini Embedding uses a single multimodal backbone, the Gemini model itself, which processes all modalities natively through a unified Transformer architecture. The embedding is extracted from this shared representation, yielding a 3072-dimensional vector for any input regardless of modality [18][81].

Matryoshka Representation Learning.

One of the most elegant features of Gemini Embedding is its support for Matryoshka representations. Named after Russian nesting dolls, this technique trains the embedding such that every prefix of the full $d$-dimensional vector is itself a valid, useful embedding at a lower dimension. Formally, let $f(x)\in {\mathrm{ℝ}}^d$ be the full embedding. For any $d^{\prime }<d$, the truncated embedding $f{(x)}_{1:d^{\prime }}\in {\mathrm{ℝ}}^{d^{\prime }}$ (the first $d^{\prime }$ coordinates) preserves as much information as possible for that dimensionality.

The practical benefit is enormous. A production system can store the full 3072-dimensional embedding for maximum accuracy, but perform initial candidate retrieval using only the first 256 or 512 dimensions for speed and memory efficiency, then re-rank using the full embedding. This yields a natural coarse-to-fine retrieval pipeline without training separate models at each dimensionality.

Task-Type Conditioning.

Gemini Embedding introduces another innovation: task-type conditioning. Rather than producing a single, task-agnostic embedding, the model accepts a task_type parameter that conditions the embedding computation for specific downstream tasks [17][82]:

• retrieval_document: optimized for indexing documents to be retrieved.

• retrieval_query: optimized for queries that will be matched against documents.

• semantic_similarity: optimized for computing pairwise similarity between inputs.

• classification: optimized for downstream classification tasks.

• clustering: optimized for grouping similar inputs.

The Modality Gap Problem

Despite the elegance of shared embedding spaces, a persistent and somewhat mysterious phenomenon plagues all multimodal embedding models: the modality gap. When one visualizes the embedding space (via t-SNE, UMAP, or even by simply examining inter-modality versus intra-modality distances), a striking pattern emerges. Embeddings from the same modality cluster together, forming modality-specific “islands” in the shared space, separated by a persistent gap. Text embeddings occupy one region, image embeddings occupy another, and audio embeddings occupy a third, even when the text, image, and audio all describe the same semantic content.

This is deeply counterintuitive. The contrastive training objective explicitly pulls semantically related cross-modal pairs together. Why, then, do the modalities refuse to fully overlap?

The gap is not a training failure; it is a geometric consequence of high-dimensional contrastive learning. The explanation lies in what has been called the cone effect.

The Cone Effect.

When modality-specific encoders are initialized independently (as they must be, since they process different input types), their initial embeddings occupy different, nearly orthogonal regions of the high-dimensional sphere ${𝕊}^{d-1}$. In high dimensions, two random unit vectors are nearly orthogonal with overwhelming probability: for ${𝒛}_1,{𝒛}_2\sim \text{Uniform}({𝕊}^{d-1})$, $𝔼[⟨{𝒛}_1,{𝒛}_2⟩]=0$ and $\text{Var}(⟨{𝒛}_1,{𝒛}_2⟩)=1/d$, so the cosine similarity concentrates tightly around zero for large $d$.

Each modality's embeddings initially lie in a narrow cone centered around a modality-specific mean direction ${\overline{𝒛}}_m$. Contrastive training pulls matched cross-modal pairs closer, but it does so within the constraint that all embeddings remain on the unit sphere. The optimization can rotate the cones toward each other, but fully merging them would require every modality encoder to produce identical representations for semantically identical inputs across fundamentally different input spaces. This is an overconstrained problem: the encoders must simultaneously maintain intra-modal discriminability (distinguishing different inputs within a modality) and achieve cross-modal alignment, and these objectives are in tension.

Comparison of Multimodal Embedding Models

The landscape of multimodal embedding models has evolved rapidly. Table 8 summarizes the key characteristics of the most influential models.

Model	Modalities	Arch.	Dim.	Matryoshka	Key Innovation
CLIP [9]	Text, Image	Sep.	512/768	No	Large-scale contrastive
					image-text pretraining
[3pt] ALIGN [19]	Text, Image	Sep.	640	No	Noisy data at extreme
					scale (1.8B pairs)
[3pt] SigLIP [20]	Text, Image	Sep.	768/1152	No	Sigmoid loss replaces
					softmax contrastive
[3pt] BLIP-2 [21]	Text, Image	Sep.	768	No	Q-Former bridge
					between modalities
[3pt] CLAP [22]	Text, Audio	Sep.	512	No	Contrastive language-
					audio pretraining
[3pt] ImageBind [15]	6 modalities	Sep.	1024	No	Image-anchored binding
					of all modalities
[3pt] Gemini Emb. [16]	5 modalities	Uni.	3072	Yes	Native multimodal
					with task conditioning

The algorithm above illustrates the coarse-to-fine retrieval pipeline enabled by Matryoshka embeddings. The coarse stage uses low-dimensional prefixes for fast candidate generation over the entire corpus, while successive re-ranking stages use progressively higher-dimensional prefixes on progressively smaller candidate sets. This yields a total retrieval cost that is dominated by the coarse stage, with re-ranking adding negligible overhead.

Several trends emerge from this comparison. First, the field is moving from bimodal (text-image only) to truly multimodal systems. Second, embedding dimensionality is increasing, reflecting the need to capture richer semantic information across more modalities. Third, the architectural trend is toward unified backbones rather than separate encoders, leveraging the internal multimodal representations of large foundation models. Fourth, Matryoshka representations are emerging as a practical necessity for production systems that must balance accuracy against latency and storage costs.

Cross-Modal Retrieval

With multimodal embedding spaces established, we turn to the central capability they enable: cross-modal retrieval, the task of using a query in one modality to retrieve relevant results in another. This is not merely a technical exercise; it is a fundamental shift in how humans can interact with information. Instead of being confined to searching text with text queries or images with image queries, cross-modal retrieval allows a user to hum a melody and find the music video, sketch a rough drawing and find photographs, or describe a sound in words and find audio recordings.

The mathematical framework is a natural extension of single-modal retrieval to the cross-modal setting.

Text-to-Image and Image-to-Text Retrieval

Text-image retrieval is the most extensively studied cross-modal task, catalyzed by the success of CLIP [9] and the abundance of web-crawled image-text pairs. The task comes in two directions: text-to-image (given a caption, find the matching image) and image-to-text (given an image, find the matching caption).

These two directions are not symmetric in difficulty. Image-to-text retrieval is generally easier because a single image typically has many valid textual descriptions (a photograph of a dog in a park can be described as “a dog playing in the grass,” “a golden retriever in a park,” “an animal outdoors on a sunny day,” etc.), so the retrieval system has many correct targets. Text-to-image retrieval is harder because a specific textual description typically matches fewer images in the corpus.

The state-of-the-art approach follows a two-stage pipeline. In the embedding stage, both the query and all corpus items are encoded into the shared space. In the retrieval stage, approximate nearest neighbor (ANN) search identifies the top-$k$ candidates. The embedding stage is dominated by the encoder's computational cost (a forward pass through a Transformer), while the retrieval stage is dominated by the ANN search cost (sublinear in $N$ for tree-based or graph-based indices).

In practice, hard negative mining is critical for achieving strong cross-modal retrieval. Training with random negatives converges quickly to a “good enough” alignment but plateaus well below the performance achievable with carefully mined hard negatives. The most effective approach uses the current model to find hard negatives within a large candidate pool, then trains on batches enriched with these hard negatives.

For SigLIP [20], the contrastive loss is reformulated using a sigmoid function applied to each query-document pair independently, rather than the softmax over the batch used in CLIP. This removes the requirement for a global normalization across the batch and enables more stable training at very large batch sizes.

Text-to-Audio and Audio-to-Text Retrieval

The extension of contrastive alignment from the visual domain to audio produced CLAP (Contrastive Language-Audio Pretraining) [22], which learns a shared embedding space for text and audio. CLAP follows the same architectural blueprint as CLIP: a text encoder (typically a pretrained language model) and an audio encoder (typically a convolutional or Transformer model operating on mel-spectrogram inputs) are trained with contrastive loss on paired (text, audio) data.

The formal objective is identical to CLIP's InfoNCE loss, with the image encoder replaced by an audio encoder: (CLAP LOSS)$$\begin{array}{ccc}& {ℒ}_{\text{CLAP}}=-\frac{1}{B}\sum _{i=1}^B\log \frac{\exp \left(⟨f_t({𝒙}_i^t),f_a({𝒙}_i^a)⟩/\tau \right)}{{\sum }_{j=1}^B\exp \left(⟨f_t({𝒙}_i^t),f_a({𝒙}_j^a)⟩/\tau \right)},& \text{<span class="tml-eqn"></span>}\end{array}$$ where $f_t$ is the text encoder, $f_a$ is the audio encoder, $({𝒙}_i^t,{𝒙}_i^a)$ are matched text-audio pairs, and $B$ is the batch size.

The unique challenges of text-audio retrieval include:

• Temporal extent: Audio events unfold over time. A text description may refer to the overall scene (“a busy street”), a specific event (“a car horn honking”), or a temporal pattern (“footsteps gradually getting louder”). The audio encoder must capture temporal structure at multiple scales.

• Polyphony: Natural audio contains multiple simultaneous sources. A recording of “birds singing in a forest with a stream” contains at least three overlapping sound classes. The embedding must represent this compositional structure.

• Data scarcity: Compared to the billions of image-text pairs available on the web, text-audio pairs are orders of magnitude rarer. AudioCaps contains only 46k clips; Clotho contains only 5k.

Despite these challenges, CLAP enables compelling applications: natural language audio search (“find me the sound of rain on a tin roof”), audio captioning (generating text descriptions of audio), and zero-shot audio classification (classifying sounds using text labels without task-specific training).

Video-Text Retrieval

Video-text retrieval adds the dimension of temporal complexity. A video is not merely a collection of frames but a sequence of events, actions, and state changes that unfold over time. A text description of a video may refer to a single moment (“the cat jumps onto the table”), a temporal sequence (“first the chef chops onions, then adds them to the pan”), or a high-level summary (“a cooking tutorial for French onion soup”).

The computational challenge is also more severe. Encoding a video requires processing dozens to hundreds of frames. Naive approaches that encode every frame independently and aggregate the embeddings lose temporal information. More sophisticated approaches use temporal attention mechanisms, 3D convolutions, or video-specific Transformer architectures, but these increase computational cost proportionally.

Any-to-Any Retrieval: The Ultimate Goal

The logical endpoint of multimodal embedding spaces is any-to-any retrieval: the ability to use a query in any modality to retrieve results in any other modality. With $M$ modalities, this subsumes all $M^2$ directed retrieval tasks (including intra-modal retrieval as a special case) under a single framework.

Zero-Shot Cross-Modal Transfer

The most intellectually satisfying capability of unified embedding spaces is zero-shot cross-modal transfer: performing retrieval between modality pairs for which no paired training data exists. ImageBind [15] demonstrates this concretely: given only (image, audio) and (image, text) training pairs, the model can perform audio-to-text retrieval without having seen a single audio-text pair.

This capability has profound implications for scalability. With $M$ modalities, direct pairwise alignment requires $\left(\genfrac{}{}{0px}{}{M}{2}\right)$ paired datasets. For $M=6$ (ImageBind's setting), that is 15 datasets. But with zero-shot transfer through an anchor modality, only $M-1=5$ datasets are needed (one per non-anchor modality paired with the anchor). The savings grow quadratically: for $M=10$ modalities, direct alignment needs 45 datasets versus only 9 for anchor-based alignment.

The mechanism is simple but profound. The image encoder creates a “bridge” in embedding space. An audio clip of a barking dog maps near the embedding of a photograph of a barking dog (because they were trained on image-audio pairs). The photograph of a barking dog maps near the embedding of the text “a dog barking” (because they were trained on image-text pairs). By transitivity, the audio clip maps near the text, enabling audio-to-text retrieval.

The practical implication is that the transfer loss grows as the sum (not the product) of the individual alignment errors, plus a geometric mean term. If both pairwise alignments are good (${\epsilon }_{mI},{\epsilon }_{In}\le 0.1$), the transfer loss is at most $0.1+0.1+2\sqrt{0.01}=0.4$, which implies a zero-shot similarity of at least $0.6$. This is tight enough for many retrieval applications.

Information-Theoretic Limits of Cross-Modal Retrieval

We now ask a deeper question: given a fixed embedding dimension $d$, what is the maximum possible retrieval performance? This is fundamentally an information-theoretic question, because the embedding acts as a lossy compression of the input, and the retrieval accuracy depends on how much information about the relevant document is preserved in the compressed representation.

The framework of rate-distortion theory [37][4] provides the right lens. We can view the embedding as a channel through which information about the query's semantic content is transmitted. The embedding dimension $d$ constrains the capacity of this channel.

Downstream Applications of Foundational Embeddings

The preceding sections have developed the mathematical foundations of multimodal embedding spaces and cross-modal retrieval. Now we turn to the question that ultimately justifies all this theory: what can you build with it? The answer, it turns out, is astonishingly broad. Foundational embeddings are not merely a better search engine. They are a new computational primitive, a universal similarity function that can be composed, filtered, combined, and reasoned over to create applications that would have been inconceivable five years ago.

We survey established applications, then venture into speculative territory where the most exciting possibilities live.

Creative Discovery and Artistic Exploration

The art world has always been organized by metadata: artist name, year, medium, movement, gallery. But the most interesting connections between artworks are often visual, connections that defy textual categorization. A Rothko color field and a Turner seascape share a luminous quality that no tag captures. A Hokusai wave and a Van Gogh night sky share a dynamic spiral energy that is felt rather than described.

Multimodal embeddings make these felt connections computable. By embedding artworks into a shared space, one can retrieve “paintings that feel like this one” without needing to articulate what “feel like” means. The embedding captures the visual gestalt, the holistic perceptual impression that resists decomposition into discrete attributes.

But the truly novel capability is cross-modal creative retrieval. Given a piece of music, find paintings that evoke the same mood. Given a poem, find photographs that capture the same atmosphere. Given a perfume description (“woody, with notes of cedar and a hint of vanilla”), find visual artworks that evoke the same sensory palette. These queries are absurd in a keyword-based system. In a multimodal embedding space, they are just nearest neighbor searches.

Accessibility Through Retrieval

For the 2.2 billion people worldwide with vision impairment, the visual world is mediated by descriptions. Alt text on web images, screen reader outputs, and audio descriptions of videos are critical accessibility tools, but they are expensive to produce manually and are missing for the vast majority of visual content.

Embedding-based retrieval offers a scalable alternative. Given an image that lacks a description, retrieve the most similar image in a database of described images, and transfer its description. The key insight is that descriptions do not need to be generated from scratch; they can be retrieved from a large pool of existing descriptions.

This approach has a crucial advantage over pure generation: the descriptions are grounded in human-written text, reducing the risk of hallucinated or misleading content. A visually impaired user receives a description that was written by a human for a visually similar image, not fabricated by a model that might confuse a cat for a dog.

The same principle extends to audio accessibility. A deaf user encountering a video can benefit from retrieved descriptions of similar audio tracks. If the video's soundtrack is similar (in embedding space) to a previously described audio clip, the existing description can be transferred: “sounds of traffic and distant sirens” provides environmental context that a visual-only experience cannot convey. Cross-modal retrieval thus becomes a bridge between sensory experiences, using the information available in one modality to augment the experience of another.

Content Moderation at Scale

Content moderation is one of the most consequential applications of embedding-based retrieval. Platforms hosting user-generated content must identify and remove harmful material, including known child sexual abuse material (CSAM), violent extremist content, copyright-infringing material, and spam, at a scale of billions of uploads per day.

The traditional approach uses perceptual hashing (e.g., PhotoDNA for images, MD5 hashes for exact duplicates), which matches against a database of known harmful content. Perceptual hashes are fast and deterministic but brittle: a simple crop, resize, or color shift can evade detection.

Embedding-based retrieval replaces the hash with a learned representation that is robust to superficial transformations. A harmful image and its cropped, resized, or color-shifted variant map to nearby points in embedding space, enabling detection even when the content has been deliberately modified to evade hash-based systems.

Scientific Discovery Across Modalities

Science communicates through a rich multimodal fabric: papers contain text, equations, figures, tables, and citations; datasets contain numbers, images, and metadata; presentations contain slides, speech, and video. Traditional scientific search operates almost exclusively on text (titles, abstracts, keywords), leaving the vast majority of scientific content unsearchable.

Multimodal embeddings open the door to searching the full fabric of scientific communication. Consider these scenarios:

• Figure-to-paper retrieval: A researcher has a figure from a conference talk but cannot remember which paper it came from. Embed the figure, search against a database of paper figures, retrieve the source paper.

• Text-to-figure retrieval: A researcher writing a survey wants to find all published figures that visualize “attention weights in transformer models.” A text query retrieves relevant figures from across the literature.

• Equation-to-paper retrieval: A researcher encounters an equation and wants to find its origin or related work. If equations are rendered as images, they can be embedded and matched against a database of equation images from published papers.

• Data-to-method retrieval: Given a dataset's characteristics (embedded as a structured description), retrieve papers that have achieved good results on similar datasets.

Multimodal Product Search

E-commerce was among the first industries to deploy multimodal embeddings at scale. The use case is intuitive: a customer sees a pair of shoes on the street, takes a photograph, and searches for similar products. Or a customer describes what they want in natural language: “a mid-century modern coffee table in walnut with tapered legs.” Or a customer provides an image of their living room and asks for furniture that would “go with” the existing decor.

Each of these queries leverages a different modality (image, text, or image-as-context), but all are resolved by the same mechanism: embed the query, search the product embedding index, return the nearest neighbors.

The commercial impact is substantial. Visual search increases conversion rates by 30–50% compared to text-only search for categories where visual appearance dominates the purchase decision (fashion, furniture, home decor). Multimodal search, combining text and image cues, performs even better because it captures both the visual gestalt and the specific attributes the customer cares about.

Cross-Modal Medical Retrieval

The medical domain presents a particularly compelling case for cross-modal retrieval because clinical decision-making inherently involves multiple modalities: radiology images (X-rays, CT scans, MRIs), clinical notes (free text), laboratory results (structured data), pathology slides (microscopy images), and patient histories (longitudinal text).

The privacy implications are severe. Medical embeddings must be designed so that patient-identifying information cannot be reconstructed from the embedding vector, even under adversarial attack. This connects to the broader field of privacy-preserving embeddings, where differential privacy or adversarial debiasing techniques are applied during embedding training to ensure that sensitive attributes are not encoded in the representation.

The formal tension between privacy and utility in medical embeddings is a special case of a fundamental information-theoretic tradeoff. The embedding must contain enough information to discriminate between different medical conditions (for retrieval to be useful) but not enough to identify individual patients (for privacy to be maintained). Formalizing this tradeoff and finding the Pareto frontier between privacy and retrieval accuracy is an active area of research with significant clinical implications.

Beyond the Horizon: Visionary Applications

The applications surveyed above are either deployed or under active development. We now venture into more speculative territory, exploring applications that become conceivable, even if not yet practical, once we take the premises of multimodal embedding spaces to their logical conclusion. These are not idle fantasies; each follows directly from capabilities that current embedding models either already possess or are close to possessing.

Emotional Retrieval.

Emotions are arguably the most important human information modality, yet they are absent from every information retrieval system ever built. What if we could embed emotional states? A user listens to a piece of music that makes them feel melancholy. They want to find a painting that evokes the same feeling, or a poem, or a scene from a film. This is not text search (“find sad paintings”) but something deeper: the query is not a text label of the emotion but the emotion itself, as evoked by a specific sensory experience.

The path to emotional retrieval lies in the observation that multimodal embeddings already capture some emotional information. CLIP embeddings of images rated as “serene” by human annotators cluster together in embedding space, separated from images rated as “anxious” or “joyful.” This is not because CLIP was trained on emotion labels; it is because the visual features that evoke serenity (soft colors, open spaces, horizontal compositions) are correlated with the text descriptions that appear alongside such images on the web. Emotional information is already latent in the embedding geometry.

The research challenge is to amplify this latent emotional signal. Fine-tuning on emotion-annotated datasets, incorporating physiological signals (heart rate, galvanic skin response) as additional modalities, or training on affective descriptions (“this image makes me feel calm”) rather than factual ones (“a sunset over the ocean”) could produce embeddings where the emotional dimension is a first-class citizen rather than an incidental byproduct.

Thought Retrieval.

If emotional retrieval seems speculative, thought retrieval borders on science fiction, but the boundary is narrower than one might expect. Recent work in neural decoding has demonstrated that fMRI signals recorded while a subject views an image can be decoded into a representation that, when projected into CLIP embedding space, correctly identifies which image the subject was viewing from a set of candidates. The brain activity is literally being embedded into the same space as images and text.

Extend this to retrieval: a subject thinks of a visual concept, a face, a landscape, an object. Their brain activity is recorded (via fMRI, EEG, or a future non-invasive neural interface), embedded into the multimodal space, and used as a query. The system retrieves images, sounds, or text that match the subject's mental imagery. This is not content-based retrieval in the traditional sense; it is intention-based retrieval, where the query is the user's thought itself.

Cross-Temporal Retrieval.

History is filled with recurrence. A medieval siege and a modern urban warfare scenario share tactical patterns. A 17th-century financial bubble and a 21st-century cryptocurrency crash follow similar dynamics. An Art Deco building facade and a modern minimalist building share geometric sensibilities that transcend their 80-year separation.

Cross-temporal retrieval uses embeddings to find these connections across time. Given a historical artifact (a photograph, a document, a map), retrieve its modern analog: the contemporary object, event, or concept that is most semantically similar. This requires embeddings that capture abstract patterns rather than surface features, embeddings that recognize the structural similarity between a Roman aqueduct and a modern pipeline, or between a medieval manuscript's illumination and a modern infographic's design language.

The educational potential is immense. Imagine a history student who can point their camera at a museum artifact and instantly see its modern parallels, not through manually curated links but through the emergent geometry of a well-trained embedding space.

Synesthetic Retrieval.

Synesthesia, the neurological phenomenon in which stimulation of one sense triggers an involuntary experience in another (seeing colors when hearing music, tasting shapes), has long fascinated artists and scientists. Multimodal embedding spaces make synesthetic associations computable.

What does the color blue sound like? In a well-trained multimodal embedding space, one can embed images of blue objects, retrieve the nearest audio clips, and listen to what the model “thinks” blue sounds like. What does a C-major chord look like? Embed the chord as audio, retrieve the nearest images. These queries exploit the cross-modal geometry of the embedding space to create synesthetic mappings that are data-driven rather than arbitrary.

Whether these computed synesthetic associations align with the experiences of human synesthetes is an open empirical question, one that connects computational embedding theory to the neuroscience of cross-modal perception.

Dream Indexing.

The most speculative application of all: indexing the content of dreams. As neural decoding technology advances from fMRI (expensive, slow, low temporal resolution) toward high-density EEG or optical neural interfaces (potentially portable and real-time), it becomes conceivable to record brain activity during REM sleep and embed the resulting signals into a multimodal space. The “dream embedding” would sit near images, sounds, and narratives that resemble the dream content. Upon waking, the dreamer could query their dream log: “Show me what I dreamed about last Tuesday” would return a collage of images, sounds, and text fragments that the embedding model associates with the recorded neural activity.

This application is decades away, if it is possible at all. But it illustrates the ultimate promise of multimodal embedding spaces: if any signal, from any source, can be embedded into the same space, then the space becomes a universal index of human experience.

Embedding-Mediated Translation.

A more near-term visionary application is embedding-mediated translation between modalities. Rather than training a separate generative model for each modality pair (text-to-image, image-to-audio, audio-to-video, etc.), one trains a single conditional generator that takes an embedding vector as input and produces output in a specified target modality. The embedding serves as the modality-agnostic representation of “what to generate,” and the generator handles the “how.”

This architecture reduces the $M^2$ translation problem to $M$ generation problems, each conditioned on the same embedding space. The embedding space serves as the interlingua, the intermediate representation through which all modality pairs communicate.

Open Problems and Research Frontiers

This conjecture, if true, would imply that the era of modality-specific retrieval is a temporary phase in the history of information systems. The destination is a single embedding space, a single index, and a single retrieval mechanism for all forms of human information. The mathematical foundations developed in this chapter, from contrastive alignment to rate-distortion bounds to modality gap analysis, are the tools that will either prove or disprove this conjecture, and that will guide the engineering of the universal retrieval systems of the future.

Generative Reranking

You have one thousand potentially relevant documents. A simple scorer, perhaps BM25 or a bi-encoder, ranked them in a fraction of a second. But a large language model can understand nuance, context, and intent in ways that a bag-of-words statistic or a dot product in embedding space simply cannot. Can the LLM rerank those documents more accurately? Yes, and this insight is transforming the entire retrieval pipeline.

The idea is disarmingly simple: take the candidate list produced by a cheap first-stage retriever, then use a more powerful (and more expensive) model to refine the ordering. The first-stage retriever provides recall: it finds documents that might be relevant. The reranker provides precision: it puts the truly relevant documents at the top. This two-stage pattern, sometimes called retrieve-then-rerank, has become the dominant architecture in modern information retrieval.

What makes recent work exciting is the realisation that generative language models, models trained to produce text rather than classify it, can serve as extraordinarily effective rerankers. The key distinction lies in how the model processes the query and document, and how many documents it considers at once. This distinction gives rise to four paradigms: pointwise, pairwise, listwise, and setwise reranking.

Pointwise Reranking

The simplest approach scores each document independently. Given a query $q$ and a candidate document $d_i$, the reranker produces a relevance score $s(q,d_i)$ without reference to any other document in the candidate set. The final ranking is obtained by sorting documents by their scores.

Cross-Encoders with BERT

The cross-encoder architecture, introduced in the context of natural language inference and adapted for retrieval, concatenates the query and document into a single input sequence and passes it through a Transformer encoder [83]. The [CLS] token representation is fed to a classification head that outputs a relevance score: (Cross Encoder)$$\begin{array}{ccc}& s(q,d)={𝐖}^{\top }\mathrm{BERT}{(\text{[𝙲𝙻𝚂]}q\text{[𝚂𝙴𝙿]}d)}_{\text{[𝙲𝙻𝚂]}}+b,& \text{<span class="tml-eqn"></span>}\end{array}$$ where $𝐖\in {\mathrm{ℝ}}^h$ is a learned projection and $b$ is a scalar bias. The critical advantage over bi-encoders is that the query and document tokens attend to each other throughout every Transformer layer, enabling fine-grained interaction modelling. The critical disadvantage is computational cost: a separate forward pass is required for every query-document pair.

monoT5: Sequence-to-Sequence Relevance Scoring

Nogueira et al. [84] proposed a remarkably elegant reformulation: instead of adding a classification head to an encoder, use a sequence-to-sequence model and ask it to generate a relevance judgment. Given query $q$ and document $d$, the input to T5 [85] is:

Query: $q$ Document: $d$ Relevant:

The model generates either “true” or “false” as output. The relevance score is derived from the probability assigned to the “true” token: (Monot5 Score)$$\begin{array}{ccc}& s(q,d)=P_{𝜽}(\text{“true”}|q,d).& \text{<span class="tml-eqn"></span>}\end{array}$$

This formulation is powerful for two reasons. First, it leverages the pre-trained knowledge encoded in T5's parameters: the model has already seen billions of tokens and developed a rich understanding of relevance, topicality, and semantic similarity. Second, it requires no architectural modifications; any sequence-to-sequence model can serve as a pointwise reranker with this prompt template.

Pairwise Reranking

Pointwise reranking treats each document in isolation. But relevance is often relative: document $d_A$ might be hard to judge on its own, but clearly more relevant than document $d_B$ when the two are compared side by side. Pairwise reranking exploits this observation.

duoT5: Pairwise Comparison via Generation

The duoT5 model extends the monoT5 idea to the pairwise setting. Given a query and two documents, the T5 model receives:

Query: $q$ Document0: $d_A$ Document1: $d_B$ More relevant:

The model generates either “Document0” or “Document1,” and the comparison score is: (Duot5 Score)$$\begin{array}{ccc}& c(q,d_A,d_B)=P_{𝜽}(\text{“Document0”}|q,d_A,d_B).& \text{<span class="tml-eqn"></span>}\end{array}$$

The principal drawback of pairwise reranking is computational. Ranking $n$ documents requires $\left(\genfrac{}{}{0px}{}{n}{2}\right)=O(n^2)$ pairwise comparisons in the worst case. Even with efficient sorting algorithms that require only $O(n\log n)$ comparisons, each comparison involves a full forward pass through the language model. For large candidate sets, this cost is prohibitive.

Listwise Reranking

The natural next step is to process the entire candidate list at once. Instead of scoring documents individually or comparing them in pairs, a listwise reranker receives the query and all $n$ candidate documents simultaneously and produces a permutation of the entire list.

LLM Permutation Generation

Hou et al. [24] demonstrated that large language models can directly generate permutations. The prompt presents the query and a numbered list of document passages, then asks the LLM to output the passage numbers in order of decreasing relevance. For example:

I will provide a query and a list of passages. Rank the passages based on their relevance to the query. Output the ranking as a list of passage numbers from most to least relevant.

Query: $q$

Passage [1]: $d_1$

Passage [2]: $d_2$

$⋮$

Passage [$n$]: $d_n$

Ranking:

The model generates a sequence such as “[3] > [1] > [5] > [2] > [4],” which is parsed into the permutation $\sigma =(3,1,5,2,4)$.

The appeal of listwise reranking is that the model can consider all documents simultaneously, making globally coherent comparisons. The challenge is context length: a list of even 20 moderately long passages may exceed the context window of many language models, and attention cost grows quadratically with sequence length.

Sliding Window Approaches

To handle lists longer than the context window allows, sliding window strategies partition the candidate list into overlapping windows and rerank each window independently. The results are then merged using a bubble-sort-like procedure: the top-ranked document from the current window is compared against documents in the next window, and the process repeats until the entire list has been processed.

The sliding window approach trades optimality for tractability. Each window reranking is a local operation, and relevant documents gradually “bubble up” through the list across multiple passes. In practice, two to three passes suffice for most applications, and the approach scales to lists of hundreds or thousands of documents.

Setwise Reranking: The Efficiency Frontier

Pointwise reranking requires $n$ LLM calls. Pairwise reranking requires $O(n\log n)$ calls at minimum and $O(n^2)$ in the worst case. Listwise reranking requires only a single call for the entire list, but is limited by context length. Setwise reranking, introduced by Zhuang et al. [23], occupies a sweet spot between these extremes.

The key insight is that the LLM processes only $k$ documents per call, keeping the context length manageable, while the selection procedure produces a globally meaningful ranking. The subset $S_t$ is typically chosen to include the top candidates from the remaining pool according to the first-stage retriever's scores.

FIRST: Fast Listwise Ranking via Single Tokens

Listwise reranking with LLMs typically requires the model to generate an entire permutation as a sequence of tokens. FIRST (Fast lIstwise Ranking via Single Token decoding) [86] observes that if we only need to identify the top document, we can do so by examining the logits over a single token position.

The idea is as follows. Present the LLM with the query and $k$ passages, each labelled with a single-character identifier (A, B, C, ). Then ask the model: “The most relevant passage is:” and examine the probability distribution over the next token. The passage whose identifier receives the highest probability is deemed most relevant: (First Score)$$\begin{array}{ccc}& d^{\ast }=d_j\text{where}j={\mathrm{arg\hspace{0.17em}max}}_{i\in \{1,\dots ,k\}}{P}_{𝜽}({\text{token}}_i|q,d_1,\dots ,d_k),& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${\text{token}}_i$ is the character identifier for passage $i$.

To rank all $n$ documents (not just the top one), FIRST combines single-token decoding with a selection procedure similar to the setwise approach: identify the best document in the current window, remove it, and repeat. Alternatively, the probability distribution over all $k$ identifiers can be used directly as scores, though this requires careful calibration.

RLCF: Learning to Rerank from Contrastive Feedback

The reranking methods described so far rely on the general capabilities of pre-trained language models. RLCF (Reinforcement Learning from Contrastive Feedback) [87] takes the next step: it fine-tunes the LLM reranker using reinforcement learning, where the reward signal comes from contrastive comparisons between the model's ranking and human preferences.

The training procedure alternates between generating rankings using the current policy ${\pi }_{𝜽}$ and updating the policy to maximise the expected contrastive reward. The contrastive structure of the reward encourages the model to focus on the relative ordering of documents rather than their absolute relevance scores, which aligns well with the evaluation metrics used in information retrieval (NDCG, MAP, MRR).

Formal Analysis: Computational Complexity

We now consolidate the computational analysis of all four reranking paradigms. Let $n$ denote the number of candidate documents, $L$ the average document length in tokens, $q$ the query length, and $C(s)$ the cost of a single LLM forward pass on a sequence of length $s$.

Empirical Comparison

Table 9 summarises the performance and efficiency characteristics of the major reranking approaches on the TREC Deep Learning benchmarks.

Method	Paradigm	NDCG@10	MRR@10	LLM Calls
BM25 (no reranking)	–	0.480	0.660	0
BERT cross-encoder	Pointwise	0.710	0.835	100
monoT5 (3B)	Pointwise	0.725	0.850	100
duoT5 (3B)	Pairwise	0.740	0.860	$\sim$660
LLM listwise (GPT-4)	Listwise	0.750	0.870	5–10
Setwise ($k=10$)	Setwise	0.738	0.855	10
FIRST	Listwise	0.735	0.852	5–10
RLCF-tuned	Listwise	0.755	0.875	5–10

Synthetic Data Generation for Retrieval

Training a neural retriever requires labelled data: pairs of queries and relevant documents, annotated by human judges. But human annotation is expensive, slow, and often inconsistent. The TREC Deep Learning track, one of the most widely used benchmarks in information retrieval, contains only a few thousand judged queries. The MS MARCO dataset, substantially larger, still represents a tiny fraction of the possible query-document pairs. And when you want to deploy a retriever in a new domain, medical literature, legal documents, internal corporate knowledge bases, the training data simply does not exist.

This is the bootstrapping problem: building a retriever for a new domain requires training data, but obtaining that training data requires either a retriever (to surface documents for annotation) or an enormous human labelling effort. The solution, it turns out, is to let the language models generate the training data themselves.

InPars: Query Generation for Domain Adaptation

Bonifacio et al. [88] proposed InPars (Inquisitive Parrots), a method that uses large language models to generate synthetic queries for documents in a target corpus. The procedure is elegant in its simplicity.

Given a document $d$ from the target corpus, prompt the LLM with a few examples of (document, query) pairs drawn from a seed dataset, followed by the target document. The LLM then generates a plausible query for which $d$ would be a relevant answer. The synthetic query-document pair $(q_{\text{syn}},d)$ becomes a positive training example for the retriever.

The synthetic training set is typically filtered to remove low-quality queries (e.g., those that are too short, too generic, or obviously unrelated to the document). A common filtering strategy uses the language model's own confidence: queries with low generation probability are discarded. The filtered set is then used to fine-tune a bi-encoder or cross-encoder retriever.

HyDE: Hypothetical Document Embeddings

If InPars generates synthetic queries for documents, HyDE (Hypothetical Document Embeddings) [89] does something conceptually reversed and strikingly beautiful: it generates synthetic documents for queries.

The core observation is this: queries and documents live in different linguistic spaces. A query is typically short, fragmentary, and ambiguous (“best treatment for lower back pain”). A relevant document is typically long, detailed, and self-contained (“Lumbar disc herniation is treated through a combination of physical therapy, anti-inflammatory medications, and in severe cases, surgical intervention”). Embedding a query and embedding a relevant document into the same space is precisely the challenge that makes dense retrieval hard.

HyDE bypasses this challenge entirely. Instead of embedding the query directly, it first uses an LLM to generate a hypothetical document: a passage that would plausibly answer the query. This hypothetical document, though potentially inaccurate in its specifics, captures the linguistic style, vocabulary, and topical focus of a relevant document. It is then embedded using a standard dense encoder, and the resulting embedding is used to retrieve real documents via nearest-neighbour search.

The beauty of HyDE is that it achieves zero-shot dense retrieval: it requires no training data whatsoever. The language model provides the domain knowledge (it knows what a relevant document would look like), and the dense encoder provides the embedding space (it maps the hypothetical document to the right neighbourhood in vector space). Neither component needs task-specific fine-tuning.

Promptagator: Few-Shot Retriever Training

Promptagator extends the InPars idea by incorporating a consistency-based filtering step. After generating synthetic queries, a retriever is trained on the synthetic data and used to re-retrieve documents for each synthetic query. If the original document $d$ is not retrieved in the top-$k$ results for its synthetic query $q_{\text{syn}}$, the pair is considered unreliable and discarded. This round-trip consistency filter significantly improves the quality of the training data.

The filtering is performed iteratively: generate synthetic data, train a retriever, filter the data using the trained retriever, retrain on the filtered data. Each iteration produces a cleaner training set and a stronger retriever.

Synthetic Hard Negatives

Training effective retrievers requires not only positive examples (relevant documents) but also informative negative examples (irrelevant documents that are hard to distinguish from relevant ones). Random negatives are easy to obtain but provide little training signal; the model quickly learns to reject obviously irrelevant documents and stops improving. Hard negatives, documents that are topically related but not truly relevant, force the model to make fine-grained distinctions.

Language models can generate synthetic hard negatives by modifying relevant documents in subtle ways: changing key facts, altering the time period, or shifting the focus slightly. For a query about “treatment options for Type 2 diabetes,” a synthetic hard negative might discuss treatment options for Type 1 diabetes, or discuss Type 2 diabetes diagnostics rather than treatment.

Risks and Limitations

Synthetic data generation for retrieval is powerful but fraught with risks. Three deserve particular attention.

Hallucination propagation.

When an LLM generates a synthetic query or document, it may hallucinate details that do not appear in the source document. If the retriever is trained to match these hallucinated details, it learns to retrieve documents based on fabricated content. This is particularly dangerous in high-stakes domains (medical, legal) where factual accuracy is paramount.

Distribution shift.

Synthetic data reflects the distribution of the language model, not the distribution of real user queries. LLM-generated queries tend to be more grammatical, more complete, and more informative than real user queries, which are often telegraphic, misspelled, and ambiguous. A retriever trained exclusively on synthetic data may underperform on real queries that fall outside the synthetic distribution.

Evaluation contamination.

This is perhaps the most insidious risk.

Generative Retrieval

What if the search index is the model? Every retrieval system we have encountered so far maintains an external data structure, an inverted index, a vector store, a graph, to hold the corpus. The model (be it a sparse scorer, a dense encoder, or a cross-encoder) queries this external structure to find relevant documents. Generative retrieval eliminates the external index entirely. The model memorises the corpus in its parameters and, given a query, directly generates the identifier of the relevant document.

This is a radical departure from the retrieve-then-read paradigm. There is no index to build, no approximate nearest-neighbour search to perform, no document embeddings to store. The entire retrieval pipeline collapses into a single autoregressive generation step: query in, document identifier out.

The idea is audacious, elegant, and deeply controversial. Let us examine it carefully.

DSI: The Differentiable Search Index

Tay et al. [90] introduced the Differentiable Search Index (DSI), the seminal work in generative retrieval. The architecture is a standard sequence-to-sequence Transformer (T5) that is trained to perform two tasks:

1. Memorisation: given a document $d$, generate its identifier $\mathrm{DocID}(d)$. This phase forces the model to encode the content of every document in the corpus into its parameters.

2. Retrieval: given a query $q$, generate the $\mathrm{DocID}$ of the most relevant document. This phase teaches the model to map queries to the correct identifiers.

DocID Design

The design of the document identifier is the most critical architectural decision in generative retrieval. Tay et al. explored three approaches:

Numeric IDs.

Each document receives an arbitrary integer, represented as a sequence of digit tokens. Document 42,857 becomes the token sequence “4”, “2”, “8”, “5”, “7”. This is the simplest approach, but it forces the model to memorise an arbitrary mapping between document content and meaningless numbers, a task that is fundamentally at odds with the language model's strengths.

String IDs.

Each document receives a human-readable identifier, such as a title or URL. The document about tidal forces might receive the ID “Tidal_Force_Wikipedia”. This is more natural for the language model, since the identifier is semantically related to the content. However, titles may be ambiguous (multiple documents with similar titles) or too long to generate efficiently.

Semantic hierarchical IDs.

Documents are organised into a hierarchy via clustering, and each document's ID is its path through the hierarchy. For example, a corpus might be divided into 100 top-level clusters, each subdivided into 100 sub-clusters, and each sub-cluster containing individual documents. Document 42,857 might receive the hierarchical ID “37-15-42”, meaning cluster 37, sub-cluster 15, position 42. If the clustering is semantic (based on document embeddings), then similar documents have similar prefixes, and the model can learn to navigate the hierarchy by progressively narrowing the topic.

Training

DSI training proceeds in two phases. In the memorisation phase, the model is trained on input-output pairs of the form $(d,\mathrm{DocID}(d))$, where the input is the full document text (or a representative passage) and the output is the document identifier. The training objective is the standard sequence-to-sequence cross-entropy loss: (DSI Memorisation)$$\begin{array}{ccc}& {ℒ}_{\text{mem}}=-\sum _{d\in 𝒟}\log P_{𝜽}(\mathrm{DocID}(d)|d).& \text{<span class="tml-eqn"></span>}\end{array}$$

In the retrieval phase, the model is fine-tuned on query-document identifier pairs $(q,\mathrm{DocID}(d^{+}))$, where $d^{+}$ is a document known to be relevant to $q$: (DSI Retrieval)$$\begin{array}{ccc}& {ℒ}_{\text{ret}}=-\sum _{(q,d^{+})\in 𝒯}\log P_{𝜽}(\mathrm{DocID}(d^{+})|q).& \text{<span class="tml-eqn"></span>}\end{array}$$

The total loss combines both phases: (DSI Total LOSS)$$\begin{array}{ccc}& {ℒ}_{\text{DSI}}={\lambda }_{\text{mem}}{ℒ}_{\text{mem}}+{\lambda }_{\text{ret}}{ℒ}_{\text{ret}},& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${\lambda }_{\text{mem}}$ and ${\lambda }_{\text{ret}}$ are balancing coefficients.

Semantic IDs via Hierarchical Clustering

The design of meaningful document identifiers has emerged as a research direction in its own right. The most promising approach constructs identifiers through hierarchical clustering of document embeddings, producing IDs that reflect semantic structure.

RQ-VAE for Identifier Learning

Residual quantisation with a variational autoencoder (RQ-VAE) provides an elegant framework for learning semantic identifiers. The procedure works as follows:

1. Encode each document $d$ into a dense embedding $𝒛=f_{𝜽}(d)\in {\mathrm{ℝ}}^D$.

2. Apply residual quantisation with $L$ levels. At level $\ell$, find the nearest codeword $c_{\ell }$ in codebook ${𝒞}_{\ell }=\{c_{\ell ,1},\dots ,c_{\ell ,K}\}$: (Rqvae Quantise)$$\begin{array}{ccc}& c_{\ell }={\mathrm{arg\hspace{0.17em}min}}_{c\in {𝒞}_{\ell }}{\Vert {𝒓}_{\ell -1}-c\Vert }^2,& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${𝒓}_0=𝒛$ and ${𝒓}_{\ell }={𝒓}_{\ell -1}-c_{\ell }$ is the residual after quantisation.

3. The semantic ID of document $d$ is the sequence of codebook indices: $\mathrm{SemanticID}(d)=(k_1,k_2,\dots ,k_L)$, where $c_{\ell }=c_{\ell ,k_{\ell }}$.

The connection to product quantisation, a well-established technique in approximate nearest-neighbour search, is direct. Product quantisation decomposes the embedding space into subspaces and quantises each subspace independently; residual quantisation applies quantisation sequentially to the residual error. Both produce compact codes that approximate the original embeddings, but residual quantisation produces naturally hierarchical codes: the first codeword captures coarse structure, and subsequent codewords refine it.

TIGER: Generative Retrieval for Recommendation

Rajput et al. [91] extended generative retrieval to the recommendation setting with TIGER (Transformer Index for GEnerative Recommenders). In recommendation, the “corpus” is a catalogue of items (products, movies, songs), and the “query” is a user's interaction history.

TIGER assigns each item a semantic ID learned via RQ-VAE on item content embeddings (e.g., text descriptions, images, or audio features). The recommendation model, a Transformer, takes the user's interaction history (a sequence of semantic IDs of previously consumed items) and autoregressively generates the semantic ID of the next item to recommend: (Tiger)$$\begin{array}{ccc}& P(i_{\text{next}}|i_1,i_2,\dots ,i_t;𝜽)=\prod _{\ell =1}^L{P}_{𝜽}(k_{\ell }^{\text{next}}|k_1^{\text{next}},\dots ,k_{\ell -1}^{\text{next}},i_1,\dots ,i_t),& \text{<span class="tml-eqn"></span>}\end{array}$$ where $(k_1^{\text{next}},\dots ,k_L^{\text{next}})$ is the semantic ID of the next item.

IDGenRec: LLM-RecSys Alignment

Tan et al. [92] proposed IDGenRec, which addresses a key limitation of previous generative recommendation models: the identifiers were designed independently of the language model. IDGenRec aligns the identifier design with the LLM's tokeniser and pre-trained knowledge.

Instead of numeric or RQ-VAE-based codes, IDGenRec generates textual IDs that consist of semantically meaningful word tokens from the LLM's vocabulary. Each item receives a textual identifier that is both unique and descriptive, e.g., “vintage leather messenger bag” rather than “42-15-7”. The LLM can leverage its pre-trained understanding of these words to make recommendations without needing to learn arbitrary code mappings from scratch.

CorpusLM: Unified Retrieval and Knowledge

Li et al. [93] proposed CorpusLM, a unified framework that trains a single language model to perform both generative retrieval and knowledge-intensive language tasks (question answering, fact verification, dialogue). The key observation is that retrieval and knowledge tasks share a common requirement: the model must internalise the corpus and produce relevant information on demand.

CorpusLM trains on a mixture of tasks:

• Retrieval: query $\to$ DocID, as in DSI.

• Generation: query $\to$ answer, using knowledge stored in the parameters.

• Memorisation: document $\to$ DocID, to maintain corpus coverage.

The multi-task training enables knowledge transfer: the retrieval task forces the model to organise its knowledge by document, while the generation task forces it to extract and synthesise information across documents.

The training objective combines all three tasks: (Corpuslm LOSS)$$\begin{array}{ccc}& {ℒ}_{\text{CorpusLM}}={\lambda }_r{ℒ}_{\text{ret}}+{\lambda }_g{ℒ}_{\text{gen}}+{\lambda }_m{ℒ}_{\text{mem}},& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${ℒ}_{\text{ret}}$ is the retrieval loss (cross-entropy over DocIDs given queries), ${ℒ}_{\text{gen}}$ is the generation loss (cross-entropy over answer tokens given queries), and ${ℒ}_{\text{mem}}$ is the memorisation loss (cross-entropy over DocIDs given documents). The balancing coefficients ${\lambda }_r$, ${\lambda }_g$, ${\lambda }_m$ control the relative importance of each task.

Limitations and Open Questions

Generative retrieval is intellectually compelling but faces fundamental challenges that temper its practical applicability.

Scalability.

A Transformer must memorise the entire corpus in its parameters. As the corpus grows, the model must grow proportionally to maintain retrieval quality. The original DSI experiments used 320,000 passages; web-scale corpora contain billions. Can a model with a fixed parameter count memorise billions of documents? Information theory suggests not.

Corpus updates.

Adding a new document to a traditional index takes milliseconds. Adding a new document to a generative retrieval model requires retraining or at minimum fine-tuning the entire model, a process that takes hours or days. For corpora that change frequently (news, social media, e-commerce), this is a severe limitation.

Hallucinated identifiers.

An autoregressive model can generate identifier sequences that do not correspond to any document in the corpus. With numeric IDs, the model might generate “42857” when no document has that ID. With semantic hierarchical IDs, the model might navigate to a leaf node that is empty. Constrained decoding can mitigate this problem by restricting generation to valid prefixes at each step (see Lemma 8), but it adds computational overhead and complexity.

Lack of explainability.

When a traditional retriever returns a document, the relevance signal is transparent: a BM25 score highlights matching terms, a dense retriever provides an embedding similarity, and a cross-encoder produces an interpretable relevance probability. When a generative retrieval model produces a DocID, the reasons for selection are opaque. The model navigated its parametric memory through a series of autoregressive token predictions, and there is no straightforward way to explain why this particular document was selected over others.

Theoretical Analysis

We now turn to the fundamental question: how many parameters does a generative retrieval model need to memorise a corpus of $N$ documents?

The Road Ahead

Generative retrieval represents a fascinating intellectual direction, but its practical impact remains uncertain. The approach excels in settings where the corpus is small, relatively static, and the model has ample capacity. For large-scale, dynamic corpora, external-index systems remain the pragmatic choice.

The most promising near-term application may be in hybrid systems that combine generative retrieval with traditional methods. A generative model can serve as one signal among many in a retrieve-then-rerank pipeline, contributing its unique ability to capture holistic query-document relationships that are difficult to express as sparse or dense similarity scores.

Retrieval-Augmented Generation – Foundations

Large language models are brilliant but forgetful. They can write sonnets, summarise legal briefs, and explain quantum mechanics to a five-year-old, yet they cannot reliably tell you who won last week's football match. They hallucinate references that do not exist. They confidently state facts that were true in 2021 but have since changed. They have no mechanism for updating their knowledge without expensive retraining. The root cause is architectural: all of the model's knowledge is compressed into its parameters during pre-training, and those parameters are frozen at inference time.

Retrieval-augmented generation (RAG) is the solution. The idea is disarmingly simple: give the language model a library card. Instead of requiring the model to memorise every fact it might ever need, equip it with a retrieval system that can look up relevant information at inference time and feed it into the generation process. The model still generates fluent language; it simply does so while consulting an external knowledge source.

This section traces the development of RAG from its original formulation [25] through modern extensions including RETRO [26], Fusion-in-Decoder [27], and Self-RAG [28]. We formalise the key architectures, analyse their training objectives, and examine the fundamental design decisions that determine when and how to retrieve.

The Original RAG Framework

The retrieval-augmented generation framework was introduced by Lewis et al. in 2020 [25]. The paper proposed a general-purpose architecture that combines a pre-trained retrieval module with a pre-trained sequence-to-sequence generator, training them jointly end-to-end.

Architecture

The RAG architecture comprises three components:

1. A parametric retriever $p_{\eta }(𝒛|𝒙)$, where $𝒙$ is the input query and $𝒛$ is a retrieved document. Lewis et al. used the Dense Passage Retriever (DPR), which encodes both queries and documents into dense vectors and retrieves by maximum inner product search: (DPR Score)$$\begin{array}{ccc}& p_{\eta }(𝒛|𝒙)\propto \exp (⟨E_q(𝒙),E_d(𝒛)⟩),& \text{<span class="tml-eqn"></span>}\end{array}$$ where $E_q$ and $E_d$ are BERT-based encoders for queries and documents respectively.

2. A parametric generator $p_{𝜽}(y_i|𝒙,𝒛,y_{1:i-1})$, instantiated as a BART model that takes as input the concatenation of the query $𝒙$ and the retrieved document $𝒛$, and generates an output sequence $𝒚=(y_1,\dots ,y_N)$ autoregressively.

3. A non-parametric knowledge index $𝒵$, a dense index over a large corpus (e.g., Wikipedia) that supports fast approximate nearest-neighbour search using MIPS (Maximum Inner Product Search).

The elegance of this design is the separation of concerns. The retriever is responsible for finding relevant information; the generator is responsible for using it. The knowledge index serves as a non-parametric memory that can be updated without retraining: to incorporate new information, one simply adds new documents to the index.

RAG-Sequence vs. RAG-Token

Lewis et al. proposed two variants that differ in how the marginalisation over retrieved documents is performed.

RAG-Sequence.

In RAG-Sequence, a single document is retrieved per output sequence. The model generates the entire output conditioned on that one document, and the final probability is a mixture over the top-$k$ retrieved documents: (RAG Sequence)$$\begin{array}{ccc}& P_{\text{RAG-Seq}}(𝒚|𝒙)=\sum _{𝒛\in \mathrm{t}\mathrm{o}\mathrm{p}\text{-}\mathrm{k}}{p}_{\eta }(𝒛|𝒙)\prod _{i=1}^N{p}_{𝜽}(y_i|𝒙,𝒛,y_{1:i-1}).& \text{<span class="tml-eqn"></span>}\end{array}$$ Each document provides a coherent context for the entire generation. This is appropriate when the answer can be found in a single passage.

RAG-Token.

In RAG-Token, the marginalisation occurs at every output token. Different tokens in the output sequence can attend to different retrieved documents: (RAG Token)$$\begin{array}{ccc}& P_{\text{RAG-Tok}}(𝒚|𝒙)=\prod _{i=1}^N\sum _{𝒛\in \mathrm{t}\mathrm{o}\mathrm{p}\text{-}\mathrm{k}}{p}_{\eta }(𝒛|𝒙)p_{𝜽}(y_i|𝒙,𝒛,y_{1:i-1}).& \text{<span class="tml-eqn"></span>}\end{array}$$ This is more expressive: the model can synthesise information from multiple documents within a single output, drawing one fact from document ${𝒛}_1$ and another from ${𝒛}_3$. However, it requires $k$ forward passes through the generator at each token position, making it more expensive.

End-to-End Training

A distinctive feature of the original RAG framework is that the retriever and generator are trained jointly. The training objective maximises the marginal log-likelihood of the output given the input: (RAG Training)$$\begin{array}{ccc}& ℒ(\eta ,𝜽)=-\sum _{(𝒙,𝒚)\in 𝒟}\log P(𝒚|𝒙)=-\sum _{(𝒙,𝒚)\in 𝒟}\log \sum _{𝒛\in \mathrm{t}\mathrm{o}\mathrm{p}\text{-}\mathrm{k}(𝒙)}{p}_{\eta }(𝒛|𝒙)p_{𝜽}(𝒚|𝒙,𝒛).& \text{<span class="tml-eqn"></span>}\end{array}$$

The gradient flows through both the generator and the retriever. The generator learns which parts of the retrieved document are useful; the retriever learns which documents lead to correct outputs. This creates a virtuous cycle: better retrieval leads to better generation, which provides better training signal for retrieval.

In practice, the document encoder $E_d$ in the retriever is kept frozen during training (updating it would require re-indexing the entire corpus at each training step), while the query encoder $E_q$ and the generator are updated. This asymmetric update strategy is a pragmatic compromise: the query encoder learns to produce queries that match the document representations in the existing index.

RETRO: Retrieval from Trillions of Tokens

The original RAG framework retrieves documents at the sequence level: given a query, retrieve $k$ passages and condition generation on them. RETRO (Retrieval-Enhanced Transformer), introduced by Borgeaud et al. [26], takes a fundamentally different approach. It retrieves at the chunk level, integrating retrieval directly into the transformer architecture at multiple points during generation.

Chunked Cross-Attention

The key architectural innovation in RETRO is chunked cross-attention (CCA). The input sequence is split into chunks of $m$ tokens each. For each chunk $C_u$ (the $u$-th chunk of the input), RETRO retrieves the top-$k$ nearest neighbours from a database of 2 trillion tokens. These neighbours are then integrated into the model through a cross-attention mechanism that operates at specific layers.

Formally, let ${𝐇}_u^{(\ell )}\in {\mathrm{ℝ}}^{m\times d}$ be the hidden states of chunk $u$ at layer $\ell$, and let ${𝐄}_u=[{𝒆}_u^{(1)};\dots ;{𝒆}_u^{(k)}]\in {\mathrm{ℝ}}^{k{m}^{\prime }\times d}$ be the encoded representations of the $k$ retrieved neighbours for chunk $u$, each of length $m^{\prime }$. The chunked cross-attention computes: (Retro CCA)$$\begin{array}{cccc}& \mathrm{CCA}({𝐇}_u^{(\ell )},{𝐄}_u)& =\mathrm{softmax}\left(\frac{{𝐇}_u^{(\ell )}{𝐖}_Q{({𝐄}_u{𝐖}_K)}^{\top }}{\sqrt{d_k}}\right){𝐄}_u{𝐖}_V,& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${𝐖}_Q,{𝐖}_K,{𝐖}_V$ are the standard query, key, and value projection matrices. The CCA output is added to the residual stream at designated “retrieval layers” (typically every third layer).

The critical advantage is efficiency. Because retrieval operates at the chunk level, the cross-attention cost scales with the chunk size $m$ rather than the full sequence length. Moreover, the retrieved neighbours are encoded by a frozen BERT encoder, avoiding the need to backpropagate through the retrieval index.

Scaling Laws: Retrieval as a Substitute for Parameters

Perhaps the most striking finding from the RETRO paper is that retrieval can substitute for model parameters. A 7.5-billion-parameter RETRO model with access to a 2-trillion-token retrieval database achieves performance comparable to models with 25 times more parameters (around 175 billion, GPT-3 scale) without retrieval on language modelling benchmarks. This suggests a new dimension in the scaling laws for language models: instead of only scaling parameters and training data, one can also scale the retrieval database.

Fusion-in-Decoder

Fusion-in-Decoder (FiD), proposed by Izacard and Grave [27], offers an elegant solution to a practical problem: how to condition generation on a large number of retrieved passages without the quadratic attention cost of concatenating them all into a single input sequence.

The key idea is simple but effective. Each retrieved passage ${𝒛}_j$ is independently concatenated with the query $𝒙$ and encoded by the encoder: (FID Encode)$$\begin{array}{ccc}& {𝒉}_j=\mathrm{Encoder}([𝒙;{𝒛}_j]),j=1,\dots ,k.& \text{<span class="tml-eqn"></span>}\end{array}$$ The encoded representations ${𝒉}_1,\dots ,{𝒉}_k$ are then concatenated and fed to the decoder, which attends over all of them jointly: (FID Decode)$$\begin{array}{ccc}& P(𝒚|𝒙,{𝒛}_{1:k})=\mathrm{Decoder}([{𝒉}_1;{𝒉}_2;\dots ;{𝒉}_k]).& \text{<span class="tml-eqn"></span>}\end{array}$$

The advantage is computational. The encoder processes each passage independently, so its cost scales linearly with $k$. Self-attention within each encoded passage is quadratic in the passage length, but since each passage is short (typically 100 tokens), the total encoder cost is $O(k\cdot n^2)$ where $n$ is the passage length. This is far cheaper than encoding a single concatenated sequence of length $kn$, which would cost $O(k^2{n}^2)$.

FiD achieves state-of-the-art results on open-domain QA benchmarks (Natural Questions, TriviaQA) by leveraging up to 100 retrieved passages, a number that would be infeasible with naive concatenation. The decoder learns to cross-attend selectively, focusing on the most relevant passages and ignoring irrelevant ones.

Adaptive Retrieval: Self-RAG

The systems described so far retrieve for every query. But retrieval is not always necessary. If you ask a language model “What is 2 + 2?” it does not need to consult Wikipedia. If you ask it to write a haiku about autumn, retrieval might actually hurt by introducing irrelevant context that distracts the generator. The question is: can the model learn when to retrieve?

Self-RAG, introduced by Asai et al. [28], answers this question affirmatively. The model is trained to generate special reflection tokens that control the retrieval process and evaluate the quality of the generated output.

Reflection Tokens

Self-RAG introduces four types of reflection tokens:

1. [Retrieve]: Should the model retrieve? Values: yes, no, continue. The model generates this token before each segment. If the value is yes, retrieval is performed; if no, the model generates from its parametric knowledge alone.

2. [IsRel]: Is the retrieved passage relevant to the query? Values: relevant, irrelevant. Generated after retrieval, this token filters out passages that the retriever selected but that the generator judges as unhelpful.

3. [IsSup]: Is the generation supported by the retrieved passage? Values: fully_supported, partially_supported, no_support. This token enforces faithfulness: the model must assess whether its own output is grounded in the evidence.

4. [IsUse]: Is the overall response useful? Values: rating from 1 to 5. This final reflection token provides an overall quality judgement.

The self-critique mechanism is the defining feature of Self-RAG. Rather than blindly incorporating retrieved information, the model reflects on the quality of retrieval and generation at each step. This transforms the model from a passive consumer of retrieved information into an active critic of its own output.

Training Self-RAG

Self-RAG is trained in two phases. First, a critic model is trained to generate reflection tokens given (query, passage, generation) triples. The critic is trained on data annotated by GPT-4, which labels each triple with the appropriate reflection token values. Second, the generator is trained to produce both the output text and the reflection tokens, using the critic's labels as supervision.

The training objective combines the standard language modelling loss with a reflection token prediction loss: (Selfrag LOSS)$$\begin{array}{ccc}& {ℒ}_{\text{Self-RAG}}={ℒ}_{\text{LM}}(𝒚|𝒙,𝒛)+\lambda \sum _{r\in {ℛ}_{\text{refl}}}{ℒ}_{\text{CE}}(r|𝒙,𝒛,𝒚),& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${ℛ}_{\text{refl}}=\{$[Retrieve], [IsRel], [IsSup], [IsUse]$\}$ and ${ℒ}_{\text{CE}}$ is the cross-entropy loss for each reflection token. The hyperparameter $\lambda$ controls the weight of the reflection loss.

The Retrieve-Then-Read vs. Generate-Then-Retrieve Debate

The architectures discussed so far follow the retrieve-then-read paradigm: first retrieve relevant documents, then generate an output conditioned on them. An alternative paradigm inverts the order: generate-then-retrieve [94]. The model first generates a preliminary answer or a set of search queries, then retrieves documents to verify, refine, or extend the initial generation.

The generate-then-retrieve approach has several advantages. First, the model's initial generation can serve as a better query than the raw input, because the model can reformulate the question, expand acronyms, and resolve ambiguities. Second, the retrieved documents can serve as a verification mechanism: if the initial generation is consistent with the retrieved evidence, confidence increases; if it contradicts the evidence, the model can revise.

A formal comparison clarifies the trade-offs. Let $𝒙$ be the input, $ℛ$ the retriever, and $𝒢$ the generator.

Retrieve-then-read.

(Retrieve THEN READ)$$\begin{array}{ccc}& \widehat{𝒚}=𝒢(𝒙,ℛ(𝒙)).& \text{<span class="tml-eqn"></span>}\end{array}$$ The retrieval query is $𝒙$ itself. If $𝒙$ is ambiguous, under-specified, or uses different terminology than the documents in the index, retrieval may fail.

Generate-then-retrieve.

(Generate THEN Retrieve)$$\begin{array}{ccc}& \widetilde{𝒚}=𝒢(𝒙),\widehat{𝒚}=𝒢(𝒙,ℛ(\widetilde{𝒚})).& \text{<span class="tml-eqn"></span>}\end{array}$$ The retrieval query is the model's preliminary generation $\widetilde{𝒚}$, which may be more informative than the raw input. However, this requires two generator passes and one retrieval call, increasing latency.

Iterative retrieval-generation.

(Iterative RAG)$$\begin{array}{ccc}& {𝒚}^{(0)}=𝒙,{𝒚}^{(t)}=𝒢({𝒚}^{(t-1)},ℛ({𝒚}^{(t-1)})),t=1,\dots ,T.& \text{<span class="tml-eqn"></span>}\end{array}$$ The model iteratively refines its output, using each intermediate result as a retrieval query for the next round. This is the most expressive but also the most expensive approach, requiring $T$ generator passes and $T$ retrieval calls.

RAG for Grounding and Verification

Large language models generate text that is fluent, grammatical, and often convincing. The problem is that fluency and truth are orthogonal properties. A model can produce a paragraph that reads beautifully and is entirely false. This is the hallucination problem, and it represents one of the most significant obstacles to deploying language models in high-stakes domains such as medicine, law, and finance.

Retrieval offers a path toward solving this problem. If every claim in the model's output can be traced to a specific source document, then the output is grounded: it is connected to verifiable evidence rather than floating free in the space of plausible-sounding text. This section examines how RAG enables grounding, how to evaluate it, and how to extend it to multi-step reasoning and formal verification.

Attribution and Faithfulness

Two related but distinct concepts govern the quality of grounded generation.

The distinction between attribution and faithfulness, on one hand, and correctness, on the other, is crucial. A claim can be faithfully attributed to a source and still be wrong, if the source itself contains an error. Attribution tells us where the information came from; it does not guarantee that the source is reliable.

RAG Evaluation Frameworks

Evaluating a RAG system requires assessing multiple dimensions simultaneously. A system that retrieves relevant documents but generates unfaithful answers is dangerous. A system that generates faithful answers to the wrong question is useless. The community has developed multi-dimensional evaluation frameworks to capture these distinctions.

RAGAS: Retrieval-Augmented Generation Assessment

The RAGAS framework [95] decomposes RAG quality into three orthogonal dimensions:

1. Context relevance: Are the retrieved passages relevant to the query? This measures retrieval quality independently of generation. Let $𝒙$ be the query and ${𝒵}_k=\{{𝒛}_1,\dots ,{𝒛}_k\}$ be the retrieved passages. Context relevance is: (Context Relevance)$$\begin{array}{ccc}& \mathrm{CR}(𝒙,{𝒵}_k)=\frac{1}{k}\sum _{j=1}^k\mathrm{𝟙}\left[{𝒛}_j\text{ is relevant to }𝒙\right].& \text{<span class="tml-eqn"></span>}\end{array}$$

2. Faithfulness: Does the generated answer accurately reflect the retrieved context? This measures whether the generator introduces information not supported by the retrieval. Let $𝒚$ be the generated answer decomposed into claims $\{c_1,\dots ,c_n\}$. Faithfulness is: (Faithfulness Score)$$\begin{array}{ccc}& \mathrm{Faith}(𝒚,{𝒵}_k)=\frac{1}{n}\sum _{i=1}^n\mathrm{𝟙}[\exists {𝒛}_j\in {𝒵}_k:\mathrm{NLI}({𝒛}_j,c_i)=\text{entail}].& \text{<span class="tml-eqn"></span>}\end{array}$$

3. Answer relevance: Does the generated answer actually address the query? A faithful answer to the wrong question scores high on faithfulness but low on answer relevance. Answer relevance is measured by generating synthetic questions from the answer and computing their semantic similarity to the original query: (Answer Relevance)$$\begin{array}{ccc}& \mathrm{AR}(𝒙,𝒚)=\frac{1}{|𝒬|}\sum _{{𝒙}^{\prime }\in 𝒬}\mathrm{sim}(𝒙,{𝒙}^{\prime }),& \text{<span class="tml-eqn"></span>}\end{array}$$ where $𝒬$ is the set of synthetic questions generated from $𝒚$ and $\mathrm{sim}$ is a semantic similarity function.

Retrieval-Specific Metrics

Beyond the RAGAS dimensions, several metrics target retrieval quality directly [96]:

• Precision@$k$: The fraction of top-$k$ retrieved passages that are relevant: (Precision AT K)$$\begin{array}{ccc}& \mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}@k=\frac{|\{{𝒛}_j\in {𝒵}_k:{𝒛}_j\text{ is relevant}\}|}{k}.& \text{<span class="tml-eqn"></span>}\end{array}$$

• Recall@$k$: The fraction of all relevant passages that appear in the top-$k$: (Recall AT K)$$\begin{array}{ccc}& \mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}@k=\frac{|\{{𝒛}_j\in {𝒵}_k:{𝒛}_j\text{ is relevant}\}|}{|\{𝒛\in 𝒵:𝒛\text{ is relevant}\}|}.& \text{<span class="tml-eqn"></span>}\end{array}$$

• Attribution accuracy: The fraction of generated claims for which the system can identify a supporting source passage. This differs from faithfulness in that attribution requires an explicit link between claim and source, not merely entailment.

Verification with Retrieved Facts

Grounding takes on special significance in formal domains where correctness can be mechanically checked. Consider theorem proving: a domain where every claim must be logically deduced from axioms and previously established results.

Theorem Proving with Retrieved Lemmas

A particularly compelling application of RAG is automated theorem proving, where the model generates proof steps and retrieves relevant lemmas from a mathematical library to support each step. This connects directly to the formal verification tools discussed in the evolutionary AI chapter, particularly the Lean theorem prover [97].

The proof verification pipeline operates as follows:

1. Conjecture. The system is given a mathematical statement to prove.

2. Generate candidate proof steps. A language model proposes a sequence of proof tactics, each of which transforms the current proof state.

3. Retrieve related theorems. For each proposed tactic, the system retrieves potentially relevant lemmas, definitions, and previously proven theorems from the mathematical library (e.g., Lean's mathlib).

4. Verify consistency. Each proof step is checked by the formal verifier (Lean's type checker) to ensure it is logically valid. Retrieved lemmas that the proof step depends on must themselves be verified theorems.

5. Iterate. If a step fails verification, the model backtracks and generates alternative steps, potentially with new retrieval queries informed by the failure.

This pipeline was spectacularly demonstrated by AlphaProof [98], which solved International Mathematical Olympiad problems by combining a language model with the Lean theorem prover. The retrieval component was essential: the model needed access to a vast library of mathematical facts to construct valid proofs.

Grounding in Multi-Step Reasoning

Many real-world queries require multi-step reasoning. The question “Did the architect of the Eiffel Tower also design the Statue of Liberty's internal structure?” requires the model to (1) identify the architect of the Eiffel Tower (Gustave Eiffel), (2) identify the designer of the Statue of Liberty's internal structure (also Gustave Eiffel), and (3) determine that they are the same person. Each step may require its own retrieval.

Two approaches have emerged for combining retrieval with chain-of-thought reasoning:

Retrieve once, reason many.

Retrieve all potentially relevant documents at the outset, then perform multi-step reasoning over the retrieved set. This is efficient (only one retrieval call) but may miss documents that become relevant only after intermediate reasoning steps.

Interleaved retrieval and generation.

Retrieve at each reasoning step, using the intermediate results to formulate new retrieval queries. This is more expressive: the model can discover that it needs information about Gustave Eiffel only after the first retrieval reveals him as the Eiffel Tower's architect.

RAG Across Modalities

The RAG paradigm was born in the world of text, but information does not respect modality boundaries. A doctor diagnosing a patient may need to retrieve medical images alongside journal articles. An engineer troubleshooting a machine may need to retrieve schematic diagrams, maintenance logs in text, and even audio recordings of the machine's operation. A student writing a report may need to retrieve figures, tables, and charts from papers.

Extending RAG to multiple modalities introduces new challenges. How do you index and retrieve images alongside text? How do you fuse information from a retrieved audio clip with a text prompt? What happens when the retrieval modality differs from the generation modality?

This section surveys multimodal RAG across vision, speech, audio, and video, tracing the rapid expansion of the RAG paradigm beyond its textual origins.

Visual Document RAG

Many real-world documents are not plain text. They contain tables, charts, diagrams, equations, and complex layouts that lose crucial information when converted to text via OCR. Visual document RAG addresses this by treating document pages as images and retrieving them based on visual similarity.

VDocRAG: Visually-Rich Document RAG

VDocRAG [29] operates on the principle that the visual rendering of a document is often more informative than its textual extraction. A table in a financial report, for instance, has a spatial structure (rows, columns, headers) that is immediately apparent from the visual rendering but is destroyed by naive text extraction.

The VDocRAG pipeline comprises:

1. Visual encoding. Each document page is encoded as an image by a vision encoder (e.g., a Vision Transformer), producing a dense representation ${𝒗}_j=\mathrm{ViT}(I_j)\in {\mathrm{ℝ}}^d$ for page image $I_j$.

2. Visual indexing. The page representations are stored in a dense index, analogous to the text index in standard RAG but over visual representations.

3. Query-visual retrieval. Given a text query $𝒙$, a cross-modal encoder maps it to the same representation space: $𝒒=E_{\text{cross}}(𝒙)\in {\mathrm{ℝ}}^d$. Retrieval is by maximum inner product search: ${\mathrm{t}\mathrm{o}\mathrm{p}\text{-}\mathrm{k}}_j⟨𝒒,{𝒗}_j⟩$.

4. Visual-grounded generation. The retrieved page images are fed to a vision-language model that generates the answer conditioned on both the query and the visual content.

The advantage over text-based RAG is substantial for documents with rich visual structure. Tables, charts, and diagrams are retrieved and interpreted in their native visual form, avoiding the information loss inherent in OCR and layout parsing.

Multi-Level Visual QA

Not all visual information is at the same granularity. A question about a specific cell in a table requires fine-grained retrieval; a question about overall trends in a chart requires coarser retrieval. Multi-level VQA [99] addresses this by retrieving at multiple granularities simultaneously: document level, page level, region level, and element level.

Knowledge-Based Visual QA

Some visual questions require external knowledge that is not present in the image. The question “What architectural style is this building?” given a photo requires the model to recognise visual features (pointed arches, flying buttresses) and link them to knowledge about Gothic architecture. KB-VQA systems retrieve from a knowledge base (e.g., Wikipedia, Wikidata) conditioned on both the image and the question, fusing visual perception with encyclopaedic knowledge.

The KB-VQA pipeline combines three types of representation:

1. A visual representation $𝒗=\mathrm{ViT}(I)$ capturing the image content.

2. A textual representation $𝒒=E_q(𝒙)$ capturing the question semantics.

3. A knowledge representation $𝒌=E_k(𝒛)$ capturing the retrieved facts from the knowledge base.

The generator fuses all three: $P(𝒚|𝒙,I)=p_{𝜽}(𝒚|𝒗,𝒒,𝒌)$. This three-way fusion is more complex than standard text RAG but enables answering questions that require both visual perception and world knowledge.

Speech RAG

Speech processing introduces unique challenges for RAG. Audio signals are high-dimensional and temporally extended. Traditional speech systems rely on an ASR (automatic speech recognition) pipeline to transcribe audio to text, then apply text-based RAG. But this pipeline approach has a fundamental flaw: ASR errors propagate through the entire system. A misrecognised proper noun in the transcription leads to a failed retrieval, which leads to a wrong answer.

ReSLM: Retrieval-Enhanced Speech Language Model

ReSLM [30] addresses the domain adaptation problem in speech recognition through retrieval. When a speech model encounters domain-specific entities (medical terms, company names, technical jargon), it often fails because these entities are underrepresented in its training data.

ReSLM works by maintaining a retrieval index of domain-specific entities along with their audio representations. When the speech model encounters an ambiguous segment, it retrieves the most likely entity from the index and biases the recognition toward it.

The architecture comprises:

1. Speech encoder. Encodes the input audio into a sequence of frame-level representations $𝐇=({𝒉}_1,\dots ,{𝒉}_T)$.

2. Entity retriever. For each audio segment, queries a joint speech-text index of domain entities. The index contains both the text form (“metformin”) and a representative audio embedding of each entity.

3. Biased decoding. The retrieved entities bias the decoder's output distribution via shallow fusion: (Reslm BIAS)$$\begin{array}{ccc}& \log P(y_t|y_{<t},𝐇)\leftarrow \log P(y_t|y_{<t},𝐇)+\alpha \cdot \mathrm{𝟙}[y_t\in {ℰ}_{\text{ret}}],& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${ℰ}_{\text{ret}}$ is the set of retrieved entities and $\alpha >0$ is a bias weight.

WavRAG: Audio-Native RAG

WavRAG [31] takes a more radical approach: it bypasses ASR entirely, performing retrieval directly in the audio domain.

WavRAG represents both audio queries and audio documents as embeddings in a shared space, using a model trained with contrastive learning. The query is an audio segment; the index contains audio segments paired with their metadata. Retrieval is by standard nearest-neighbour search in the embedding space.

Audio RAG for Generation

The RAG paradigm extends naturally to audio generation. Just as a text RAG system retrieves passages to condition text generation, an audio RAG system retrieves audio clips to condition audio synthesis.

Re-AudioLDM: Retrieval-Augmented Audio Generation

Re-AudioLDM [32] augments a latent diffusion model for audio generation with a retrieval component based on CLAP (Contrastive Language-Audio Pretraining).

The system operates as follows:

1. Text prompt encoding. A text caption (e.g., “a dog barking in the rain”) is encoded by the CLAP text encoder into an embedding $𝒕\in {\mathrm{ℝ}}^d$.

2. Audio retrieval. The CLAP embedding is used to retrieve the top-$k$ most similar audio clips from a large audio database: ${𝒜}_k={\mathrm{t}\mathrm{o}\mathrm{p}\text{-}\mathrm{k}}_{a\in 𝒜}⟨𝒕,E_a(a)⟩$, where $E_a$ is the CLAP audio encoder.

3. Feature extraction. Acoustic features are extracted from the retrieved clips (e.g., mel-spectrograms, CLAP embeddings) and aggregated into a conditioning vector ${𝒄}_{\text{ret}}$.

4. Conditioned diffusion. The latent diffusion model generates audio conditioned on both the text embedding and the retrieval features: (Reaudioldm)$$\begin{array}{ccc}& {\widehat{𝒛}}_0=\mathrm{LDM}({𝒛}_T,𝒕,{𝒄}_{\text{ret}}),& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${𝒛}_T$ is noise and ${\widehat{𝒛}}_0$ is the denoised latent, which is decoded to a waveform by a vocoder.

The retrieved audio serves as an acoustic “reference” that guides the diffusion model toward realistic sound textures. Without retrieval, the model must generate the characteristics of rain, dog barking, and their interaction entirely from its parametric knowledge. With retrieval, it can copy acoustic textures from real recordings, producing outputs with greater fidelity and naturalness.

RECAP: Retrieval-Augmented Audio Captioning

The inverse task, describing audio in natural language, also benefits from retrieval. RECAP [33] retrieves similar audio clips along with their human-written captions and uses both the audio similarity and caption similarity to improve captioning quality.

The model computes a similarity-weighted aggregation of retrieved captions: (Recap)$$\begin{array}{ccc}& {𝒄}_{\text{cap}}=\sum _{j=1}^k\frac{\exp (⟨𝒆,E_a(a_j)⟩/\tau )}{{\sum }_{j^{\prime }}\exp (⟨𝒆,E_a(a_{j^{\prime }})⟩/\tau )}\cdot E_t({\text{caption}}_j),& \text{<span class="tml-eqn"></span>}\end{array}$$ where $𝒆$ is the query audio embedding, $a_j$ is the $j$-th retrieved audio clip, ${\text{caption}}_j$ is its caption, $E_t$ is a text encoder, and $\tau$ is a temperature parameter. This aggregated caption embedding conditions the captioning model, providing both semantic guidance (what sounds are present) and stylistic guidance (how to describe them).

Video RAG

Video introduces the temporal dimension to multimodal RAG. Retrieving relevant video clips for question answering requires understanding both the visual content and its temporal evolution. A question like “Show me the moment when the goal was scored” requires retrieving a specific temporal segment from a longer video.

Video RAG systems typically employ a two-stage approach. First, a video is segmented into clips (either by fixed temporal windows or by scene boundary detection). Each clip is encoded into a dense vector by a video encoder (e.g., a Video-Swin Transformer or a CLIP-based model). Second, at query time, the text query is encoded and matched against the clip representations. The retrieved clips condition the generation of the answer, which may be text (video QA), a temporal boundary (moment retrieval), or a summary (video captioning).

Formally, let $V=(f_1,f_2,\dots ,f_T)$ be a video of $T$ frames. The video is partitioned into $T/w$ clips of $w$ frames each. Each clip $C_i=(f_{(i-1)w+1},\dots ,f_{iw})$ is encoded as ${𝒄}_i=E_V(C_i)\in {\mathrm{ℝ}}^d$. At query time, the text query $𝒙$ is encoded as $𝒒=E_q(𝒙)\in {\mathrm{ℝ}}^d$, and the top-$k$ clips are retrieved: (Video RAG)$$\begin{array}{ccc}& {𝒞}_k={\mathrm{t}\mathrm{o}\mathrm{p}\text{-}\mathrm{k}}_i⟨𝒒,{𝒄}_i⟩.& \text{<span class="tml-eqn"></span>}\end{array}$$ The generator then produces the answer conditioned on the query and the retrieved clips: $P(𝒚|𝒙,V)=p_{𝜽}(𝒚|𝒙,{𝒞}_k)$.

Multimodal RAG Architecture

We now present a unified view of multimodal RAG, drawing together the modality-specific systems described above.

System	Input	Output	Retrieval	Key Innovation
RAG [25]	Text	Text	DPR	End-to-end retriever-generator training
RETRO [26]	Text	Text	Chunk-level	Chunked cross-attention from trillion-token DB
FiD [27]	Text	Text	DPR	Independent passage encoding, joint decoding
Self-RAG [28]	Text	Text	Adaptive	Reflection tokens for when-to-retrieve
VDocRAG [29]	Text	Text	Visual	Retrieval over rendered document pages
ReSLM [30]	Speech	Text	Speech+Text	Entity-biased speech recognition
WavRAG [31]	Audio	Text	Audio	Direct audio retrieval, no ASR
Re-AudioLDM [32]	Text	Audio	CLAP	Retrieval-conditioned audio diffusion
RECAP [33]	Audio	Text	CLAP	Retrieval-augmented audio captioning

Memory-Augmented Retrieval

Every time you chat with a large language model, something remarkable happens: the model produces fluent, contextually appropriate responses that give the impression of understanding and recall. Yet the moment the context window ends, every detail of your conversation vanishes. The model does not remember your name, your preferences, or the intricate problem you spent forty minutes explaining. It wakes up, tabula rasa, with every new session.

This is not merely an engineering inconvenience. It is a fundamental limitation that separates current language models from anything resembling genuine intelligence. The solution is memory, and the central thesis of this section is that memory is retrieval. The two concepts are not merely analogous; they are computationally identical, differing only in the time scale over which information is stored and recalled.

The Memory-Retrieval Equivalence

The cognitive science of memory, from Tulving's seminal distinction between episodic and semantic memory [36] to Squire's taxonomy of memory systems [41], reveals a striking correspondence with the architecture of modern retrieval systems. Consider the three fundamental operations that any memory system must support:

1. Encoding. A new experience must be transformed into an internal representation suitable for storage. In the brain, this involves hippocampal pattern separation and the formation of engrams [34][35]. In a retrieval system, this is indexing: a document is embedded into a vector space and stored in an index.

2. Storage. The encoded representation must persist over time. Biological memory achieves this through synaptic consolidation, transferring fragile hippocampal traces into stable neocortical representations. A retrieval system achieves this through persistent vector databases, inverted indices, or key-value stores.

3. Retrieval. Given a cue (a thought, a sensation, a question), the system must recover the stored representation. In the brain, this is pattern completion: a partial cue activates the full memory trace. In a retrieval system, this is a query: the cue is embedded and the nearest neighbours in the index are returned.

The mapping between biological memory and retrieval is not merely metaphorical. It is structural: each component of Definition 60 has a direct neural counterpart.

To make the equivalence precise, we define a relevance scoring function that combines the factors governing both biological recall and computational retrieval.

This three-factor model captures a fundamental insight from psychology: we recall memories that are relevant to the current context (semantic similarity), that were important when they were formed (importance weighting), and that are recent (recency bias), unless their importance is sufficiently high to overcome the decay.

The Context Window Problem

The context window of a transformer-based language model is the maximum number of tokens it can attend to simultaneously. For GPT-4, this is 128K tokens. For Claude, it ranges up to 200K tokens. For Gemini 1.5 Pro, it is an impressive 1M tokens. These numbers sound large, but they are woefully insufficient for many real-world applications.

Consider a software engineer who wants an AI assistant to understand their entire codebase. A medium-sized project might contain 500,000 lines of code, or roughly 10 million tokens. Even Gemini's 1M-token context window can hold only 10% of this codebase. A lawyer reviewing discovery documents for a complex case may face millions of pages. A doctor's complete medical history may span decades of records. In every case, the context window is a bottleneck.

The fundamental issue is computational. Self-attention in a standard transformer has $O(n^2)$ complexity in the sequence length $n$. Doubling the context window quadruples the attention cost. Even with efficient attention variants (linear attention, FlashAttention, ring attention), the memory and compute costs of very long contexts remain substantial.

Three main strategies have emerged for managing context when the conversation history exceeds the window:

Sliding Window.

The simplest approach: keep only the most recent $L$ tokens. This discards all information from earlier in the conversation. Formally, ${𝒞}^{\prime }=(c_{T-L+1},\dots ,c_T)$. The sliding window is cheap and simple but catastrophically forgetful: if the user mentioned a crucial constraint in the first message, it is gone forever once the window slides past it.

Summarisation.

Periodically compress older portions of the conversation into a summary. The context then contains the summary plus the recent turns: ${𝒞}^{\prime }=(\text{summary}(c_1,\dots ,c_{T-L/2}),c_{T-L/2+1},\dots ,c_T)$. Summarisation preserves more information than the sliding window, but the compression is lossy. Details are lost, nuances are flattened, and the quality of the summary itself depends on the model that generates it.

Retrieval-Augmented Context.

Store the entire conversation history in a retrieval index. At each turn, embed the current query and retrieve the most relevant past turns. The context becomes ${𝒞}^{\prime }=ℛ(𝒒,𝒮)\cup (c_{T-r+1},\dots ,c_T)$, where the last $r$ turns are always included for coherence and the retrieved turns provide relevant long-term context. This is the approach advocated by Definition 60: the conversation history is the corpus, each turn is a document, and the current query drives retrieval.

Memory-Augmented Architectures

The retrieval-augmented context strategy described above operates outside the model: the retrieval system selects context, and the model processes it as if it were ordinary input. A more ambitious approach integrates memory directly into the model architecture.

MemoryBank.

The MemoryBank framework [100] equips an LLM with a persistent external memory that is updated after each conversation. After each session, the system: (i) extracts key facts and events from the conversation, (ii) embeds them using the model's own encoder, (iii) stores them in a vector database with importance scores and timestamps, and (iv) applies the Ebbinghaus-inspired forgetting mechanism of (Relevance). At the start of each new session, the system retrieves relevant memories and injects them into the prompt.

Memorising Transformers.

The Memorising Transformer [101] takes a more architectural approach: it adds a $k$NN-augmented attention layer that can attend to a large external memory of key-value pairs. During training, the model stores key-value pairs from previous batches in a memory bank. During inference, the attention mechanism computes attention scores over both the local context and the external memory: (MEM Transformer)$$\begin{array}{ccc}& \mathrm{Attn}(𝒒,𝐊,𝐕)=\mathrm{softmax}\left(\frac{𝒒{[{𝐊}_{\text{local}};{𝐊}_{\text{mem}}]}^{\top }}{\sqrt{d}}\right)[{𝐕}_{\text{local}};{𝐕}_{\text{mem}}],& \text{<span class="tml-eqn"></span>}\end{array}$$ where $[\cdot ;\cdot ]$ denotes concatenation, ${𝐊}_{\text{local}}$ and ${𝐕}_{\text{local}}$ are the keys and values from the current context, and ${𝐊}_{\text{mem}}$ and ${𝐕}_{\text{mem}}$ are retrieved from the external memory.

Connection to Engram Theory.

The concept of an engram, a physical trace of memory in the brain [34], provides a compelling biological parallel for memory-augmented architectures. In neuroscience, an engram is a specific pattern of neural activation that encodes a memory. Retrieval involves reactivating this pattern, which is precisely what a $k$NN lookup over stored key-value pairs achieves. For a deeper treatment of memory architectures in neural networks, including their connection to continual learning, we refer the reader to our dedicated memory chapter (24).

Retrieval for Continual Learning

The connection between memory and retrieval extends beyond conversation management to one of the deepest problems in machine learning: catastrophic forgetting. When a neural network is trained on a sequence of tasks, learning new tasks tends to overwrite the weights responsible for old tasks, causing performance on earlier tasks to collapse.

Retrieval offers an elegant solution. Rather than relying entirely on the network's parameters to encode all knowledge, we can store experiences (input-output pairs from previous tasks) in an external memory and retrieve them during training on new tasks. This is experience replay, one of the most effective strategies for continual learning (see 26 for a comprehensive treatment).

The bound reveals two key insights. First, the forgetting rate decreases as $1/B$: more retrieved examples means less forgetting. Second, forgetting is worst when the gradients of the new task and the old task are highly conflicting (large gradient difference norm), which is precisely when replay is most needed.

The key design choice in retrieval-based continual learning is which experiences to store and retrieve. Random sampling works surprisingly well, but targeted strategies can do better:

• Reservoir sampling maintains a uniform sample over all past experiences, regardless of which task they came from.

• Herding-based selection chooses examples whose embeddings best approximate the mean embedding of each task, preserving the distributional properties of the original data.

• Retrieval by gradient conflict: store examples whose gradients most conflict with the current task, since these are the ones most at risk of being forgotten.

Vector Embeddings for Memory

The practical implementation of memory-augmented retrieval hinges on the quality of the embeddings used to represent memories. A good memory embedding must satisfy several properties:

1. Semantic faithfulness. Semantically similar memories should have similar embeddings: $\mathrm{sim}(ℰ(m_i),ℰ(m_j))$ should correlate with the semantic similarity of $m_i$ and $m_j$.

2. Temporal awareness. The embedding should encode (or be accompanied by) temporal information, enabling recency-based retrieval.

3. Compositional structure. The embedding space should support operations like analogy (“what was the equivalent of $X$ in context $Y$?”) and negation (“retrieve memories unlike $X$”).

4. Efficient indexing. The embeddings must be compatible with approximate nearest-neighbour data structures for fast retrieval.

In practice, conversation turns are typically embedded using the same encoder used for document retrieval (e.g., a sentence transformer or an API-based embedding model). The temporal component is handled separately, as in (Relevance), by storing timestamps alongside embeddings.

Retrieval-Augmented Generation as Memory

With the memory-retrieval equivalence established, we can now view retrieval-augmented generation [25] through a completely new lens. A RAG system is not merely a clever engineering trick for injecting knowledge into a language model. It is a cognitive architecture: the language model is the “thinking” component (analogous to working memory and executive function), and the retrieval system is the “remembering” component (analogous to long-term memory).

This perspective immediately suggests improvements. If RAG is a cognitive architecture, then insights from cognitive science should transfer. For example:

• Chunking: Cognitive science shows that working memory has limited capacity (the famous “7 $\pm$ 2” items of Miller). Similarly, the context window has limited capacity. Just as humans chunk information to fit more into working memory, RAG systems should chunk retrieved documents into semantically coherent units rather than arbitrary token spans.

• Spaced retrieval: In human memory, spaced repetition strengthens recall. In RAG, retrieving a document multiple times across different queries should increase its importance weight, making it more accessible in the future.

• Interference: In human memory, similar memories interfere with each other (the fan effect). In retrieval, many similar documents in the index can reduce precision. De-duplication and diversity-promoting retrieval mitigate this.

The RETRO architecture [26] takes this cognitive perspective to its logical conclusion by interleaving retrieval directly into the transformer layers. At every chunk boundary, the model retrieves relevant passages from a massive external database and fuses them into the residual stream via cross-attention. The result is a model that “thinks and remembers” simultaneously, rather than treating retrieval as a separate pre-processing step.

Information-Theoretic Analysis of Retrieval

When you compress a 10,000-word document into a 3,072-dimensional vector embedding, how much information do you lose? And what are the consequences of that loss for retrieval accuracy?

These are not idle questions. Every retrieval system in production today, from Google Search to the RAG pipeline powering your favourite AI assistant, relies on a lossy compression step: documents are mapped to finite-dimensional vectors, and retrieval operates on these compressed representations rather than on the documents themselves. The information lost in this compression imposes a fundamental ceiling on retrieval accuracy, a ceiling that no amount of clever engineering can overcome.

In this section, we develop a rigorous information-theoretic framework for understanding this ceiling. We draw on Shannon's foundational work [4] and the modern theory of rate-distortion [37] to derive precise bounds on what embedding-based retrieval can and cannot achieve.

The Embedding as a Lossy Channel

Consider a document $X$ drawn from a source distribution $p(X)$ over a document space $𝒳$. An embedding function $ℰ:𝒳\to {\mathrm{ℝ}}^d$ maps $X$ to a $d$-dimensional vector $𝒛=ℰ(X)\in {\mathrm{ℝ}}^d$. From the perspective of information theory, the embedding function defines a channel from documents to embeddings, and the mutual information $𝖨(X;𝒛)$ quantifies how much information about the document is preserved in the embedding.

The critical observation is that this channel is necessarily lossy: a real-valued vector in ${\mathrm{ℝ}}^d$ cannot, in general, encode all the information in an arbitrary document. The Data Processing Inequality provides the first constraint.

Rate-Distortion Theory for Retrieval

Shannon's rate-distortion theory provides a precise framework for understanding the trade-off between compression and fidelity. We now apply it to the retrieval setting.

Let $X$ be a random document and $𝒛$ its embedding. Define a distortion function $d:𝒳\times {\mathrm{ℝ}}^d\to {\mathrm{ℝ}}_{\ge 0}$ that measures how much information relevant to retrieval is lost when $X$ is represented by $𝒛$. A natural choice is the retrieval distortion: (Distortion)$$\begin{array}{ccc}& d(X,𝒛)=1-\mathrm{Pr}[X\in \mathrm{top-}k(𝒛,𝒮)],& \text{<span class="tml-eqn"></span>}\end{array}$$ which is 0 if the document is always in the top-$k$ results when relevant and 1 if it is never retrieved.

The rate-distortion function tells us the minimum number of bits of mutual information needed between document and embedding to achieve a given level of retrieval accuracy. If we want distortion at most $D$, we need at least $R(D)$ bits of information in the embedding.

Information Loss in Embedding

The bottleneck theorem (Theorem 12) gives a worst-case bound based on the raw capacity of the embedding. We can obtain tighter bounds by reasoning about the structure of the information that embeddings capture and the information they discard.

The more practically relevant bound concerns the quality of information captured by the embedding, not just its quantity.

The Modality Gap

When a retrieval system maps different modalities (text, images, audio) to a shared embedding space, a new source of information loss emerges: the modality gap. Each modality carries information that is specific to it and cannot be expressed in other modalities. The colour palette of an image, the timbre of a voice, the syntactic structure of a sentence: these modality-specific features are necessarily lost when heterogeneous inputs are projected to a common representation.

The theorem has a clear practical message: if your embedding wastes capacity on modality-specific features ($𝖨(M_2;{𝒛}_2)$ is large), then less capacity remains for the shared semantic content that enables cross-modal retrieval. This explains why models like CLIP, which are trained specifically to align image and text representations around shared semantics, dramatically outperform models trained independently on each modality.

The Information Bottleneck Perspective

The analysis so far has focused on capacity: how much information can an embedding hold? A complementary perspective comes from the Information Bottleneck (IB) method [102], which asks: what is the optimal trade-off between compression and relevance?

In the IB framework, we seek an embedding $𝒛$ that maximises the information about the relevance variable $Y$ (e.g., the query) while minimising the information about the input $X$. Formally, the IB objective is (IB)$$\begin{array}{ccc}& \underset{p(𝒛|X)}{\min }𝖨(X;𝒛)-\beta 𝖨(𝒛;Y),& \text{<span class="tml-eqn"></span>}\end{array}$$ where $\beta >0$ controls the trade-off. As $\beta \to \infty$, the objective favours maximum relevance; as $\beta \to 0$, it favours maximum compression.

The IB perspective provides a principled explanation for why contrastive learning produces good retrieval embeddings. The InfoNCE loss, the standard training objective for modern embedding models, can be shown to be a variational lower bound on $𝖨(𝒛;Y)$. At the same time, the finite dimensionality of the embedding implicitly constrains $𝖨(X;𝒛)$. Together, these two forces push the embedding toward the IB-optimal point on the retrieval information curve.

Consequences for RAG

The information-theoretic analysis has direct consequences for retrieval-augmented generation (RAG) systems. When a RAG system retrieves documents to ground a language model's output, errors in retrieval propagate through the generation step and compound.

Fundamental Limits and Open Questions

We close this section with a conjecture that captures what we believe to be a deep truth about embedding-based retrieval.

Retrieval for Verification and Theorem Proving

Generative models produce outputs that look correct. They are fluent, confident, and well-structured. But looking correct is not the same as being correct. A language model can generate a mathematical proof with every step formatted perfectly, every symbol placed precisely, and a devastating logical gap in the middle that would take an expert minutes to find. It can generate a clinical summary that reads like a textbook but contains a drug interaction that could harm a patient. It can produce a legal brief that cites cases with perfect formatting, where half the cases do not exist.

The verification problem, how to check whether a generated output is actually correct, is one of the central challenges of modern AI. In this section, we show how retrieval provides a powerful tool for verification: by retrieving known facts, established theorems, and documented evidence, we can check the consistency of generated outputs against a trusted knowledge base.

This is not the same as retrieval-augmented generation. In RAG, retrieval happens before generation to provide context. In retrieval-based verification, retrieval happens after generation to check the result. The distinction is illustrated in fig:ret:verification-pipeline.

Retrieval for Formal Mathematics

Mathematical theorem proving is an ideal testbed for retrieval-based verification because it has an unambiguous standard of correctness: a proof is either valid or it is not, as determined by a formal proof checker. There is no room for subjective interpretation or partial credit.

Modern proof assistants such as Lean [97], Coq, and Isabelle maintain large libraries of formalised mathematics: thousands of definitions, lemmas, and theorems that have been machine-verified. When a generative model attempts to construct a proof, it needs access to these libraries, and finding the right lemma is a retrieval problem.

The AlphaProof system [98] demonstrated the power of combining retrieval with formal verification at the 2024 International Mathematical Olympiad. AlphaProof's architecture integrates three components:

1. A generative model that proposes proof steps, trained on a large corpus of mathematical text.

2. A retrieval system that finds relevant lemmas, definitions, and previously proved results from a formalised library.

3. A formal verifier (Lean) that checks each proof step for correctness, providing an absolute ground truth.

The key insight is that the verifier provides a reward signal for the generative model: correct proof steps receive positive reward, incorrect steps receive negative reward. This enables reinforcement learning over the space of proofs, with retrieval providing the “knowledge base” of available mathematical facts.

Fact-Checking with Retrieval

Formal mathematics provides an unambiguous verification signal, but most real-world verification tasks are messier. When a language model generates a claim about history, medicine, or current events, there is no Lean-like verifier to consult. Instead, we must rely on evidence: documents from trusted sources that either support or contradict the claim.

The retrieval-based fact-checking pipeline consists of three stages:

Stage 1: Claim Decomposition.

Complex statements are broken into atomic, verifiable claims. For example, the statement “Einstein won the Nobel Prize in 1922 for his work on general relativity” decomposes into: (a) Einstein won the Nobel Prize, (b) the year was 1922, and (c) the prize was for general relativity. (Claims (a) and (b) are approximately correct; claim (c) is false: the prize was for the photoelectric effect.)

Stage 2: Evidence Retrieval.

For each atomic claim, retrieve the most relevant documents from a knowledge base. The quality of this retrieval step is the primary determinant of the system's accuracy. If the relevant evidence is not retrieved, the claim cannot be verified.

Stage 3: Entailment Checking.

For each (claim, evidence) pair, determine whether the evidence supports, refutes, or is neutral with respect to the claim. This is a natural language inference (NLI) task, and modern NLI models achieve high accuracy when given the right evidence.

Scientific Verification

The verification challenge becomes particularly acute in scientific contexts, where claims are complex, evidence is nuanced, and the consequences of error are severe.

Drug Interaction Checking.

A language model summarising a patient's chart might suggest a medication without checking for interactions with the patient's existing prescriptions. Retrieval-based verification can cross-reference the suggested medication against a drug interaction database (e.g., DrugBank), flagging potential conflicts before the recommendation reaches a clinician.

Experimental Reproducibility.

When a model generates a description of a scientific experiment, retrieval can surface similar experiments from the literature, enabling comparison of methods, sample sizes, and reported effects. Discrepancies between the generated description and the retrieved literature may indicate errors or hallucinations.

Consistency with Physical Laws.

For scientific claims involving quantitative predictions (e.g., “the boiling point of this compound is 450K”), retrieval can surface known data points for similar compounds, enabling a sanity check. If the predicted value falls far outside the range of known values, it warrants scrutiny.

Information-Theoretic Limits on Verification

Can a retrieval-based verification system catch all errors? The answer, perhaps unsurprisingly, is no. There are fundamental limits on what verification can achieve, and understanding these limits is essential for deploying verification systems responsibly.

Self-Verification and Reflexive Retrieval

A recent line of work explores self-verification: the model retrieves evidence and checks its own output without external intervention. Self-RAG [28] introduces special tokens that allow the model to decide when to retrieve, what to retrieve, and whether the retrieved evidence supports the generated output.

The formal framework is elegant. At each generation step, the model outputs one of three reflection tokens:

1. [Retrieve]: indicates the model's confidence is low and evidence should be retrieved before continuing.

2. [IsRel]: after retrieval, evaluates whether the retrieved passage is relevant to the query.

3. [IsSup]: evaluates whether the retrieved passage supports the generated text.

This transforms the model from a passive consumer of retrieved context into an active participant in the retrieval-verification loop. The model learns, through training, when its own knowledge is insufficient and external evidence is needed.

The proposition reinforces a theme that runs through this entire chapter: the quality of retrieval is the foundation upon which everything else is built. Better retrieval improves RAG, improves memory, improves verification, and improves self-monitoring.

Retrieval as a Bridge Between Neural and Symbolic AI

Retrieval-based verification occupies a unique position at the intersection of neural and symbolic AI. The generative model is neural: it produces outputs through pattern matching over learned representations. The verifier, in the case of formal mathematics, is symbolic: it checks proofs through logical deduction. Retrieval is the bridge between these two paradigms, selecting from a symbolic knowledge base the facts that the neural model needs to ground its output.

This bridge architecture, neural generation + retrieval + symbolic verification, may represent a more robust paradigm for building trustworthy AI systems than either pure neural or pure symbolic approaches alone. The neural component provides flexibility and natural language understanding. The symbolic component provides guarantees and precision. And retrieval provides the interface between them.

Recommendation as Retrieval

Netflix, Spotify, Amazon – recommendation is arguably the most economically impactful retrieval problem in existence. When a user opens a streaming service, the platform must retrieve a handful of relevant items from a catalogue containing millions of movies, songs, or products. Every recommendation is, at its core, a retrieval operation: given the user as a query, find the items most likely to satisfy their needs. The economic stakes are enormous. Netflix has estimated that its recommendation system is worth over one billion dollars per year in retained subscriptions [103]. Amazon attributes roughly 35% of its revenue to recommendations. Spotify's Discover Weekly playlist, a pure retrieval product, reaches over 100 million users.

This section develops the formal connection between recommendation and retrieval, traces the evolution from classical collaborative filtering to modern generative recommendation, and examines how large language models are reshaping the field. Along the way, we will see that the same mathematical abstractions – embeddings, similarity functions, approximate nearest neighbours – appear in both domains, and that advances in one have consistently accelerated the other.

The Recommendation-Retrieval Connection

The architecture of every modern recommendation system reveals the retrieval machinery underneath. The standard two-stage pipeline consists of candidate generation followed by reranking, and these stages map directly to the retrieval-then-rank paradigm that pervades information retrieval.

Candidate generation IS retrieval.

From a catalogue $𝒞$ of millions of items, the candidate generation stage selects a manageable set of a few hundred candidates. This is precisely the retrieval problem: given a query (the user context), find the top-$k$ relevant documents (items) from a large corpus. The user context $𝒒\in {\mathrm{ℝ}}^d$ encodes everything the system knows about the user – their history, demographics, current session – and each item ${𝒅}_j\in {\mathrm{ℝ}}^d$ lives in the same embedding space. Candidate generation reduces to approximate nearest-neighbour search: (Candidate Generation)$$\begin{array}{ccc}& {𝒞}_k(𝒒)={\mathrm{arg\hspace{0.17em}max}}_{𝒮\subseteq 𝒞,|𝒮|=k}\sum _{j\in 𝒮}s(𝒒,{𝒅}_j),& \text{<span class="tml-eqn"></span>}\end{array}$$ where $s(\cdot ,\cdot )$ is a scoring function, typically the inner product or cosine similarity.

Reranking IS ranking.

The reranking stage takes the $k$ candidates and produces a final ordering based on richer, more expensive features. This is the learning-to-rank problem that we develop in detail in Learning to Rank. The reranker may use cross-attention between user and item features, incorporate real-time signals (time of day, device type, recently clicked items), and optimize for business-specific objectives beyond pure relevance – diversity, freshness, revenue.

Collaborative Filtering as Embedding Retrieval

The oldest and most influential approach to recommendation is collaborative filtering: predict a user's preferences from the collective behaviour of all users. The key insight, formalized by matrix factorization, is that collaborative filtering is embedding-based retrieval.

Matrix factorization.

Let $𝐑\in {\mathrm{ℝ}}^{M\times N}$ be the (sparse) user-item interaction matrix, where $R_{ui}$ records user $u$'s rating (or implicit feedback) for item $i$. Matrix factorization seeks low-rank embeddings ${𝒒}_u\in {\mathrm{ℝ}}^d$ for each user and ${𝒅}_i\in {\mathrm{ℝ}}^d$ for each item such that (Matrix Factorization)$$\begin{array}{ccc}& R_{ui}\approx ⟨{𝒒}_u,{𝒅}_i⟩+b_u+b_i+\mu ,& \text{<span class="tml-eqn"></span>}\end{array}$$ where $b_u$, $b_i$ are user and item biases and $\mu$ is the global mean. The training objective minimizes the regularized squared error over observed entries: (MF Objective)$$\begin{array}{ccc}& \underset{\{{𝒒}_u\},\{{𝒅}_i\},\{b_u\},\{b_i\}}{\min }\sum _{(u,i)\in 𝒪}(R_{ui}-⟨{𝒒}_u,{𝒅}_i⟩-b_u-b_i-\mu {)}^2+\lambda ({\Vert {𝒒}_u\Vert }^2+{\Vert {𝒅}_i\Vert }^2+b_u^2+b_i^2),& \text{<span class="tml-eqn"></span>}\end{array}$$ where $𝒪$ is the set of observed interactions. Once trained, recommendation reduces to retrieving the items with the highest inner products $⟨{𝒒}_u,{𝒅}_i⟩$ – a standard maximum inner product search (MIPS) problem that can be solved with the same ANN indices used in dense retrieval.

Neural collaborative filtering.

The linearity of the inner product in (Matrix Factorization) limits the expressiveness of matrix factorization. Neural collaborative filtering (NCF) replaces the inner product with a learned scoring function: (NCF)$$\begin{array}{ccc}& {\hat{R}}_{ui}=f_{𝜽}({𝒒}_u,{𝒅}_i),& \text{<span class="tml-eqn"></span>}\end{array}$$ where $f_{𝜽}$ is a multilayer perceptron. While more expressive, this approach sacrifices the decomposability that enables efficient retrieval: scoring each user-item pair requires a forward pass through $f_{𝜽}$, making exhaustive scoring over millions of items prohibitive at serving time.

Two-tower models.

The two-tower architecture resolves the tension between expressiveness and efficiency. It computes user and item embeddings through separate deep networks (towers) and scores by inner product: (TWO Tower)$$\begin{array}{ccc}& s(u,i)=⟨g_{𝜽}(u),h_{𝝓}(i)⟩,& \text{<span class="tml-eqn"></span>}\end{array}$$ where $g_{𝜽}:𝒰\to {\mathrm{ℝ}}^d$ is the user tower and $h_{𝝓}:ℐ\to {\mathrm{ℝ}}^d$ is the item tower. Each tower can be arbitrarily deep (incorporating transformer layers, attention over the user's history, etc.), yet the final scoring remains a simple inner product. This means item embeddings $h_{𝝓}(i)$ can be precomputed and indexed offline, reducing serving to a single forward pass through the user tower followed by an ANN lookup.

The generalization bound reveals two important insights. First, the embedding dimension $d$ appears under the square root, suggesting that moderate dimensions ($d=64$–$256$) suffice for generalization even with large catalogues. Second, the product $B_u{B}_i$ controls the effective capacity: regularizing embedding norms (a standard practice) directly tightens the generalization bound.

Content-Based Recommendation

Collaborative filtering relies entirely on interaction patterns and fails when those patterns are sparse or absent. Content-based recommendation addresses this by deriving item representations from the items themselves – their text descriptions, images, audio, or metadata.

Item embeddings from content features.

Given an item $i$ with associated content $c_i$ (a product description, an image, an audio clip), a content encoder $\varphi :𝒳\to {\mathrm{ℝ}}^d$ produces the item embedding ${𝒅}_i=\varphi (c_i)$. The encoder $\varphi$ may be a pretrained language model (BERT, sentence transformers), a vision encoder (CLIP, ViT), or a domain-specific model. The user embedding ${𝒒}_u$ is then computed as an aggregation of the content embeddings of items the user has interacted with: (Content USER Embedding)$$\begin{array}{ccc}& {𝒒}_u=\frac{1}{|{\mathscr{H}}_u|}\sum _{i\in {\mathscr{H}}_u}{\alpha }_{ui}\varphi (c_i),& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${\mathscr{H}}_u$ is the user's interaction history and ${\alpha }_{ui}$ are attention weights reflecting the recency and relevance of each interaction.

Cross-modal recommendation.

Modern e-commerce items are inherently multimodal: a product has a title (text), images (vision), specifications (structured data), and sometimes video or audio. A winter jacket's appeal depends on its visual design (captured by images), its technical specifications (captured by structured metadata), and user reviews (captured by text). No single modality tells the full story. Cross-modal recommendation fuses these modalities into a unified item embedding. The typical approach uses modality-specific encoders followed by a fusion layer: (Cross Modal)$$\begin{array}{ccc}& {𝒅}_i=\text{Fuse}({\varphi }_{\text{text}}(c_i^{\text{text}}),{\varphi }_{\text{image}}(c_i^{\text{image}}),{\varphi }_{\text{meta}}(c_i^{\text{meta}})),& \text{<span class="tml-eqn"></span>}\end{array}$$ where the fusion may be concatenation followed by a projection, cross-attention, or a mixture-of-experts gating mechanism. The quality of the content encoders is critical: systems that use CLIP for vision and a fine-tuned sentence transformer for text consistently outperform those with simpler encoders. The fusion layer must also handle missing modalities gracefully: not every item has images, and not every product has reviews. Dropout-based training, where random modalities are zeroed out during training, produces fusion layers that degrade gracefully when modalities are absent at serving time.

LLM-Based Recommendation

The emergence of large language models has introduced a radically different approach to recommendation: instead of learning embeddings and computing similarities, simply describe the task in natural language and let the LLM rank items.

Zero-shot ranking with LLMs.

Hou et al. [24] demonstrate that LLMs can serve as zero-shot rankers for recommendation, bypassing the entire embedding-based pipeline. The approach is disarmingly simple: encode the recommendation task as a natural language prompt and let the LLM's world knowledge and reasoning capabilities produce a ranking. Given a user's interaction history and a set of candidate items, the LLM receives a prompt of the form:

The user has previously watched: [Movie A], [Movie B], [Movie C]. Rank the following movies by how likely the user is to enjoy them: 1. [Candidate 1] 2. [Candidate 2] $\dots$ $k$. [Candidate $k$]

The LLM produces a permutation of the candidates. Remarkably, this approach – which uses no trained recommendation model and no interaction data beyond what fits in the prompt – achieves competitive performance on standard benchmarks. The LLM implicitly leverages its knowledge of user preferences (absorbed from reviews, forums, and social media during pretraining) and its understanding of item similarities (absorbed from product descriptions and comparisons) to produce reasonable rankings.

Position bias and popularity bias.

LLM-based recommendation inherits biases from the language model's pretraining. Two biases are especially pernicious. Position bias: the LLM's ranking is sensitive to the order in which candidates appear in the prompt. Items presented earlier or later receive disproportionate attention, distorting the ranking. Popularity bias: the LLM has seen popular items more frequently during pretraining and tends to rank them higher regardless of the user's actual preferences. A user who prefers obscure arthouse films will still receive mainstream blockbuster recommendations.

Generative Recommendation

The most radical departure from the retrieve-then-rank paradigm is generative recommendation: instead of scoring existing items, the model generates the identifier of the recommended item. This is a conceptual leap. Traditional recommendation asks “which of these items is most relevant?” Generative recommendation asks “what item should I construct?” The distinction is analogous to the difference between retrieval and generation in NLP: retrieval selects from a fixed set, generation produces something new. This reframes recommendation as a sequence-to-sequence problem, where the input is the user's history and the output is a sequence of tokens representing the recommended item.

TIGER: Semantic IDs for items.

TIGER (Transformer Index for GEneRative Recommenders) [91] assigns each item a semantic ID – a short sequence of discrete tokens derived from the item's content embedding via residual quantization. Given a user's interaction history $(i_1,i_2,\dots ,i_t)$, TIGER autoregressively generates the semantic ID of the next item: (Tiger)$$\begin{array}{ccc}& p(i_{t+1}|i_1,\dots ,i_t)=\prod _{l=1}^{L}p(c_l^{(t+1)}|c_1^{(t+1)},\dots ,c_{l-1}^{(t+1)},i_1,\dots ,i_t),& \text{<span class="tml-eqn"></span>}\end{array}$$ where $(c_1^{(t+1)},\dots ,c_L^{(t+1)})$ is the $L$-token semantic ID of item $i_{t+1}$. The semantic ID encodes hierarchical item semantics: the first token captures coarse category (e.g., “action movie”), and subsequent tokens refine to specific items. This hierarchy enables efficient beam search during generation: the model first narrows down to a coarse category (pruning most of the catalogue), then refines within that category, and so on. At each level of the hierarchy, the beam search considers only $|𝒱|$ options, making the total search cost $O(L\cdot |𝒱|\cdot B)$ where $B$ is the beam width – far more efficient than scoring all $N$ items individually.

IDGenRec: Textual ID learning.

IDGenRec [92] takes a different approach: instead of quantized codes, it generates human-readable textual identifiers. Each item receives a unique textual ID (e.g., “sci-fi_2023_inception-like_7842”) learned jointly with the recommendation model. The advantage is direct compatibility with pretrained language models: the recommendation task becomes a standard text generation task, enabling the use of instruction-tuned LLMs without architectural modification. The textual IDs are also human-interpretable, which aids debugging: when a model generates an incorrect recommendation, developers can read the generated ID and understand which semantic features the model was targeting.

The Cold-Start Problem and Generative Solutions

The cold-start problem arises when a new user (with no interaction history) or a new item (with no interactions) enters the system. Collaborative filtering, which relies entirely on interaction patterns, is helpless in this setting: the user's row in the interaction matrix is entirely empty, and no embedding can be learned. Content-based approaches partially mitigate the problem by using item features, but they cannot capture collaborative signals. The cold-start problem is not merely academic – in a typical e-commerce platform, 10–30% of items in the catalogue have fewer than five interactions, and new items enter the catalogue continuously.

Generative recommendation offers a natural solution. Because TIGER and IDGenRec derive item IDs from content features, new items receive meaningful IDs immediately upon entering the catalogue. The model can recommend a new item by generating its semantic ID, even if no user has ever interacted with it. For new users, the generative framework can condition on side information (stated preferences, demographic features, or even a natural language description of the user's interests) rather than requiring an interaction history.

Approach	Cold-Start	Latency	Expressiveness	Scalability
Matrix factorization	Poor	Very low	Low	Excellent
Neural CF	Poor	Moderate	High	Moderate
Two-tower	Moderate	Low	High	Excellent
Content-based	Good	Low	Moderate	Excellent
LLM zero-shot	Excellent	High	Very high	Poor
TIGER (generative)	Good	Moderate	High	Good
IDGenRec	Good	Moderate	Very high	Good

Learning to Rank

Retrieval finds the candidates; ranking orders them. The learning to rank (LTR) problem is deceptively simple to state: given a query $q$ and a set of candidate documents $\{d_1,\dots ,d_n\}$, produce a permutation $\pi :\{1,\dots ,n\}\to \{1,\dots ,n\}$ such that $d_{\pi (1)}$ is the most relevant document, $d_{\pi (2)}$ the second most relevant, and so on. The difficulty lies in the gap between the objective we care about – ranking quality as measured by metrics like nDCG or MAP – and the objectives we can optimize with gradient-based methods. Ranking metrics are discrete, non-differentiable functions of the predicted permutation. This section traces the three classical approaches to bridging this gap (pointwise, pairwise, listwise) and then examines how neural models and LLMs have reshaped the landscape.

Pointwise Approaches

The simplest approach ignores the ranking structure entirely and treats each query-document pair independently.

Regression.

Given a query-document pair $(q,d)$ with a graded relevance label $y\in \{0,1,2,3,4\}$ (e.g., from “not relevant” to “perfectly relevant”), pointwise regression learns a scoring function $f_{𝜽}(q,d)\in \mathrm{ℝ}$ that minimizes the squared error: (Pointwise Regression)$$\begin{array}{ccc}& L_{\text{point}}=\sum _{(q,d,y)\in 𝒟}(f_{𝜽}(q,d)-y{)}^2.& \text{<span class="tml-eqn"></span>}\end{array}$$ The ranking is then induced by sorting documents by their predicted scores. The simplicity of this formulation is appealing, but it ignores the fundamental insight that ranking is a relative task: the absolute score of a document matters far less than its score relative to other documents for the same query.

Classification.

An even simpler variant treats relevance as a binary label $y\in \{0,1\}$ and trains a classifier: (Pointwise Classification)$$\begin{array}{ccc}& L_{\text{class}}=-\sum _{(q,d,y)\in 𝒟}[y\log \sigma (f_{𝜽}(q,d))+(1-y)\log (1-\sigma (f_{𝜽}(q,d)))].& \text{<span class="tml-eqn"></span>}\end{array}$$

Pairwise Approaches

Pairwise methods reduce ranking to binary classification on pairs of documents: given two documents $d_i$ and $d_j$ for the same query, learn to predict which one should rank higher.

Formal pairwise loss.

Let ${𝒫}_q=\{(i,j):y_i>y_j\}$ be the set of pairs where document $i$ is more relevant than document $j$ for query $q$. The general pairwise loss is (Pairwise LOSS)$$\begin{array}{ccc}& L_{\text{pair}}=\sum _q\sum _{(i,j)\in {𝒫}_q}\ell (f_{𝜽}(q,d_i)-f_{𝜽}(q,d_j),y_{ij}),& \text{<span class="tml-eqn"></span>}\end{array}$$ where $y_{ij}=+1$ indicates that $d_i$ should rank above $d_j$ and $\ell$ is a margin-based or logistic loss.

RankSVM.

RankSVM uses a hinge loss: $\ell (s,y)=\max (0,1-y\cdot s)$, treating each pair as a binary classification instance with a margin constraint. This reduces ranking to a standard SVM problem with $O(n^2)$ training pairs per query. The margin constraint encourages the model to produce score differences that are large enough to be robust to noise, a property inherited from the SVM tradition.

RankBoost.

RankBoost applies the AdaBoost framework to pairwise ranking, iteratively constructing weak rankers that focus on the most frequently misordered pairs. The boosting weights are updated based on pairwise classification errors.

LambdaRank.

LambdaRank [43] introduces the key insight that unlocked modern neural ranking: instead of defining a loss and deriving its gradient, define the gradient directly. The LambdaRank gradient for a pair $(i,j)$ is (Lambdarank)$$\begin{array}{ccc}& {\lambda }_{ij}=-\sigma (s_i-s_j)\cdot |\mathrm{\Delta }{\text{nDCG}}_{ij}|,& \text{<span class="tml-eqn"></span>}\end{array}$$ where $s_i=f_{𝜽}(q,d_i)$, $\sigma (\cdot )$ is the logistic sigmoid, and $|\mathrm{\Delta }{\text{nDCG}}_{ij}|$ is the absolute change in nDCG that would result from swapping documents $i$ and $j$ in the current ranking. The $\mathrm{\Delta }\text{nDCG}$ weighting ensures that swaps near the top of the ranking (which affect nDCG most) receive larger gradients. This is a form of cost-sensitive learning: the model pays more attention to errors that matter.

The total gradient on the score of document $i$ is (Lambda Gradient)$$\begin{array}{ccc}& \frac{\partial L}{\partial s_i}=\sum _{j:y_j\ne y_i}{\lambda }_{ij}=\sum _{j:y_j>y_i}{\lambda }_{ij}-\sum _{j:y_j<y_i}{\lambda }_{ji}.& \text{<span class="tml-eqn"></span>}\end{array}$$ Because ${\lambda }_{ij}$ incorporates the nDCG change, optimizing this “pseudo-loss” approximately optimizes nDCG itself, even though nDCG is a discrete, non-differentiable function of the ranking.

Listwise Approaches

While pairwise methods are a step up from pointwise, they still decompose the ranking into independent pairs and lose global structure. Listwise methods optimize over the entire ranked list at once.

LambdaMART.

LambdaMART combines the lambda gradients from LambdaRank with the MART (Multiple Additive Regression Trees) gradient boosting framework. At each boosting iteration, a regression tree is fit to the lambda gradients ${\lambda }_{ij}$ summed over all pairs for each document. The tree structure naturally captures feature interactions, and the lambda weighting ensures that the boosting process focuses on the most impactful ranking errors.

LambdaMART has been the dominant non-neural ranking model for over a decade, winning multiple learning-to-rank competitions and forming the backbone of commercial search engines. Its success stems from the combination of three strengths: gradient boosting handles heterogeneous features naturally, the lambda weighting focuses optimization on metric-impactful errors, and the regression tree ensemble is fast enough for online serving (typically under 5 milliseconds for 100 candidates).

ListNet.

ListNet defines a probability distribution over permutations using the Plackett-Luce model: (Plackett LUCE)$$\begin{array}{ccc}& P_s(\pi )=\prod _{j=1}^n\frac{\exp (s_{\pi (j)})}{{\sum }_{l=j}^n\exp (s_{\pi (l)})},& \text{<span class="tml-eqn"></span>}\end{array}$$ where $s_i=f_{𝜽}(q,d_i)$ are predicted scores. ListNet minimizes the KL divergence between the predicted permutation distribution and the target distribution (derived from relevance labels). Computing the full permutation distribution is combinatorially expensive ($n!$ permutations), so ListNet uses the “top-one probability” approximation: the probability that document $i$ is ranked first, (Listnet TOP1)$$\begin{array}{ccc}& P_s(d_i\text{ is top-1})=\frac{\exp (s_i)}{{\sum }_{j=1}^n\exp (s_j)},& \text{<span class="tml-eqn"></span>}\end{array}$$ which is simply a softmax over scores. This approximation reduces the combinatorial Plackett-Luce loss to a standard cross-entropy loss between two softmax distributions – one derived from the predicted scores and one from the true relevance labels. The gradient computation is $O(n)$, making ListNet practical for large candidate sets.

ListMLE.

ListMLE directly maximizes the likelihood of the ground-truth permutation under the Plackett-Luce model: (Listmle)$$\begin{array}{ccc}& L_{\text{ListMLE}}=-\sum _q\log P_s({\pi }_q^{\ast }),& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${\pi }_q^{\ast }$ is the ground-truth ranking for query $q$. This is computationally tractable because the Plackett-Luce likelihood decomposes as a product of $n$ terms, each requiring a softmax over the remaining documents.

Neural Ranking Models

Classical LTR methods (RankSVM, LambdaMART) operate on hand-crafted features: BM25 scores, PageRank, click-through rates, URL depth, and dozens of other signals. Neural ranking models replace the feature engineering with end-to-end learning from raw text.

BERT as a cross-encoder ranker.

The most direct application of neural models to ranking is the cross-encoder: concatenate the query and document as a single input to BERT and use the [CLS] token representation to predict relevance: (Cross Encoder)$$\begin{array}{ccc}& s(q,d)={𝐖}^{\top }\cdot \text{BERT}([q;\text{[𝚂𝙴𝙿]};d])+b,& \text{<span class="tml-eqn"></span>}\end{array}$$ where $𝐖\in {\mathrm{ℝ}}^d$ and $b\in \mathrm{ℝ}$ are learned parameters. The cross-encoder attends jointly over query and document tokens, enabling rich interaction patterns (exact term matching, paraphrase detection, coreference resolution) that bi-encoder models cannot capture. For example, for the query “Who invented the telephone?” and a passage containing “Bell is credited with patenting the first practical telephone,” the cross-encoder can attend from “invented” to “credited with patenting” and from “Who” to “Bell” – interactions that a bi-encoder, which encodes query and document separately, fundamentally cannot capture.

The cost is inference: every query-document pair requires a full forward pass, making cross-encoders unsuitable for first-stage retrieval but ideal for reranking a small candidate set.

monoT5: Sequence-to-sequence ranking.

Nogueira et al. [84] reframe ranking as a text-to-text problem using T5. The input is “Query: $q$ Document: $d$ Relevant:” and the model generates either “true” or “false.” The relevance score is the log-probability assigned to “true”: (Monot5)$$\begin{array}{ccc}& s(q,d)=\log p_{\text{T5}}(\text{"𝚝𝚛𝚞𝚎"}|\text{"𝚀𝚞𝚎𝚛𝚢: }q\text{ 𝙳𝚘𝚌𝚞𝚖𝚎𝚗𝚝: }d\text{ 𝚁𝚎𝚕𝚎𝚟𝚊𝚗𝚝:"}).& \text{<span class="tml-eqn"></span>}\end{array}$$ This seemingly wasteful reformulation is surprisingly effective: monoT5 achieves state-of-the-art reranking on MS MARCO with a single unified architecture.

The evolution of LTR.

The trajectory of LTR reveals a consistent pattern: each generation shifts more of the ranking intelligence from features to the model, trading hand-crafted inductive biases for learned representations. The result is systems that are more general but also more data-hungry and more opaque.

1. Feature-based LTR (2000s): Hand-crafted features + simple models (SVM, boosted trees).

2. Neural LTR (2015–2019): Learned representations + pairwise/listwise losses.

3. Pretrained LTR (2019–present): Large pretrained models (BERT, T5) fine-tuned for ranking.

4. LLM LTR (2023–present): Prompting LLMs for ranking without task-specific fine-tuning.

The monoT5 approach deserves further examination. By casting ranking as text generation, it enables a single T5 model to handle multiple tasks (passage ranking, document ranking, question generation) with different input-output templates. The model can also produce natural language explanations for its rankings by extending the output template, providing a form of ranking interpretability that classical LTR models lack entirely.

LLMs as Rankers

The latest paradigm shift in ranking is the use of large language models as zero-shot or few-shot rankers, with no task-specific training.

Prompting strategies for ranking.

Three prompting strategies have emerged, each mirroring one of the classical LTR paradigms:

1. Pointwise prompting: Ask the LLM to rate each document's relevance on a scale (e.g., 0–3). Simple but suffers from calibration issues: the LLM may assign different scores to identical documents depending on surrounding context.

2. Pairwise prompting: Present two documents and ask which is more relevant. More robust to calibration issues but requires $O(n^2)$ comparisons, which is expensive at LLM inference costs.

3. Listwise prompting: Present all candidates and ask the LLM to produce a ranking. Most efficient in terms of API calls but limited by context window length and subject to position bias (the order of candidates in the prompt affects the output ranking).

The choice among these strategies involves a three-way trade-off between cost (number of LLM calls), robustness (sensitivity to prompt formatting), and quality (ranking accuracy).

In-context learning for relevance assessment.

By providing a few examples of query-document-relevance triples in the prompt, LLMs can learn the relevance criteria for a specific domain. This is especially valuable for specialized domains (legal, medical, scientific) where labeled ranking data is scarce. The in-context examples serve as an implicit specification of what “relevance” means in the target domain. For instance, in legal search, “relevance” might mean “cites the same statute,” while in medical search it might mean “describes the same treatment for the same condition.” These nuances are difficult to encode in a training loss but easy to communicate through a few examples in a prompt.

Benchmarks and Evaluation

A retrieval system is only as good as the benchmarks used to evaluate it, and a benchmark is only as good as the community's understanding of its limitations. This section surveys the major benchmarks that have shaped retrieval research, introduces evaluation frameworks for the new generation of retrieval-augmented systems, and examines the cracks that are forming as benchmarks age.

BEIR: Heterogeneous Zero-Shot Evaluation

BEIR (Benchmarking IR) [104] assembles 18 diverse retrieval tasks spanning nine domains: bio-medical, financial, scientific, open-domain question answering, fact verification, citation prediction, duplicate question detection, argument retrieval, and entity retrieval. The datasets range from a few hundred to millions of documents, with query styles ranging from keyword queries to natural language questions to complex claims.

The critical design choice is zero-shot evaluation: models are evaluated on each task without any task-specific training data. This reflects the reality of deployed retrieval systems, which encounter new domains constantly. A legal search engine must handle new areas of law; a scientific search engine must index new subfields. A customer support search engine must handle new products. Zero-shot evaluation measures the robustness that matters most in practice: the ability to perform well on tasks the model was never explicitly trained for.

BEIR revealed a startling finding: dense retrieval models that achieved state-of-the-art performance on MS MARCO often performed worse than BM25 on out-of-domain tasks. Models that had been celebrated as breakthroughs were, in fact, overfitting to the idiosyncrasies of a single benchmark. This finding catalyzed a shift toward more robust training strategies: harder negative mining, multi-task training, and distillation from cross-encoders.

MS MARCO: The Workhorse of Passage Ranking

The Microsoft MAchine Reading COmprehension dataset [105] is the most widely used benchmark in passage retrieval. It consists of approximately 8.8 million passages from web documents, 500,000 training queries derived from Bing search logs, and sparse relevance judgments (typically one relevant passage per query).

Sparse judgments.

Each query in MS MARCO has, on average, only 1.1 relevant passages marked, despite the corpus containing 8.8 million passages. This means that the vast majority of relevant passages are unlabeled – they appear as negatives in evaluation, even though they may be perfectly relevant. This “false negative” problem biases evaluation: a model that retrieves a truly relevant but unlabeled passage is penalized. The bias is systematic: the better the model, the more unlabeled relevant passages it retrieves, and the more it is unfairly penalized. Re-assessment campaigns (such as TREC Deep Learning Track re-judging) have shown that top-ranked dense retrieval models find 30–50% more relevant passages than the original judgments contain.

Open development.

MS MARCO's open evaluation server has enabled rapid iteration. Over 3,000 submissions have been made to the leaderboard, with MRR@10 improving from 0.187 (BM25 baseline) to over 0.45 (modern dense retrievers with cross-encoder reranking). The open nature of the benchmark has been both its greatest strength (enabling fast progress) and its weakness (enabling overfitting through repeated evaluation).

KILT: Knowledge-Intensive Language Tasks

KILT [106] unifies five knowledge-intensive task families under a single benchmark and a single knowledge source (Wikipedia):

1. Fact checking: Given a claim, retrieve evidence and classify as supported, refuted, or not enough info (FEVER).

2. Entity linking: Given a mention in context, link it to the correct Wikipedia entity (AIDA CoNLL-YAGO).

3. Slot filling: Given an entity and relation, retrieve passages containing the answer (T-REx, zero-shot RE).

4. Open-domain QA: Given a question, retrieve passages and extract or generate the answer (Natural Questions, HotpotQA, TriviaQA).

5. Dialogue: Given a conversation history, retrieve relevant knowledge for the next response (Wizard of Wikipedia).

KILT's value lies in its unification: by using the same knowledge source across all tasks, it enables apples-to-apples comparison of retrieval quality across fundamentally different use cases. A retriever that excels at QA but fails at fact checking has a different failure mode than one that works for both. The unified knowledge source also simplifies experimentation: researchers can test a single retrieval pipeline across all five task families without corpus preprocessing overhead.

The tasks in KILT vary dramatically in their retrieval difficulty. Open-domain QA (Natural Questions) requires finding a single passage containing the answer – a relatively focused retrieval task. Entity linking requires matching a potentially ambiguous mention to the correct Wikipedia page among millions of candidates. Slot filling requires finding passages that contain a specific fact about a specific entity, which may be mentioned only in passing. This diversity makes KILT a stringent test of retrieval robustness.

MTEB: Massive Text Embedding Benchmark

MTEB [107] evaluates text embeddings across a comprehensive suite of tasks, extending far beyond retrieval. The benchmark includes eight task categories:

Category	Tasks	Description
Retrieval	15	Find relevant documents for a query
Classification	12	Classify text using embedding features
Clustering	11	Group similar texts by embedding proximity
Pair classification	3	Predict relationship between text pairs
Reranking	4	Reorder candidates by relevance
STS	10	Predict semantic similarity scores
Summarization	1	Evaluate summary quality via embeddings
Bitext mining	3	Find parallel sentences across languages

MTEB has become the de facto leaderboard for text embedding models. Its breadth is its strength: a model that achieves high scores across all eight categories produces genuinely general-purpose embeddings. However, this breadth also means that the aggregate score can mask significant weaknesses. A model with excellent retrieval performance but poor clustering ability may still rank highly overall.

Multimodal Retrieval Benchmarks

Retrieval is not limited to text. A growing family of benchmarks evaluates cross-modal retrieval: finding relevant items across modality boundaries. These benchmarks are essential for evaluating the multimodal embedding models (CLIP, ImageBind, and their successors) that power image search, video recommendation, and audio discovery systems.

Image-text retrieval.

MSCOCO [9] and Flickr30k are the standard benchmarks for image-text retrieval. Each image is paired with five human-written captions. Evaluation measures recall at $k$ (R@$k$) for both image-to-text and text-to-image retrieval. CLIP-based models dominate these benchmarks, with R@1 exceeding 80% on MSCOCO for text-to-image retrieval.

Video-text retrieval.

MSR-VTT [10] contains 10,000 video clips with 200,000 natural language descriptions. Video-text retrieval is substantially harder than image-text retrieval because the model must capture temporal dynamics: a description like “a man walks across the room and picks up a book” requires understanding the sequence of actions, not just individual frames.

Audio-text retrieval.

AudioCaps [13] provides captions for audio clips from the AudioSet ontology. Audio-text retrieval evaluates whether models can match sounds (“birds chirping in a forest”) to their descriptions. This is the least mature of the cross-modal retrieval tasks, with significant room for improvement.

Benchmark	Modality	Tasks	Primary Metric	Year
MS MARCO	Text	1	MRR@10	2016
BEIR	Text	18	nDCG@10	2021
KILT	Text	11	R-Precision	2021
MTEB	Text	59	Various	2023
MSCOCO	Image-text	2	R@1, R@5	2014
Flickr30k	Image-text	2	R@1, R@5	2014
MSR-VTT	Video-text	2	R@1, R@5	2016
AudioCaps	Audio-text	2	R@1, R@10	2019

RAG-Specific Evaluation

Retrieval-augmented generation introduces a new evaluation challenge: the quality of the final output depends on both the retrieval component and the generation component, and disentangling their contributions is non-trivial.

RAGAS framework.

The RAGAS framework [95] proposes component-wise evaluation for RAG systems:

1. Faithfulness: Does the generated answer use only information from the retrieved context? Measured by decomposing the answer into claims and checking each claim against the context.

2. Answer relevance: Does the answer address the question? Measured by generating synthetic questions from the answer and computing similarity to the original question.

3. Context precision: Are the retrieved documents relevant to the question? Measured by checking whether the answer can be derived from each retrieved document.

4. Context recall: Do the retrieved documents contain all information needed to answer? Measured by checking whether each sentence in the ground-truth answer is supported by the context.

Component-wise vs. end-to-end evaluation.

End-to-end evaluation (measuring only the final answer quality) is simpler but hides failure modes. A system might retrieve irrelevant documents but still produce a correct answer from the LLM's parametric knowledge. Conversely, a system might retrieve perfect documents but generate a hallucinated answer. Component-wise evaluation exposes these failure modes, enabling targeted improvement.

A Primer on Retrieval Metrics

Before examining benchmark limitations, we consolidate the evaluation metrics that appear throughout this chapter. The choice of metric encodes assumptions about what constitutes a good ranking, and different metrics can lead to contradictory conclusions about the same system.

Precision and recall at $k$.

For a ranked list of $k$ documents returned for query $q$: (PREC Recall)$$\begin{array}{ccc}& \text{P@}k=\frac{|\text{relevant docs in top-}k|}{k},\text{R@}k=\frac{|\text{relevant docs in top-}k|}{|\text{all relevant docs}|}.& \text{<span class="tml-eqn"></span>}\end{array}$$ Precision rewards systems that avoid returning irrelevant documents. Recall rewards systems that find all relevant documents. The tension between the two is fundamental: aggressive retrieval (returning many documents) increases recall at the expense of precision.

Mean reciprocal rank (MRR).

MRR measures how quickly the system returns the first relevant document: (MRR)$$\begin{array}{ccc}& \text{MRR}=\frac{1}{|Q|}\sum _{q\in Q}\frac{1}{{\text{rank}}_q},& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${\text{rank}}_q$ is the position of the first relevant document for query $q$. MRR is appropriate when users care primarily about the top result (e.g., navigational queries).

Normalized discounted cumulative gain (nDCG).

nDCG is the dominant metric for graded relevance: (NDCG)$$\begin{array}{ccc}& \text{nDCG@}k=\frac{\text{DCG@}k}{\text{IDCG@}k},\text{DCG@}k=\sum _{i=1}^k\frac{2^{y_{\pi (i)}}-1}{{\log }_2(i+1)},& \text{<span class="tml-eqn"></span>}\end{array}$$ where $y_{\pi (i)}$ is the relevance grade of the document at rank $i$ and IDCG@$k$ is the DCG of the ideal ranking. The logarithmic discount reflects the assumption that users are less likely to examine lower-ranked results. The exponential gain ($2^y-1$) gives disproportionate credit to highly relevant documents: a document with relevance 3 contributes $2^3-1=7$ times as much as a document with relevance 1.

Mean average precision (MAP).

MAP averages the precision at each relevant document position: (MAP)$$\begin{array}{ccc}& \text{AP}(q)=\frac{1}{|{\text{Rel}}_q|}\sum _{k=1}^n\text{P@}k\cdot \mathrm{𝟙}[d_k\in {\text{Rel}}_q],\text{MAP}=\frac{1}{|Q|}\sum _{q\in Q}\text{AP}(q).& \text{<span class="tml-eqn"></span>}\end{array}$$ MAP is appropriate for binary relevance and rewards systems that rank all relevant documents highly, not just the first one.

Benchmark Limitations

Every benchmark, no matter how carefully constructed, has a shelf life. Three forces conspire to erode the value of retrieval benchmarks over time.

Saturation.

As models improve, the gap between the best and worst models shrinks, and the benchmark ceases to discriminate meaningfully. On MS MARCO passage ranking, the gap between BM25 and the state of the art has narrowed from 28 nDCG@10 points in 2016 to under 10 points in 2024. On MSCOCO text-to-image retrieval, R@1 has exceeded 80%, leaving limited headroom. When benchmarks saturate, the community loses its ability to measure meaningful progress, and papers increasingly report improvements within the noise level of the evaluation methodology.

Contamination.

Training data leaking into test sets is a growing concern, especially for LLM-based systems. An LLM pretrained on the open web may have ingested MS MARCO queries, BEIR passages, or even the relevance judgments themselves. This makes it impossible to distinguish genuine generalization from memorization. The problem is especially acute for zero-shot evaluation: if the model has “seen” the test queries during pretraining, the evaluation is no longer zero-shot in any meaningful sense.

Fine-grained discrimination.

Most retrieval benchmarks use binary or coarsely graded relevance labels, which cannot distinguish between a “somewhat relevant” passage and a “perfectly relevant” one. MS MARCO's binary judgments (relevant vs. not relevant) are particularly limiting: they cannot evaluate whether a model retrieves the best passage or merely an acceptable one. This coarseness masks important quality differences between systems.

Consolidation with Prior Chapters

Throughout this book we have encountered retrieval in many guises, sometimes explicit and sometimes hidden beneath layers of abstraction. In this section we draw together the threads, demonstrating that retrieval is not merely one technique among many but a unifying lens through which the entire landscape of modern AI can be viewed. Every model that stores information and later uses it is, at its core, performing retrieval. The question is never whether a system retrieves, but how.

Generative Models as Retrieval Systems

The generative models surveyed in earlier parts of this book, including variational autoencoders, generative adversarial networks, and diffusion models, are conventionally understood as systems that create new data. We argue that they can equally be understood as systems that retrieve from a learned, continuous index.

VAE latent spaces as retrieval spaces.

A variational autoencoder (15) learns an encoder $q_{𝜽}(𝒛|𝒙)$ that maps data $𝒙$ to a latent code $𝒛\in {\mathrm{ℝ}}^d$ and a decoder $p_{𝜽}(𝒙|𝒛)$ that maps codes back to data. Viewed through the retrieval lens, the encoder is a query encoder that transforms an input into a representation, and the decoder is a generative retrieval system that, given a query in latent space, “retrieves” the most probable data point.

The ELBO objective that trains the VAE simultaneously optimises two retrieval-relevant properties: reconstruction quality (the retrieved item should match the query) and index regularity (the latent space should support smooth interpolation, analogous to a well-organized index that groups similar items nearby).

GAN discriminator features for retrieval.

Generative adversarial networks (9) train a generator $G$ and a discriminator $D$ in a minimax game. While $G$ receives most of the attention in the generation literature, the discriminator learns remarkably powerful feature representations as a byproduct of its training. The intermediate layers of $D$ capture hierarchical features that distinguish real from generated data, and these features transfer effectively to retrieval tasks.

Formally, let ${\varphi }_D^{(\ell )}(𝒙)$ denote the activation of the $\ell$-th layer of a trained discriminator. For retrieval, we define the similarity between a query $𝒒$ and a document $𝒅$ as (GAN SIM)$$\begin{array}{ccc}& s(𝒒,𝒅)=\sum _{\ell =1}^L{w}_{\ell }\cdot \cos ({\varphi }_D^{(\ell )}(𝒒),{\varphi }_D^{(\ell )}(𝒅)),& \text{<span class="tml-eqn"></span>}\end{array}$$ where $w_{\ell }$ are learned or hand-tuned weights. This multi-scale similarity captures both low-level texture matches (early layers) and high-level semantic similarity (deep layers).

Diffusion features (DIFT) for retrieval.

Perhaps the most surprising connection comes from diffusion models (19). The denoising process in a diffusion model requires the network to understand the data at every noise level, from coarse global structure (high noise) to fine local detail (low noise). Tang et al. showed that the intermediate features of a diffusion model, which they call DIFfusion FeaTures (DIFT), provide excellent representations for retrieval and correspondence tasks [60].

The reason diffusion features work so well for retrieval is deep: denoising IS retrieval. Given a noisy observation ${𝒙}_t$, the diffusion model must retrieve the clean data point ${𝒙}_0$ from its learned representation of the data distribution. The features learned in service of this implicit retrieval task transfer naturally to explicit retrieval.

Neural Architecture Search for Retrieval

The neural architecture search methods of 30 provide a natural toolkit for designing retrieval models. Rather than hand-crafting encoder architectures, we can search for architectures optimised for retrieval-specific objectives.

A particularly important design choice is the embedding dimension, which controls the trade-off between representational capacity and index size. NAS can discover architectures that achieve high retrieval quality at surprisingly low dimensions by learning to allocate capacity where it matters most.

Evolutionary AI and Retrieval

The evolutionary methods of 31 connect to retrieval in two ways: evolution as a tool for optimising retrieval systems, and retrieval as a component within evolutionary algorithms.

Evolving retrieval strategies.

The AlphaEvolve framework (31) demonstrated that LLM-guided program evolution can discover improvements to hash functions and scheduling algorithms. The same approach applies directly to retrieval: we can evolve scoring functions, index structures, and query expansion strategies.

Retrieval within evolutionary search.

Conversely, retrieval is a critical component within evolutionary AI. Quality-diversity algorithms like MAP-Elites maintain archives of diverse solutions, and finding the most similar archived solution to a new candidate is itself a retrieval problem. As archives grow to millions of entries, efficient approximate nearest-neighbour search becomes essential for scalable evolutionary computation.

Continual Learning and Retrieval

The continual learning methods of 26 face the fundamental challenge of catastrophic forgetting: when a model learns new tasks, it tends to overwrite the knowledge needed for old ones. Retrieval provides one of the most effective defences.

Privacy in Retrieval Systems

The membership inference attacks studied in 29 take on a distinctive character in retrieval systems. In a standard classifier, membership inference asks: “Was this sample in the training set?” In a retrieval system, the question becomes: “Is this document in the index?” and the answer is often trivially “yes” because the system returns it. The deeper privacy questions are more subtle.

The distinction matters because a document can be in the retrieval index without having influenced the encoder's training. If a document was in the training data, the encoder may produce an embedding that is unusually well-separated from non-relevant documents, a signal that an attacker can exploit.

Memory and Retrieval: The Duality

The memory architectures studied in 24 are, at their core, retrieval systems. The concepts of engrams, working memory, and long-term consolidation all have direct retrieval-theoretic interpretations.

An engram in neuroscience is the physical trace of a memory in the brain. In a neural retrieval system, the analogue is the embedding of a stored item: a distributed representation that encodes the item's content in a form amenable to similarity-based lookup. Working memory corresponds to the retrieval system's active cache: the small set of items currently in use, maintained in a fast-access buffer (analogous to an LLM's context window). Long-term memory corresponds to the persistent index, vast in capacity but slower to access.

The memory hierarchy as a retrieval hierarchy.

Modern AI systems increasingly employ hierarchical memory, from registers (attention heads), through caches (KV caches), to main memory (external datastores), to disk (web-scale indices). This hierarchy mirrors the memory hierarchy in computer architecture, but the principles are the same: frequently accessed items are kept close (low latency, limited capacity), while the long tail resides in slower but larger stores.

Alignment Through Retrieval

The alignment techniques of 25 can be reframed as retrieval problems. When we train a model to produce helpful, harmless, and honest responses, we are training it to retrieve appropriate responses from the space of all possible continuations.

The Grand Unification: AI as Retrieval

We now arrive at the central claim of this section: all of AI can be viewed through the retrieval lens.

This theorem reveals a profound truth: learning IS building a retrieval system. The training process takes raw experience and compiles it into an efficient index (the parameters $𝜽$) that supports fast retrieval at inference time. Gradient descent is the indexing algorithm. The loss function specifies the retrieval quality metric. Regularisation controls the index's capacity and generalisation properties.

Open Problems and Future Directions

The preceding sections have traced retrieval from its classical roots in library science through dense embeddings, cross-modal matching, and retrieval-augmented generation. Yet for all the progress, the most ambitious goals remain out of reach. In this section we survey the open problems that define the frontier of retrieval research, ranging from the engineering challenge of real-time updating to the philosophical question of whether retrieval and generation are ultimately the same thing.

Universal Retrieval

Current retrieval systems are specialists. CLIP retrieves images given text. ColBERT retrieves passages given queries. CLAP retrieves audio given descriptions. But no single model retrieves anything from any modality with human-level competence.

What would a truly universal embedding look like? The challenge is not merely engineering but information-theoretic. Different modalities carry different kinds of information: text is symbolic and compositional, images are spatial and textural, audio is temporal and spectral. Can a single embedding space faithfully represent all these informational structures without destructive interference?

Real-Time Updating

The world changes continuously, but retrieval indices do not. An LLM's knowledge is frozen at its training cutoff. A dense retrieval index must be rebuilt when new documents arrive. This temporal mismatch is one of the most pressing practical problems in retrieval.

The news problem.

Consider a question-answering system asked about an event that occurred one hour ago. The LLM has never seen this event. The retrieval index may not yet contain articles about it. Even if articles exist on the web, the system must crawl, index, and embed them before they become retrievable. The end-to-end latency from event occurrence to retrievability defines the system's temporal resolution.

Streaming indexing at scale.

Reducing $\mathrm{\Delta }t$ requires streaming indexing: continuously incorporating new documents into the index without full rebuilds. For sparse indices (BM25), this is relatively straightforward because the inverted index supports incremental updates. For dense indices, the challenge is more severe: adding a new document requires computing its embedding (a forward pass through a large model) and inserting it into an ANN index that may need periodic rebalancing.

Compositional Retrieval

Humans naturally compose retrieval queries from multiple concepts: “find me a sunset like this photo but over the ocean” or “a paper that combines the method from Smith et al. with the dataset from Lee et al.” Current retrieval systems struggle with such compositional queries because embeddings capture holistic similarity rather than compositional structure.

Addressing compositional retrieval requires moving beyond single-vector representations. Promising directions include multi-vector representations (like ColBERT's token-level embeddings), set-based retrieval where queries and documents are represented as sets of concept vectors, and neuro-symbolic approaches that decompose the query into a structured program executed against the index.

Causal Retrieval

Standard retrieval optimises for relevance: the retrieved documents should be topically related to the query. But relevance is not causation. In many applications, what we actually want is documents that would cause the right downstream outcome.

The distinction matters enormously for RAG systems. A document may be highly relevant to the query (high cosine similarity) but actually cause the LLM to produce a worse answer, perhaps because it contains plausible-sounding misinformation or because it confuses the model by introducing conflicting evidence. Causal retrieval asks: if we intervene by providing this document as context, does the answer improve?

Retrieval with Reasoning

Many real-world information needs require multi-step reasoning over multiple retrieved documents. “What is the GDP of the country where the inventor of the telephone was born?” requires retrieving the inventor (Alexander Graham Bell), his birthplace (Edinburgh, Scotland, UK), and the UK's GDP, a chain of three retrieval steps where each step depends on the result of the previous one.

Current multi-hop retrieval systems decompose such queries into sub-queries and retrieve iteratively, but they lack the ability to plan the retrieval strategy in advance or to backtrack when an intermediate retrieval leads to a dead end. Integrating retrieval with the planning and reasoning capabilities of LLMs is a key open challenge.

Privacy-Preserving Retrieval

Can we search an index without revealing the query? Can we build an index without revealing the documents? These questions define the field of privacy-preserving retrieval, which sits at the intersection of information retrieval and cryptography.

Classical PIR protocols work for exact retrieval by index, but extending them to similarity-based retrieval (where the server must compute distances without knowing the query) is extremely challenging. Homomorphic encryption allows computation on encrypted data but incurs orders-of-magnitude overhead. Secure multi-party computation distributes the index across non-colluding servers. Both approaches remain far too slow for web-scale retrieval.

Retrieval for Embodied Agents

A robot operating in a kitchen has accumulated hours of experience: grasping mugs, pouring liquids, opening cabinets. When confronted with a new task (e.g., “make coffee”), it must retrieve relevant past experiences and adapt them to the current situation. This is episodic retrieval in the sense of Tulving [36]: retrieving specific episodes from autobiographical memory, not just semantic facts.

The challenges of embodied retrieval include the high dimensionality of experience (video, proprioception, force feedback), the need for temporal alignment (the relevant part of an episode may be a brief sub-sequence), and the requirement for real-time operation (the robot cannot pause for minutes while searching its memory).

The Generation-Retrieval Spectrum

Is generation just retrieval from an implicit, continuous index? Is retrieval just constrained generation? As models grow more powerful, the boundary between generation and retrieval dissolves.

A generative model like GPT-4 can be viewed as a retrieval system over its training data, where the “retrieval” is mediated by billions of parameters rather than an explicit index. Conversely, a retrieval system that returns a verbatim passage is performing a degenerate form of generation: it “generates” output by copying from its index.

This duality suggests that the “right” point on the generation-retrieval spectrum depends not on fundamental capability differences but on practical considerations: latency, controllability, attribution, freshness, and privacy.

Frontier Open Problems

We conclude with a collection of open problems that push the boundaries of what retrieval could become.

Predictive retrieval: retrieving from the future.

Can a retrieval system anticipate what information a user will need before they ask for it? This is not as fantastical as it sounds. Predictive caching in web systems pre-fetches pages that users are likely to visit. Extending this to semantic retrieval requires modelling user intent trajectories: given the sequence of past queries, predict the next information need and pre-retrieve relevant documents.

Biological neural retrieval.

Can biological neural networks be queried as retrieval systems? Neuroscientists can now read neural activity patterns with increasing fidelity using fMRI, EEG, and neural probes. If we view the brain's neural code as an embedding space, can we perform nearest-neighbour retrieval against it? That is, given a stimulus, can we find the stored memory whose neural representation is most similar? Early work on neural decoding (reconstructing images from brain activity) suggests that the answer is tentatively “yes,” but the resolution and speed of current measurement techniques remain severe bottlenecks.

Thermodynamic limits of retrieval.

What is the minimum energy required for one retrieval operation? Landauer's principle establishes that erasing one bit of information requires at least $k_{B}T\ln 2$ joules of energy, where $k_B$ is Boltzmann's constant and $T$ is temperature. Retrieval does not erase information, but it does require reading it.

Self-improving retrieval.

Can we build a retrieval system that improves itself through use? Every query provides implicit feedback about the index structure: queries that return good results validate the current embeddings, while queries that fail reveal deficiencies. A truly self-improving system would use this feedback to continuously refine its embeddings, re-organize its index, and identify gaps in its corpus, all without human intervention.

Historical Timeline

The history of retrieval stretches from the clay tablets of ancient Sumeria to the multimodal embedding models of the present day. In this section we trace the key milestones, organised into five eras that reflect fundamental shifts in how humans and machines find information.

Era I: Manual Retrieval (Antiquity–1945)

Era II: Classical Information Retrieval (1945–2000)

Era III: Learning to Rank (2000–2018)

Era IV: Dense Retrieval Revolution (2018–2022)

Era V: Generative Retrieval and Beyond (2022–Present)

Exercises

Standard Exercises

Challenge Problems

Chapter Summary

This chapter has told a story that spans millennia. It began with Plato's doctrine of anamnesis, the idea that all learning is recollection, that to know something is to retrieve it from the soul's prior acquaintance with the Forms. It ends with Gemini's multimodal embeddings, where a single model maps text, images, audio, and video into a shared geometric space in which “retrieval” is a nearest-neighbour lookup and “knowledge” is a point in ${\mathrm{ℝ}}^{3072}$.

Between Plato and Gemini lies a remarkable intellectual journey. Let us gather the key lessons.

From keywords to meaning.

The first great transition in retrieval was from lexical matching to semantic understanding. TF-IDF and BM25 retrieve documents that share words with the query. Dense retrievers like DPR and ColBERT retrieve documents that share meaning with the query, even when no words overlap. This transition was enabled by the convergence of three forces: large-scale pre-training (BERT), the contrastive learning framework (InfoNCE), and hardware capable of encoding billions of documents into dense vectors.

From unimodal to multimodal.

The second great transition was from text-only retrieval to cross-modal retrieval. CLIP showed that a shared embedding space could bridge vision and language. Subsequent models extended this to audio (CLAP), video, 3D shapes, code, and scientific data. The lesson is that meaning is modality-independent: a sunset looks the same whether you describe it in words, photograph it, or paint it, and a good embedding captures this invariance.

From retrieval to augmented generation.

The third great transition was the fusion of retrieval with generation. RAG showed that a retriever plus a generator is more than the sum of its parts: the retriever provides factual grounding, the generator provides fluent synthesis, and together they produce responses that are both accurate and natural. This fusion is not merely a trick but a reflection of a deep duality: retrieval and generation are two sides of the same coin, two strategies for transforming a query into a response.

From static to dynamic.

The fourth transition, still underway, is from static indices to living, adaptive retrieval systems. Future systems will update their indices in real time, learn from every query, and anticipate information needs before they are expressed. They will be not passive databases but active cognitive agents, reasoning about what to retrieve, when to retrieve it, and how to integrate it with existing knowledge.

The convergence of retrieval and generation.

Perhaps the deepest insight of this chapter is that retrieval and generation are converging. A generative model that has memorised its training data is performing implicit retrieval. A retrieval system that interpolates between stored items is performing implicit generation. The generation-retrieval duality (Proposition 45) makes this precise: as both systems approach the true data distribution, they become indistinguishable.

The practical consequence is that the choice between retrieval and generation is not a matter of fundamental capability but of engineering trade-offs: attribution (retrieval makes provenance explicit), freshness (retrieval accesses current information), efficiency (retrieval avoids redundant computation), and creativity (generation can produce novel combinations). The most powerful systems will seamlessly combine both, operating at whatever point on the retrieval-generation spectrum best serves each query.

Key takeaways by theme.

We organise the chapter's central lessons into five themes:

1. Representation: the quality of retrieval is determined by the quality of the embedding. Contrastive learning, multi-vector representations, and Matryoshka embeddings represent three strategies for learning better representations, each with distinct trade-offs in capacity, granularity, and computational cost.

2. Efficiency: web-scale retrieval demands sub-millisecond latency over billions of items. The ANN algorithms (IVF, HNSW, product quantisation) make this possible by trading exact correctness for speed, and the art lies in controlling the recall loss.

3. Integration: retrieval is most powerful when integrated with generation (RAG), reasoning (multi-hop retrieval), and planning (agentic retrieval). The system that retrieves must also know when to retrieve, what to retrieve, and how to use what it finds.

4. Evaluation: retrieval metrics (nDCG, MAP, recall) measure different aspects of quality. No single metric suffices, and the choice of metric shapes the system's behaviour through the optimisation objective.

5. Universality: the boundaries between modalities, between retrieval and generation, and between memory and computation are dissolving. The trajectory of the field points towards universal systems that transcend these distinctions.

A philosophical reflection.

In the age of large language models, what does it mean to “know” something? If a model can retrieve the answer to any factual question, but stores none of the answers in its parameters, does it “know” the facts? Conversely, if a model has memorised a fact in its weights but cannot access it without the right prompt, does it “know” the fact?

Tulving's distinction between semantic memory (knowing that Paris is the capital of France) and episodic memory (remembering learning this fact) [36] offers a useful framework. A generative model has semantic knowledge compiled into its parameters. A retrieval-augmented model has episodic access to a corpus of specific documents. Neither alone captures the full richness of human knowledge, which integrates semantic understanding with episodic memory, contextual reasoning, and the ability to recognise the limits of one's own knowledge.

The retrieval systems of the future will move closer to this integration: not just finding relevant documents but understanding why they are relevant, how they relate to what is already known, and what further information is needed. In this sense, the story of retrieval is the story of intelligence itself: the ongoing quest to bridge the gap between what we know and what we need to know.

[heading=subbibliography]