26 Continual Learning

A child learns to ride a bicycle at age five, surviving dozens of falls, scraped knees, the gradual discovery that balance is a continuous feedback loop between vestibular sense and motor command. At sixteen, the same child learns to drive a car; at thirty, to fly a small aircraft. Each new competence builds on prior motor and cognitive knowledge without destroying it: the bicycle skill remains intact through decades of subsequent learning, and sub-skills (steering, judging distance) transfer seamlessly across domains.

Now consider a standard neural network. Train it on Task A until convergence; then train it on Task B. Upon re-evaluation on Task A, performance has collapsed, often to chance level. The weights that encoded Task A have been overwritten by the gradient updates for Task B. This phenomenon, known as catastrophic forgetting [1][2], is the central obstacle to building learning systems that accumulate knowledge across a lifetime.

Continual learning (also called lifelong learning, incremental learning, or sequential learning) is the study of algorithms that close this gap: methods that enable a single model to learn from a non-stationary stream of tasks while preserving previously acquired knowledge and transferring it to accelerate future learning.

This chapter culminates the book's arc. Parts II–VI developed generative modelling (VAEs, flows, diffusion, GANs, autoregressive LMs); Part VII studied reasoning; Part VIII (24) examined memory management; Part IX (25) addressed alignment. Continual learning asks the unifying question: can a single model do all of these (generate, reason, remember, and align) across a sequence of tasks, domains, and time?

Prerequisites.

We assume familiarity with the alignment pipeline (fig:alignment:pipeline), LoRA (Parameter-Efficient Fine-Tuning: LoRA, Definition 2), and the KL-constrained objective ((Central Objective)). From the memory chapter: memory systems (Definition 20), the memory framework (The Memory System Framework), and SSMs as implicit memory (Proposition 10). From the reasoning chapter: Act (Adaptive Computation Time) and self-refinement (Self-Refine).

Why Continual Learning?

The question “why continual learning?” admits answers at multiple levels: philosophical, cognitive, biological, and engineering. We address each in turn, building motivation for the formal framework of Mathematical Foundations.

Philosophical Roots

Heraclitus of Ephesus observed that “no man ever steps in the same river twice, for it is not the same river and he is not the same man.” Both the learner and the environment are in constant flux. A learning system that assumes a fixed data distribution, the standard $\stackrel{\mathrm{iid}}{\sim }$ assumption, is a convenient fiction, inadequate for the real world where distributions shift and new categories emerge.

Consider the forms of knowledge a learning system might possess:

1. Weight-encoded knowledge (parametric memory): the information stored in the model's weights through gradient-based training. This is slow to acquire, expensive to modify, and accessed implicitly during every forward pass. In the terminology of 24, this is parametric memory.

2. Retrieval-based knowledge (external memory): the information stored in databases, documents, or memory banks that the model can query at inference time. This is fast to update (insert a new document), cheap to scale (add more storage), but requires a retrieval mechanism. In 24 we studied retrieval-augmented generation and memory-augmented architectures.

3. Architectural capacity (innate structure): the inductive biases encoded in the architecture itself (attention patterns, positional encodings, layer normalization) which determine what the model can learn, independent of what it has learned.

Continual learning operates primarily at the first level, updating weight-encoded knowledge without destroying prior knowledge, but the most effective approaches combine all three, using architectural innovations for separate capacity and retrieval-based memory for replay.

What Humans Retain and Forget

Human memory follows quantitative laws studied for over a century. Understanding these laws provides both inspiration and benchmarks for artificial continual learning.

A power law often provides a better fit over long time scales [38]: (Power LAW Forgetting)$$\begin{array}{ccc}& M(t)=a{t}^{-b},& \text{<span class="tml-eqn"></span>}\end{array}$$ where $a>0$ is a scale and $b>0$ the decay rate. The power law decays more slowly for large $t$, reflecting that well-learned memories resist forgetting (Jost's law).

Procedural vs. declarative memory.

Declarative memory stores facts and events; procedural memory stores skills and habits. Procedural memories are more resistant to forgetting, suggesting different consolidation mechanisms. In neural network terms: declarative $\approx$ retrievable knowledge in external memory; procedural $\approx$ weight-encoded computations.

The Biological Learning System

The mammalian brain is the most successful continual learning system we know of. Its architecture provides a blueprint to be abstracted into computational principles.

The CLS theory explains why humans avoid catastrophic forgetting: the hippocampus buffers new experiences, which are then interleaved with replayed old experiences during sleep consolidation, allowing the neocortex to accommodate both.

Hebbian learning and synaptic consolidation.

At the synaptic level, learning follows Hebb's rule: “neurons that fire together wire together.” Crucially, synapses that have been repeatedly strengthened become less modifiable; they are “consolidated.” This synaptic consolidation directly inspires EWC (Elastic Weight Consolidation) and SI (Synaptic Intelligence), which protect important parameters from large updates.

Mathematical Foundations

We now formalize the continual learning problem, establish its Bayesian interpretation, quantify the stability–plasticity trade-off, and derive generalization bounds for sequential learning.

The Continual Learning Problem

The literature distinguishes several scenarios based on test-time information.

The standard evaluation metrics are as follows.

Bayesian Sequential Learning

The Bayesian perspective provides the theoretical gold standard: what an ideal learner should do, even if exact computation is intractable.

The Stability–Plasticity Dilemma

Every continual learning method navigates the tension between learning new information and retaining old knowledge.

Generalization Bounds for Continual Learning

We now derive generalization bounds accounting for sequential learning and inter-task transfer.

Proposition 2 is the theoretical foundation of EWC: parameters with high Fisher information should be changed as little as possible.

Catastrophic Forgetting

We now study catastrophic forgetting: its causes, formal measures, and the taxonomy of methods designed to combat it.

What Causes Forgetting

Three mechanisms drive catastrophic forgetting:

1. Weight overwriting.

Gradient descent on task $B$ modifies the same weights that encode task $A$. Since all inputs share the parameter space (the hallmark of distributed representations), any weight update for $B$ perturbs the function computed for $A$.

2. Activation drift.

Even if final-layer weights for task $A$ are frozen, changes to earlier layers alter intermediate representations. If ${\varphi }_t(𝒙)$ denotes the features after task $t$, training on task $t+1$ changes them to ${\varphi }_{t+1}(𝒙)$, and the task-$t$ classifier $w_t^{\top }{\varphi }_t(𝒙)$ evaluated on ${\varphi }_{t+1}(𝒙)$ produces arbitrary outputs.

3. Distribution shift.

Continual learning inherently violates the $\stackrel{\mathrm{iid}}{\sim }$ assumption: the distribution $p_t(𝒙,y)$ may differ substantially from $p_s(𝒙,y)$ for $s\ne t$. Standard generalization guarantees do not apply across tasks.

Measuring Forgetting

A Taxonomy of Continual Learning Methods

We organize continual learning methods into three families.

The Memory–Refinement Boundary

Not all performance changes after learning a new task constitute forgetting; some are beneficial refinement. We decompose the loss change to distinguish the two.

Loss decomposition.

Consider the change in loss on task $t$ after training on task $t+1$: (LOSS Decomposition)$$\begin{array}{ccc}& \mathrm{\Delta }{ℒ}_t={ℒ}_t({𝜽}_{t+1}^{\ast })-{ℒ}_t({𝜽}_t^{\ast })=\underset{\text{representation improvement}}{\underbrace{\mathrm{\Delta }{ℒ}_{\mathrm{refine}}}}+\underset{\text{new capability cost}}{\underbrace{\mathrm{\Delta }{ℒ}_{\mathrm{new}}}}+\underset{\text{catastrophic forgetting}}{\underbrace{\mathrm{\Delta }{ℒ}_{\mathrm{forget}}}}.& \text{<span class="tml-eqn"></span>}\end{array}$$ Here $\mathrm{\Delta }{ℒ}_{\mathrm{refine}}\le 0$ captures beneficial representation improvement; $\mathrm{\Delta }{ℒ}_{\mathrm{new}}\ge 0$ is the irreducible cost of accommodating a new task; and $\mathrm{\Delta }{ℒ}_{\mathrm{forget}}\ge 0$ is the avoidable loss from uncontrolled weight overwriting.

Regularization-Based Methods

Regularization-based methods add penalty terms to the loss that discourage large changes to parameters important for previous tasks. They require no stored data (or only a small buffer for gradient constraints) and leave the architecture unchanged, making them attractive for memory-constrained settings.

Elastic Weight Consolidation

Elastic Weight Consolidation (EWC) [3] uses the Fisher information matrix to identify parameters important for previous tasks and penalizes changes to those parameters.

The justification comes from the Laplace approximation to the Bayesian posterior.

Synaptic Intelligence

Synaptic Intelligence (SI) [31] accumulates importance estimates online during training via a path integral, rather than computing them as a snapshot at the optimum.

Learning without Forgetting

Learning without Forgetting (LwF) [32] uses knowledge distillation [17] to preserve the model's outputs on new-task data rather than protecting parameters directly.

The old model's predictions on new data encode information about what it has learned; matching these predictions preserves behavior without stored exemplars.

Gradient Episodic Memory

Gradient Episodic Memory (GEM) [33] directly constrains gradient updates to avoid increasing the loss on previous tasks.

[16] proposed Averaged GEM (A-GEM), replacing per-task constraints with a single averaged constraint $⟨\widetilde{𝒈},\overline{𝒈}⟩\ge 0$ where $\overline{𝒈}=\frac{1}{t-1}{\sum }_{k<t}{𝒈}_k$, reducing the QP to one projection.

Comparison of Regularization Methods

Table 1 summarizes the four methods.

Method	Importance	Data from past	Per-task memory	Connection
EWC	Fisher (snapshot)	No	$O(d)$ diagonal	Laplace approx.
SI	Path integral	No	$O(d)$ accumulators	Online importance
LwF	Output-based	No (distillation)	$O(1)$ (old model)	KD / alignment
GEM	Gradient-based	Yes (memory buffer)	$O(Mn)$ exemplars	Constrained opt.

Replay-Based Methods

Regularisation methods (Regularization-Based Methods) protect old knowledge by penalising parameter drift; they never revisit old data. A complementary family of methods takes the opposite approach: store (or regenerate) representative samples from previous tasks and interleave them with new-task data during training. This strategy, experience replay, draws direct inspiration from the Complementary Learning Systems (CLS) theory [11], which posits that the mammalian brain consolidates episodic memories by replaying hippocampal traces into the neocortex during sleep.

The central trade-off is memory versus fidelity: a larger replay buffer yields better retention but consumes more storage and compute. We develop the mathematical framework for buffer-based replay, then examine three important variants: generative self-replay (LAMOL), gradient-constrained replay (A-GEM), and replay at the LLM scale.

Experience Replay

The simplicity of (ER LOSS) belies its effectiveness. Empirically, even a small buffer ($m\approx 500$) can reduce catastrophic forgetting (Definition 9) by 50–70% on standard benchmarks, often matching or exceeding regularisation-based methods [16].

Forgetting analysis.

Replay reduces forgetting because the gradient on replayed samples counteracts the drift induced by new-task gradients. More precisely, if $g_t={\nabla }_{𝜽}\ell (𝜽;{𝒟}_t)$ and $g_{ℳ}={\nabla }_{𝜽}\ell (𝜽;ℳ)$, the effective update direction is $g_t+\alpha g_{ℳ}$. When $⟨g_t,g_{ℳ}⟩<0$ (conflicting gradients), the replay term pulls the update back toward the old optimum, directly mitigating the mechanism underlying catastrophic forgetting (Proposition 2).

LAMOL: Language Model as Replay

Storing raw training data raises practical and legal concerns: the buffer size is fundamentally limited, and data-retention policies may prohibit keeping old samples. [15] and later [14] proposed an elegant alternative: use the language model itself as a generative replay mechanism.

A-GEM and Variants

Gradient Episodic Memory (GEM) [33] uses the replay buffer not for data replay but for gradient constraints: the update direction must not increase the loss on any previous task. For $T$ tasks, this requires solving a quadratic programme with $T$ constraints at every gradient step, an $O(T^2)$ cost that is prohibitive for long task sequences.

The constraint $⟨\widetilde{𝒈},{\overline{𝒈}}_{\mathrm{ref}}⟩\ge 0$ ensures that the update does not increase the average loss over previously seen tasks. This is weaker than GEM's per-task guarantees, but empirically A-GEM performs comparably while scaling to hundreds of tasks [16].

Replay for Large Language Models

(Optimal Buffer) reveals the fundamental tension: the statistical term scales as $\sqrt{T/m}$, so the buffer must grow linearly with the number of tasks to keep forgetting constant. When $m$ is fixed, forgetting grows as $\sqrt{T}$, inevitable unless the task-shift term is simultaneously controlled (e.g., via regularisation or architecture-based methods).

Architecture-Based Methods

Regularisation constrains what the parameters can do; replay constrains what the model sees. Architecture-based methods take a third path: they modify the structure of the network so that different tasks occupy different parameters. The ideal is zero forgetting: old tasks are literally untouched because their weights are frozen or isolated, at the cost of growing model capacity.

We survey four families of increasing sophistication: progressive networks that add entire columns, mask-based methods that carve task subnetworks out of a shared backbone, LoRA-based methods that attach lightweight adapters, and hypernetwork-based methods that generate adapters on the fly.

Progressive Neural Networks

PackNet and Piggyback

Progressive networks guarantee zero forgetting but at quadratic parameter cost. Mask-based methods achieve a more favorable trade-off by sharing a single set of weights and assigning each task a binary mask that selects a task-specific subnetwork.

The capacity bound (Packnet Capacity) is surprisingly tight in practice: with $p=0.25$ (retaining the top 25% of weights per task), PackNet accommodates exactly 4 tasks. This motivates the search for more parameter-efficient alternatives.

LoRA for Continual Learning

The parameter-efficiency revolution brought by Low-Rank Adaptation (Definition 2, Parameter-Efficient Fine-Tuning: LoRA) offers a natural architecture-based strategy for continual learning: freeze the pretrained backbone and learn a small, separate LoRA adapter for each task.

Doc-to-LoRA

Continual LoRA (Definition 23) requires training a separate adapter for each task, which still takes time proportional to the fine-tuning dataset size. Can we amortise this cost by learning a single hypernetwork that maps any document (or task specification) to its corresponding LoRA adapter in a single forward pass?

Continual Compositionality

A single LoRA adapter per task treats each task as an independent modular unit. But many tasks share substructure: a medical question-answering task might benefit from both “medical knowledge” and “reasoning” skills. Compositional methods aim to learn a library of skills and compose them at inference time.

Continual Learning for Large Language Models

The preceding sections developed continual learning as a general framework. We now turn to the specific challenges posed by large language models (LLMs), which are continual learners by economic necessity: retraining a frontier model from scratch costs millions of dollars in compute, yet the world's knowledge changes daily. Continual pre-training, continual fine-tuning, and continual alignment are no longer academic curiosities; they are production requirements.

This section surveys the LLM-specific landscape, organised around four themes: continual pre-training, learning-rate dynamics, sparse memory, and continual fine-tuning. We draw on the comprehensive surveys of [7], [6], and [8].

Continual Pre-Training

Learning Rate Governs Forgetting

The learning rate is perhaps the single most important hyperparameter for controlling the stability–plasticity trade-off in continual learning. A large learning rate provides plasticity (fast adaptation to new data) but causes instability (large deviations from the old optimum). A small learning rate preserves stability but may lack the capacity to learn genuinely new knowledge. The following theorem makes this trade-off precise.

Theorem 5 has a clear prescriptive message: to reduce forgetting, one should use a small learning rate ${\eta }_B$ (scaling linearly with forgetting), train on tasks whose gradients are small at the previous optimum (low task conflict), and prefer models with well-conditioned loss landscapes (small $\kappa (H_A)$). The connection to EWC (Definition 12) is immediate: EWC's Fisher penalty reshapes the loss landscape to increase the effective curvature along directions important for task $A$, making $\kappa (H_A)$ act in the model's favour.

Sparse Memory Fine-Tuning

An orthogonal approach to mitigating forgetting during LLM fine-tuning is to replace the dense feed-forward network (FFN) layers with sparse memory layers that can absorb new knowledge without overwriting old parameters.

Continual Fine-Tuning

LLM Forgetting in Practice

Neural Memory for Continual Learning

The previous sections treated continual learning as a problem to be solved, a pathology (forgetting) to be cured by external interventions (regularisation, replay, architecture tricks). A more radical perspective, developed in this section, is that continual learning is not a bug but a feature: the ability to update internal state in response to new information is precisely what memory means. From this viewpoint, a model that cannot continually learn cannot truly remember.

This section surveys four recent lines of work that embed continual learning directly into the architecture as a form of neural memory: Titans (meta-learned memory modules), Miras (a unifying framework connecting attention and online learning), S6MOD (state-space models with mixture of discretisation), and nested learning (hierarchical memory with multiple timescales). These connect to the memory systems developed in Memory-Augmented Architectures and The Memory System Framework, but with the specific focus on learning within the forward pass itself.

Titans: Neural Long-Term Memory

[35] propose augmenting the Transformer with a neural long-term memory module that learns to store and retrieve information within the forward pass, using gradient descent on a surprise-based loss as the write mechanism.

The Miras Framework

The proliferation of sequence models (Transformers, linear attention, state-space models, recurrent networks) raises a natural question: are these architectures fundamentally different, or are they variations on a common theme? The Miras framework [36] answers decisively: they are all instances of the same computation, differing only in their memory policy.

S6MOD: State-Space Models with Mixture of Discretisation

State-space models (SSMs) like Mamba (Definition 19) process sequences through a fixed-dimensional recurrent state, which acts as a compressed memory of the past (Proposition 10). A limitation of standard SSMs is that a single discretisation step $\mathrm{\Delta }$ imposes a single temporal resolution: the model either captures fine-grained recent detail or coarse long-range dependencies, but not both simultaneously.

Nested Learning and the Continuum Memory System

The CLS theory [11] postulates two complementary learning systems: a fast hippocampal system and a slow neocortical system. But the brain does not operate with just two timescales; it has synaptic, cellular, circuit-level, and systems-level plasticity, each operating at a different rate. [37] propose nested learning, which generalises CLS from two levels to $L$ levels, creating a continuum of memory timescales.

Connection to existing architectures.

The Miras framework (Definition 32) can be viewed as implementing a two-level nested system: the attention weights ${\alpha }_{t,s}$ provide fast, input-dependent memory access (level 1), while the decay factor $\lambda$ implements slow forgetting (level 2). Titans (Definition 31) implements a three-level system: core attention (fast), neural memory (medium), and persistent memory (slow). S6MOD (Definition 33) implements $J$-level memory within the SSM framework via multiple discretisation heads. The nested learning perspective unifies all of these as instances of the same principle: effective memory requires multiple timescales.

Continual Learning for Diffusion Models

Diffusion models have become the dominant paradigm for image, video, and audio generation. Yet the very property that makes them powerful, the ability to learn rich, multi-scale representations of data distributions, also makes them acutely susceptible to forgetting. A Stable Diffusion model fine-tuned on portrait photographs may lose the ability to generate mountain landscapes; a music diffusion model trained on jazz may forget how to produce classical compositions. The denoising objective, which operates across a continuum of noise levels $t\in [0,T]$, creates a high-dimensional forgetting surface that is qualitatively different from the scalar or per-class forgetting studied in classification.

This section develops the mathematics of continual learning for diffusion models, drawing on the foundations of Regularization-Based Methods, Architecture-Based Methods, and the alignment perspective of Alignment for Diffusion Models.

Forgetting in Diffusion Models

In classification, forgetting is measured by the drop in accuracy on previously learned classes. For generative models, we must measure distributional divergence. The Fréchet Inception Distance (FID) provides a practical, if imperfect, metric.

The denoising objective of a diffusion model operates across noise levels $t$, and different noise levels encode different types of information. High noise levels ($t$ close to $T$) capture coarse, global structure, such as the overall layout and colour distribution of an image. Low noise levels ($t$ close to $0$) capture fine details: textures, edges, and precise spatial relationships. This hierarchy has direct consequences for forgetting: a model that forgets its high-noise predictions loses the ability to generate globally coherent compositions, while a model that forgets its low-noise predictions produces blurry or textureless images.

The noise-level hierarchy creates a natural connection to the HSD framework (Hierarchical Sparse-Dense Framework): we can view different noise levels as different “levels” of the hierarchy, with high-noise levels corresponding to the slow, stable, shared level (since coarse structure is similar across tasks) and low-noise levels corresponding to the fast, task-specific level (since fine details vary dramatically between, say, landscapes and portraits). This perspective motivates the noise-level-aware EWC approach developed below.

Regularization for Diffusion Continual Learning

The EWC framework (Definition 12) applies to any differentiable loss. For diffusion models, the loss is the denoising score-matching objective (Denoise LOSS)$$\begin{array}{ccc}& {ℒ}_{\mathrm{denoise}}(𝜽)={𝔼}_{t\sim 𝒰[0,T]}{𝔼}_{{𝒙}_0\sim p_{\mathrm{data}}}{𝔼}_{𝝐\sim 𝒩(\mathrm{𝟎},𝐈)}[w(t){\Vert {𝝐}_{𝜽}({𝒙}_t,t)-𝝐\Vert }^2],& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${𝒙}_t=\sqrt{{\bar{\alpha }}_t}{𝒙}_0+\sqrt{1-{\bar{\alpha }}_t}𝝐$ and $w(t)$ is a noise-level weighting. The Fisher information matrix inherits this noise-level structure.

LoRA-Based Diffusion Continual Learning

The parameter-efficient approach of Definition 23 is particularly natural for diffusion models, where each “task” often corresponds to a visual concept (a particular artistic style, a specific object category, or a user's face).

The composition in (Concept Compose) raises a subtle mathematical question: when do linearly composed LoRA adapters produce coherent samples? If concepts $c_1$ (“oil painting”) and $c_2$ (“portrait”) are trained independently, their composition ${\gamma }_1{B}_1{A}_1+{\gamma }_2{B}_2{A}_2$ may interfere destructively. This connects directly to the compositionality analysis of Definition 25.

In practice, adapters trained independently are not exactly orthogonal, but their interference can be mitigated by explicit orthogonality regularisation during training: adding a penalty $\mu {\sum }_{c_1<c_2}{\Vert A_{c_1}^{\top }{B}_{c_1}^{\top }{B}_{c_2}{A}_{c_2}\Vert }_F^2$ to the training loss encourages the adapters to occupy distinct subspaces of the model's representation. This creates a direct connection between the compositionality of diffusion LoRA adapters and the orthogonal subspace methods studied in the architecture-based CL literature.

Connection to Alignment

The alignment techniques of Alignment for Diffusion Models (DDPO, DRaFT, and Diffusion-DPO) can be viewed through the continual learning lens. Each alignment method fine-tunes a pretrained diffusion model to satisfy human preferences, which is precisely a two-task continual learning problem: task 0 is “generate diverse, high-quality images” and task 1 is “generate images that humans prefer.”

Multimodal and Federated Continual Learning

The preceding sections studied continual learning for unimodal systems, language models that process text, or diffusion models that generate images. Modern AI systems, however, are fundamentally multimodal: CLIP aligns vision and language, GPT-4V processes interleaved text and images, and Gemini handles text, images, audio, and video. When such systems must learn continually, the challenges multiply. Forgetting can propagate across modalities: updating the vision encoder may degrade text-conditioned generation even if the text encoder is untouched.

Simultaneously, the deployment of large models across many devices and organisations creates a federated learning setting where data cannot be centralised. When each client also faces a non-stationary data stream, we arrive at federated continual learning, a doubly challenging problem that combines the difficulties of both fields.

Continual CLIP: Vision-Language Continual Learning

The central finding of C-CLIP is striking:

MultiFCL: Federated Continual Learning

Cross-Modal Continual Learning

The most insidious form of multimodal forgetting is cross-modal: updating one modality's parameters causes degradation in another modality's performance, mediated by the shared embedding space.

A Unified Mathematical Framework

We have now surveyed a wide landscape of continual learning methods: regularization-based approaches that penalize movement in parameter space (Regularization-Based Methods), replay-based methods that maintain a buffer of past experiences (Replay-Based Methods), architecture-based methods that allocate dedicated capacity per task (Architecture-Based Methods), and neural memory systems that blur the line between architecture and learning rule (Neural Memory for Continual Learning). These methods were developed by different communities with different motivations, yet they share deep structural similarities.

This section develops a unified mathematical framework that captures all known CL methods as special cases. The framework reveals why certain combinations of design choices succeed and others fail, and it points toward the design of future systems that may overcome the limitations of any single approach.

The Lifelong Learning System

We begin by defining the most general notion of a continual learner.

Information-Theoretic View

The fundamental tension of continual learning, fitting new tasks without forgetting old ones, is, at its core, a tension between information storage and information overwriting. We can make this precise using information theory.

Hierarchical Sparse-Dense Framework

The most powerful unifying abstraction for CL methods is the hierarchical sparse-dense (HSD) framework, which decomposes a continual learner into levels that differ in their density of update, speed of change, and degree of task-specificity.

Connecting Methods to the HSD Framework

The power of the HSD framework lies in its ability to represent every major CL method as a particular instantiation. Table 2 provides the complete mapping.

Method	Levels	Dense/Sparse	Update Rule	Memory
EWC	1	Dense	$\theta -\lambda F(\theta -{\theta }_0)$	Fisher diagonal
SI	1	Dense	$\theta -\lambda \mathrm{\Omega }(\theta -{\theta }_0)$	Path integral
PackNet	1	Sparse	Prune + freeze	Binary mask
GEM/A-GEM	1	Dense	Projected gradient	Episodic buffer
Experience Replay	1	Dense	SGD on mixed batch	Reservoir buffer
LwF	1	Dense	KD + task loss	Soft labels
PNN	$T$	Sparse per level	Freeze old columns	None (grow)
LoRA-CL	2	Dense L1, Sparse L2	SGD on adapter	Frozen backbone
Doc-to-LoRA	2	Dense L1, Sparse L2	Hypernetwork	Document memory
Titans	3	Dense L1–2, Sparse L3	Surprise-gated	Neural memory
Nested Learning	3	${\rho }_l$ decreasing	Multi-rate SGD	Level buffers
Engram + TRM	2	Dense L1, Sparse L2	Attention + write	External table

The Ideal Continual Learner

Having established the HSD framework, we can now ask: what properties should an ideal continual learner possess?

The intuition behind Conjecture 1 is that the five properties create a system of constraints that over-determines the design space. Consider the tension between (P1) and (P3): bounded forgetting with sublinear parameter growth means the model must compress information about old tasks as new ones arrive. But compression is lossy, and an adversary can always construct a task sequence where the compressed representation of task $t$ is destroyed by task $t+1$ (exploiting the pigeonhole principle on the finite parameter space).

Similarly, (P4) and (P5) are in tension: constant-time inference requires a fixed-size model (no growing memory to search), while online capability precludes the multi-pass consolidation that would allow efficient packing of information into a fixed-size model. The brain appears to resolve this tension through sleep consolidation, a mechanism that violates (P5) by performing offline processing.

The Continual Learning Landscape

We can visualise the trade-offs of Conjecture 1 by plotting CL methods in a two-dimensional space defined by their stability (resistance to forgetting) and plasticity (ability to learn new tasks).

Connections and Synthesis

Continual learning does not exist in isolation. It intersects deeply with memory management (24), alignment and fine-tuning (25), and reasoning (Adaptive Computation Time, Self-Refine). These connections are not superficial analogies; they are mathematical equivalences that allow us to transfer techniques, theorems, and intuitions across domains. This section makes these connections precise.

Continual Learning Meets Memory

The relationship between continual learning and memory management is the deepest of the three connections. At a high level, memory provides the mechanism by which a system resists forgetting, and continual learning provides the dynamics by which memory is tested.

CL Concept	Memory Concept	Mathematical Link
Replay buffer	Hippocampal replay	$ℬ\leftrightarrow$ episodic memory
Weight consolidation	Neocortical consolidation	$\mathrm{ℱ}\leftrightarrow$ synaptic strength
Task-specific adapters	Context-dependent retrieval	LoRA $\leftrightarrow$ memory addressing
Catastrophic forgetting	Retroactive interference	$F_t\leftrightarrow$ overwriting
Progressive networks	Neurogenesis	New columns $\leftrightarrow$ new neurons
Experience replay	Memory reconsolidation	Rehearsal $\leftrightarrow$ reactivation
Knowledge distillation	Teaching consolidation	Soft labels $\leftrightarrow$ schema formation
Sparse memory updates	Sparse coding in cortex	$k$-hot $\leftrightarrow$ population coding
Surprise-gated learning	Novelty detection	$S_t\leftrightarrow$ dopamine signal

The correspondences in Table 3 are not merely poetic analogies; they reflect genuine mathematical equivalences. For instance, the correspondence between replay buffers and hippocampal replay is grounded in the fact that both implement the same operation: sampling from a stored distribution of past experiences and interleaving these samples with new data during learning. The Widrow–Hoff update rule used by Titans' neural memory (Definition 31) is the same learning rule that computational neuroscience models attribute to hippocampal place cells. Even the surprise-gated learning of Titans has a biological counterpart in the dopaminergic novelty signal that gates learning in the hippocampus.

Continual Learning Meets Alignment

The standard alignment pipeline (fig:alignment:pipeline) consists of three stages: pretraining, supervised fine-tuning (SFT), and reinforcement learning from human feedback (RLHF). From the continual learning perspective, this is a three-task CL problem.

The $\beta$–$\lambda$ correspondence of Corollary 2 has immediate practical implications. In the alignment literature, there is extensive work on tuning $\beta$: too small leads to reward hacking, too large leads to insufficient alignment. In the CL literature, there is parallel work on tuning $\lambda$: too small leads to catastrophic forgetting, too large leads to intransigence. The correspondence means that any technique for tuning one parameter can be directly translated to the other. For example, the “warm-up” strategy common in RLHF (starting with large $\beta$ and gradually decreasing) corresponds to starting with strong regularisation and gradually relaxing it, precisely the schedule proposed by [28] in the CL context.

Conversely, the CL insight that $\lambda$ should be adaptive (different for different parameters, based on their Fisher information) suggests that $\beta$ in DPO should also be parameter-dependent: parameters that are more important for maintaining pretrained capabilities should be regularised more strongly, while parameters that can safely change should be regularised less. This idea, parameter-adaptive DPO, is a direct prediction of the CL–alignment connection that has not yet been explored in the alignment literature.

Continual Learning Meets Reasoning

The connection between continual learning and reasoning is more subtle but equally important.

Open Problems

Despite the rapid progress surveyed in this chapter, fundamental questions remain unanswered. We highlight five that we consider most pressing.

1. Optimal CL architecture at 100B+ scale. Current CL methods have been validated at the 7B–13B parameter scale. How do they behave at 100B+? Does the increased capacity of larger models provide natural resistance to forgetting (as suggested by the capacity bound of Theorem 8), or do new failure modes emerge?

2. Forgetting in reasoning chains. When a continually-trained model loses knowledge, how does this affect multi-step reasoning? A forgotten fact at step 3 of a 10-step chain may cause cascading errors. Formal analysis of reasoning-chain fragility under forgetting is needed.

3. Theoretical limits of LoRA-based CL. LoRA-CL (Definition 23) has emerged as the dominant practical approach, yet its theoretical properties are poorly understood. What is the maximum number of tasks representable by rank-$r$ adapters before interference dominates? How does composition quality degrade with the number of composed adapters?

4. Continual RLHF. Human preferences evolve over time. A deployed model must continually align to shifting preferences without forgetting previously learned safety constraints. The interaction between reward model drift and policy forgetting is largely unexplored.

5. Multi-modal CL at scale. As shown in Multimodal and Federated Continual Learning, cross-modal forgetting amplifies the challenge. Scaling multimodal CL to systems with $>2$ modalities (text, image, audio, video, code) introduces combinatorial cross-modal interference that current frameworks cannot address.

Full Taxonomy

We conclude the synthesis with a comprehensive taxonomy of all CL methods discussed in this chapter, visualised as a mind map that highlights both the categorical structure and the cross-connections.

Historical Timeline

The story of continual learning is, in many ways, the story of how science repeatedly discovers, forgets, and rediscovers the problem of forgetting. The phenomenon was first documented rigorously in the 1880s, forgotten by the machine learning community for a century, rediscovered with alarm in the 1980s, and has since oscillated between periods of intense activity and relative neglect. We are currently in the most sustained period of progress, driven by the practical necessity of updating large language models without retraining from scratch.

The significance of Grossberg's contribution cannot be overstated. At a time when most connectionist researchers viewed learning as a monolithic process (present data, update weights, repeat) Grossberg recognised that the dynamics of learning matter as much as the objective. A system that learns too eagerly will overwrite its past; a system that learns too cautiously will fail to adapt. This insight, expressed mathematically through the stability–plasticity dilemma, remains the central challenge of continual learning forty years later.

French's review is worth quoting directly: “Connectionist models, as currently implemented, are fundamentally unable to incrementally learn new information without catastrophically forgetting previously learned information.” This conclusion, while overly pessimistic in retrospect, accurately captured the state of the art at the turn of the millennium. The field entered a quiet period: catastrophic forgetting was treated as a known, unsolvable limitation of neural networks, and researchers either accepted it or worked on entirely different architectures (support vector machines, tree ensembles) that did not suffer from it.

This consolidation period established two crucial insights. First, there is no single best CL method: the optimal approach depends on the scenario (task-incremental vs. class-incremental vs. domain-incremental), the resource constraints (memory budget, compute budget), and the nature of the tasks (similar vs. dissimilar). Second, parameter efficiency is a form of anti-forgetting: methods that update fewer parameters (either explicitly through masking or implicitly through low-rank constraints) consistently achieve better stability, even without any explicit regularisation. This insight would prove prophetic as LoRA emerged as the dominant practical tool.

Exercises

The exercises below span the full range of topics covered in this chapter, from foundational calculations to open-ended design problems. Exercises 1–6 test understanding of the core CL mechanisms (forgetting curves, Bayesian CL, EWC, Fisher information, GEM, reservoir sampling). Exercises 7–10 address LLM-specific methods (LAMOL, PNN, LoRA, Doc-to-LoRA). Exercises 11–18 explore the advanced topics of continual pre-training, learning rate scheduling, sparse memory, Titans, Miras, S6MOD, and nested learning. Exercises 19–21 cover diffusion models, C-CLIP, and federated CL. Exercises 22–26 probe the unified mathematical framework: knowledge capacity, HSD bounds, Pareto frontiers, alignment connections, and scaling laws. The final four exercises (27–30) are more open-ended, connecting CL to biology, online learning theory, compositionality, and system design.