29 Membership Inference Attacks and Model Privacy

The Privacy Challenge in Generative Models

In January 2021, a team led by Nicholas Carlini published a paper that sent shockwaves through the machine learning community. The researchers demonstrated that they could extract verbatim training data from GPT-2, a large language model trained on internet text [12]. The extracted data included personal phone numbers, email addresses, IRC chat handles, physical addresses, and even a complete 128-bit UUID that appeared exactly once in the training corpus. The attack required nothing more than cleverly crafted prompts and a willingness to generate thousands of text completions. The model, it turned out, had not merely learned the statistical patterns of language; it had memorised specific data points from the training set and could be coerced into regurgitating them.

The implications were immediate and unsettling. If a model could reproduce training data verbatim, then anyone with API access could potentially recover sensitive information that was never intended for public release. Two years later, in late 2023, a deceptively simple prompt, “Repeat the word `poem' forever,” applied to ChatGPT caused the model to abandon its safety guardrails and emit pages of memorised training text, including fragments of personal data, news articles, and software documentation. The attack, reported by carlini2023quantifying, demonstrated that even models subjected to extensive reinforcement learning from human feedback could be tricked into revealing their training data.

Meanwhile, in the computer vision domain, the Clearview AI controversy illustrated a different facet of the same problem. Clearview AI scraped billions of facial images from social media platforms without consent and used them to train a facial recognition system sold to law enforcement agencies worldwide. The fundamental question was not whether the model was accurate (it was), but whether individuals whose faces were used as training data had any recourse to discover this fact, and what protections they deserved.

These incidents share a common thread: the training data of a machine learning model is both its greatest asset and its deepest vulnerability. The model's utility derives entirely from the patterns it extracts from training data, yet those same patterns carry fingerprints of the individual data points that produced them. This tension between utility and privacy is not a bug in any particular system; it is a fundamental consequence of statistical learning.

This chapter develops the mathematical and algorithmic tools needed to reason about, detect, and defend against a specific class of privacy attacks called membership inference attacks, in which an adversary seeks to determine whether a particular data point was used to train a given model. We will see that the problem is rich, touching on hypothesis testing, information theory, differential privacy, and the deep geometry of overparameterised models. The detective story we are about to unfold has a clear antagonist (the attacker), a clear victim (the data subject), and a clear weapon (the statistical fingerprints that training leaves on model behaviour). Our task is to understand the weapon, measure its power, and design shields against it.

Why Privacy Matters: Real-World Incidents and Stakes

Imagine a generative model as a vault. The training data is the treasure locked inside; the model architecture and its learned parameters are the walls and the lock; and the attacker possesses only a keyhole through which to peer, namely, the ability to query the model and observe its outputs. The central question of membership inference is deceptively simple: by peering through this keyhole, can the attacker determine whether a specific item of treasure resides inside?

The stakes of this question vary dramatically across application domains, and understanding these stakes motivates the technical development that follows.

Healthcare and medical imaging.

Consider a diffusion model trained to generate synthetic chest X-rays for medical education. If an attacker can determine that a specific patient's X-ray was in the training set, they learn that the patient visited a particular hospital and underwent a specific procedure. For patients with stigmatised conditions (HIV, certain psychiatric diagnoses, substance abuse treatment), this information leakage can have devastating consequences, from employment discrimination to insurance denial. The Health Insurance Portability and Accountability Act (HIPAA) in the United States and the General Data Protection Regulation (GDPR) in Europe both impose strict requirements on the handling of medical data, and a successful membership inference attack could constitute a reportable data breach.

In 2020, researchers demonstrated membership inference attacks against medical image classifiers with AUC scores exceeding $0.85$, meaning an attacker could correctly identify training set members with high confidence [27]. The attack exploited the fact that medical imaging datasets are typically small (thousands to tens of thousands of images), leading to significant overfitting, which as we shall see in Connection to Overfitting and Memorisation, is the primary driver of membership inference vulnerability.

Copyright and intellectual property.

Large language models trained on books, articles, and code raise profound copyright questions. Authors and publishers have filed lawsuits alleging that their copyrighted works were used as training data without permission. In 2023, a group of prominent authors including Sarah Silverman filed a class-action lawsuit against OpenAI and Meta, alleging that their books were ingested into LLM training data without authorisation. In the visual arts domain, a separate class action alleged that Stability AI and Midjourney trained their image generation models on billions of copyrighted images scraped from the internet.

Membership inference provides the forensic tool to answer the factual question: was this specific book, article, or code snippet part of the training corpus? The answer has direct legal and financial implications. A successful membership inference attack establishes a factual predicate for copyright claims; an unsuccessful attack (on a model known to be trained on the data) reveals the limits of forensic detection. Both outcomes are informative for courts and policymakers.

Identity and biometric data.

Face generation models (StyleGAN and its successors) trained on datasets of real faces raise the question of whether a specific individual's likeness was used. This concern extends to voice cloning models, where determining whether a speaker's voice was in the training set has implications for consent and deepfake attribution. The European Union's AI Act classifies biometric data as “high-risk,” and regulations such as Illinois' Biometric Information Privacy Act (BIPA) impose strict consent requirements for the collection and use of facial geometry data. A membership inference tool that can determine whether a specific face appeared in a generative model's training set would have direct regulatory applications.

Financial and legal data.

Models trained on financial documents, legal contracts, or corporate communications may inadvertently memorise sensitive business information. A membership inference attack against a financial LLM could reveal that specific confidential documents were in the training corpus, with implications for trade secret protection and regulatory compliance. Similarly, models trained on court records may memorise details of sealed proceedings that should not be publicly accessible.

The Generative Model Privacy Landscape

Before we dive into the technical machinery of membership inference, it is important to understand how the attack surface differs across generative model families. A membership inference attack against a diffusion model exploits fundamentally different signals than one against an autoregressive language model, even though both attacks answer the same binary question. The reason is architectural: each model family processes and represents data in a unique way, and the “fingerprints” that training leaves on the model's behaviour take correspondingly different forms. Understanding these differences is essential for designing effective attacks and, equally importantly, for developing targeted defences.

Discriminative vs. generative models.

The earliest membership inference attacks [28] targeted discriminative classifiers, where the attack signal was the model's confidence on the true class label. Generative models present a richer and more challenging attack surface. A classifier maps an input $𝒙$ to a discrete label $y$; the information available to the attacker is a probability vector $p(y|𝒙)$ of modest dimension (equal to the number of classes). A generative model, by contrast, defines a distribution over the entire data space, potentially yielding a real-valued likelihood $p_{𝜽}(𝒙)$ or, in the case of diffusion models, a loss trajectory ${\ell }_t(𝒙)$ across all noise levels $t$. This high-dimensional output provides far more bits of information that an attacker can exploit.

Diffusion models: pixel-level memorisation.

Diffusion models learn to reverse a noise-corruption process. At each timestep $t$, the model predicts the noise ${𝝐}_{𝜽}({𝒙}_t,t)$ that was added to produce the noisy version ${𝒙}_t$ of the clean data ${𝒙}_0$. The reconstruction error ${\Vert 𝝐-{𝝐}_{𝜽}({𝒙}_t,t)\Vert }^2$ at each timestep provides a granular signal: memorised images tend to have systematically lower reconstruction error across all timesteps, with particularly distinctive patterns at intermediate noise levels. We develop this intuition formally in later sections of this chapter.

Large language models: token-level memorisation.

Autoregressive language models assign probabilities to tokens in sequence: $p(x_t|x_1,\dots ,x_{t-1})$. The per-token log-likelihood provides a natural membership signal: sequences that appeared in the training data tend to have lower perplexity (higher average log-likelihood per token) than sequences drawn from the same distribution but not seen during training. However, the signal is confounded by the fact that common, high-probability sequences (e.g., “the cat sat on the mat”) naturally receive low perplexity regardless of training set membership. Disentangling memorisation from generalisation is the central challenge in LLM membership inference.

Vision-language models: cross-modal signals.

Models such as CLIP and Stable Diffusion that operate across modalities introduce cross-modal membership signals. An image-text pair $(𝒙,c)$ that appeared in training may produce an unusually high similarity score $\text{sim}(f_{\text{image}}(𝒙),f_{\text{text}}(c))$, providing a signal that is unavailable when attacking unimodal models.

Chapter Roadmap

This chapter is organised into two major halves, reflecting the two dominant generative model paradigms.

Part I: Membership inference in diffusion models (Sections The Privacy Challenge in Generative Models–14).

We begin with the theoretical foundations of membership inference (this section and the next three), establishing the hypothesis testing framework, information-theoretic bounds, and the shadow model paradigm. We then specialise to diffusion models, developing loss-based attacks that exploit the denoising loss at specific timesteps, likelihood-based attacks that compute or approximate the data log-likelihood under the diffusion process, gradient-based attacks that analyse the sensitivity of model parameters to specific inputs, and defences specific to the denoising diffusion framework including regularisation, data augmentation, and differentially private training.

Part II: Membership inference in large language models (Sections 15–30).

The second half adapts and extends the theoretical framework to autoregressive language models, addressing the unique challenges of token-level memorisation, the confounding effect of text frequency (common phrases receive low perplexity regardless of training membership), reference-based attacks that compare target model scores against reference model baselines, and the interaction between membership inference and alignment techniques such as RLHF. We also cover the Min-K% method, neighbourhood attacks, and the latest calibration-based approaches that have pushed the state of the art in LLM membership inference.

fig:mia:taxonomy provides a bird's-eye view of the privacy attack landscape in which membership inference sits.

Membership Inference as Hypothesis Testing

At its core, membership inference is a detection problem. The attacker observes some signal derived from a trained model and must decide between two competing explanations: the data point was part of the training set, or it was not. This framing maps naturally onto the classical theory of statistical hypothesis testing, which provides both a rigorous vocabulary for describing attacks and fundamental limits on what any attack can achieve.

In this section we formalise the membership inference problem, connect it to the Neyman–Pearson framework for optimal hypothesis tests, define the threat models that govern what information an attacker possesses, and introduce the metrics used to evaluate attack performance. By the end of this section, the reader will have a precise mathematical vocabulary for discussing membership inference, a clear understanding of what it means for an attack to be “optimal,” and a taxonomy of attacker capabilities that organises the vast literature into coherent categories.

Formal Problem Statement

We begin with a precise statement of the membership inference problem. The notation we establish here will be used throughout the remainder of this chapter and should be committed to memory before proceeding.

Let $p_{\text{data}}$ denote the true data-generating distribution over a data space $𝒳$. A training algorithm $𝒜$ takes a dataset $D_{\text{train}}=\{{𝒙}_1,\dots ,{𝒙}_n\}$ of $n$ samples drawn i.i.d. from $p_{\text{data}}$ and produces model parameters $𝜽=𝒜(D_{\text{train}})$. The trained model defines a function $f_{𝜽}:𝒳\to 𝒴$, where the output space $𝒴$ depends on the model type (e.g., loss values, probability distributions, or generated samples).

We are now ready for the central definition of this chapter.

This definition frames membership inference as a binary classification problem, but one with a crucial twist: the “labels” (membership status) are properties of the training process, not of the data distribution itself. Two data points $𝒙$ and ${𝒙}^{\prime }$ drawn from exactly the same distribution may have opposite membership labels simply because one was selected for the training set and the other was not. This makes membership inference fundamentally different from standard classification: the signal to be detected is not an inherent property of the data point but a subtle statistical artifact of how the model's parameters were shaped by the training procedure.

The challenge is further complicated by the fact that the attacker must distinguish the effect of a single data point on a model that may have been trained on millions or billions of examples. Each training point contributes a tiny perturbation to the parameter vector, and the attacker must detect this perturbation from the model's observable behaviour. It is as if someone added a single drop of dye to an ocean and asked you to determine, from a sample of seawater, whether the dye was added.

The attack can be equivalently cast as a binary hypothesis test.

The beauty of this formulation is that it immediately connects membership inference to a century of statistical theory. Questions about the “best” attack become questions about the most powerful test, and fundamental limits on attack success become consequences of information-theoretic bounds on distinguishability.

Several remarks on the hypothesis testing formulation are in order. First, the null and alternative hypotheses are not symmetric: in most practical settings, declaring a non-member to be a member (a false positive) has different consequences than missing a true member (a false negative). A privacy auditor may prefer very high specificity (few false positives), accepting lower sensitivity, because falsely accusing a model of training on specific data has legal and reputational implications. Second, the test statistic $T$ can be any function of the model's observable behaviour: the loss value, output probabilities, gradient norms, or even the generated samples themselves. The choice of test statistic is where the art of membership inference lies, and different choices lead to different attack families that we will explore throughout this chapter.

The Neyman–Pearson Framework

Having framed membership inference as a hypothesis test, we can ask: what is the most powerful test possible? The answer comes from one of the crown jewels of classical statistics: the Neyman–Pearson lemma, which characterises the test that maximises detection power at any given false alarm rate. This result is not merely of theoretical interest; it tells us exactly what the ideal attack looks like and provides a benchmark against which all practical attacks can be measured.

To set up the framework, let us define the distributions that an ideal attacker would reason about. For a given data point $𝒙$ and a model $f_{𝜽}$, let $S(𝒙)=T(𝒙,f_{𝜽})$ denote some observable signal (e.g., the model's loss on $𝒙$). Define: (Member Nonmember Dists)$$\begin{array}{cccc}& p_1(s)& =p(S(𝒙)=s|𝒙\in D_{\text{train}}),\text{(member distribution)},& \text{<span class="tml-eqn"></span>}\\ & p_0(s)& =p(S(𝒙)=s|𝒙\notin D_{\text{train}}),\text{(non-member distribution)}.& \text{<span class="tml-eqn"></span>}\end{array}$$ The member distribution $p_1$ describes the distribution of the signal $S$ when the data point is a training set member; the non-member distribution $p_0$ describes the signal when the data point was not used for training. The membership inference problem reduces to distinguishing samples from $p_1$ versus $p_0$.

Threat Models

Before designing an attack, we must specify what the adversary knows and can do. In security research, this specification is called the threat model, and it is arguably the most important design choice in any security analysis. A threat model that is too strong (giving the attacker unrealistic capabilities) leads to attacks that are theoretically interesting but practically irrelevant; a threat model that is too weak (restricting the attacker beyond what is realistic) leads to a false sense of security.

The threat model determines the attack surface and, consequently, which attacks are feasible. We define four canonical threat models arranged in order of decreasing attacker capability.

The four threat models span the full range of practical scenarios. White-box access is available when models are released as open-source (e.g., Stable Diffusion, LLaMA), which is increasingly common in the generative AI ecosystem. Black-box access is the norm for commercial APIs (OpenAI, Anthropic, Google), where users submit queries and receive outputs but cannot inspect model internals. Gray-box access arises in research collaborations or when APIs inadvertently expose additional information (e.g., log-probabilities alongside generated text). Label-only access represents the most restrictive production settings where even confidence scores are suppressed.

tab:mia:threat-models summarises the four threat models and their practical relevance.

The threat models form a natural hierarchy: any attack that works in a weaker threat model also works in all stronger ones, but the converse does not hold. fig:mia:threat-hierarchy illustrates this containment structure.

The Advantage Metric and AUC-ROC

To compare attacks, we need a metric that captures how well an attack distinguishes members from non-members. The most natural metric arises directly from the hypothesis testing framework.

For a given attack $𝒜$ with threshold $\tau$, define the true positive rate (TPR) and false positive rate (FPR): (TPR FPR)$$\begin{array}{cccc}& \text{TPR}(\tau )& =\mathrm{Pr}[𝒜(𝒙)=1|𝒙\in D_{\text{train}}],& \text{<span class="tml-eqn"></span>}\\ & \text{FPR}(\tau )& =\mathrm{Pr}[𝒜(𝒙)=1|𝒙\notin D_{\text{train}}].& \text{<span class="tml-eqn"></span>}\end{array}$$ The TPR measures the fraction of true members correctly identified; the FPR measures the fraction of non-members incorrectly flagged as members.

The advantage metric is attractive because it has a clean interpretation: it measures the gap between the attack's detection rate for true members and its false alarm rate for non-members. A random guessing attack (which ignores the model entirely and predicts membership with some fixed probability $q$) achieves $\text{TPR}=q=\text{FPR}$, giving $\mathrm{Adv}=0$. Any non-zero advantage therefore represents a genuine ability to distinguish members from non-members.

The advantage connects directly to the area under the ROC curve, a connection we now make precise.

The connection between advantage and AUC is important because AUC is the most widely reported metric in the membership inference literature, while advantage has a more direct operational interpretation. The bound tells us that a high AUC is necessary (but not sufficient) for a high advantage: an attack with AUC $=0.65$ can achieve at most ${\mathrm{Adv}}^{\ast }\le 0.30$, while an attack with AUC $=0.95$ can achieve up to ${\mathrm{Adv}}^{\ast }\le 0.90$. In practice, the achieved advantage is often close to the bound for well-calibrated attacks.

fig:mia:roc-anatomy illustrates the anatomy of an ROC curve in the context of membership inference.

Information-Theoretic Foundations

The hypothesis testing framework of Membership Inference as Hypothesis Testing tells us how to evaluate and compare attacks, but it does not explain why membership inference works at all. Why should a model's output carry any information about whether a specific data point was in the training set? The answer lies in information theory: the training process creates a statistical dependence between the model parameters and the training data, and this dependence propagates through the model's outputs to the attacker's observations.

In this section we make this intuition precise. We bound the membership advantage in terms of mutual information, establish that post-processing can only reduce leakage (the data processing inequality), connect membership leakage to the familiar notion of overfitting, and derive fundamental limits on attack success using Fano's inequality. These results provide the theoretical scaffolding on which all practical attacks rest: they tell us which attacks are possible, which are impossible, and where the fundamental barriers lie.

The reader familiar with information theory will recognise many of the tools we deploy here (mutual information, Pinsker's inequality, the data processing inequality, Fano's inequality). The novelty lies not in the tools themselves but in their application to the membership inference setting, where they yield crisp, interpretable bounds with direct implications for model design and privacy engineering.

Mutual Information Between Model and Training Data

The central quantity governing membership leakage is the mutual information between the trained model and the training data. Intuitively, if the model “remembers” its training data, this manifests as statistical dependence between $𝜽$ and $D_{\text{train}}$; mutual information quantifies this dependence. Recall that mutual information $𝖨(X;Y)$ measures the reduction in uncertainty about $X$ that comes from observing $Y$, or equivalently, the amount of information that $Y$ carries about $X$. In our setting, $𝖨(𝜽;D_{\text{train}})$ measures how much the model parameters tell us about the training data.

This quantity has an elegant interpretation. If the training algorithm is deterministic and the training data is fixed, then $𝖧(𝜽|D_{\text{train}})=0$ and the mutual information equals $𝖧(𝜽)$: the model parameters are entirely determined by the data. At the other extreme, if the model completely ignores the training data (e.g., it outputs random noise regardless of input), then $𝖨(𝜽;D_{\text{train}})=0$ and no attack can succeed.

Between these extremes lies the interesting regime: the model has learned useful patterns from the data (non-zero mutual information) but may or may not have memorised individual data points. The key question for membership inference is not the total mutual information $𝖨(𝜽;D_{\text{train}})$ (which is typically large, since the model must depend on the data to be useful), but the mutual information with the membership status of a single data point, which we now formalise.

For membership inference, we are often interested not in the mutual information with the entire dataset, but with the membership status of a single data point. Let $M_i=\mathrm{𝟙}[{𝒙}_i\in D_{\text{train}}]$ denote the binary membership indicator for data point ${𝒙}_i$.

Data Processing Inequality and Membership Bounds

A black-box attacker does not observe the model parameters $𝜽$ directly; instead, the attacker observes the model's output $f_{𝜽}(𝒙)$ on specific queries. This observation is a post-processing of the parameters, and the data processing inequality tells us that post-processing cannot create information. This is one of the most fundamental results in information theory: you cannot extract more information about a source by processing its output; you can only lose information or, at best, preserve it.

Applied to the membership inference setting, the DPI establishes a formal hierarchy of attacker capability: more restricted access means strictly less information available for the attack.

Connection to Overfitting and Memorisation

The information-theoretic bounds of the previous subsections are elegant but abstract. In this subsection we forge a direct link between membership inference and a concept every machine learning practitioner encounters: overfitting. The key insight is that overfitting creates a measurable gap between a model's behaviour on training data and its behaviour on unseen data, and this gap is precisely the signal that membership inference attacks exploit.

The generalisation gap is a quantity that every practitioner monitors (via training loss versus validation loss curves), but its implications for privacy are rarely appreciated. A model with a large generalisation gap is not merely “overfit” in the benign sense of having suboptimal test accuracy; it is privacy-vulnerable, because the gap creates a detectable signature that reveals membership.

The connection between the generalisation gap and membership inference was formalised by yeom2018privacy in a remarkably clean result.

fig:mia:info-flow traces the information flow from training data to attacker inference, showing how mutual information decreases at each stage.

Fano's Inequality and Fundamental Limits

The bounds derived so far tell us how much information is available for membership inference. They are upper bounds on what an attacker can achieve. Fano's inequality approaches the problem from the opposite direction, establishing a lower bound on the error probability of any membership classifier, regardless of its computational complexity or the cleverness of its design. In other words, Fano's inequality tells us how badly every attack must fail when there is insufficient information to distinguish members from non-members.

Let us consolidate the information-theoretic results of this section with a summary of how the bounds relate to each other and to practical quantities.

Differential Privacy and the Shadow Model Paradigm

The previous section established that membership inference succeeds when the training process leaks information about individual data points. This section addresses both sides of the coin: differential privacy as the principled defence that bounds information leakage, and the shadow model paradigm as the practical attack methodology that exploits it.

Together, these two concepts form the theoretical and algorithmic backbone of the membership inference literature. Differential privacy provides the guarantee that says “this model is safe”; shadow models provide the attack that says “let me check.”

The interplay between the two is adversarial and productive. DP provides provable upper bounds on what any attack can achieve; shadow model attacks provide empirical lower bounds on what specific attacks do achieve. When the gap between these bounds is small, we have a satisfying understanding of a model's privacy properties. When the gap is large, it indicates either that our attacks are weak (room for stronger attacks) or that our DP bounds are loose (the model may be safer than the guarantee suggests). Narrowing this gap is an active research frontier.

$(\epsilon ,\delta )$-Differential Privacy

Differential privacy (DP), introduced by dwork2006calibrating and subsequently refined by Dwork, McSherry, Nissim, and Smith, provides a mathematical guarantee that the output of a computation does not depend “too much” on any single input record. It has become the gold standard for privacy guarantees in both theoretical computer science and applied machine learning, adopted by major technology companies (Apple, Google, Microsoft) and government agencies (the US Census Bureau). Applied to machine learning, it ensures that the trained model would have been “almost the same” whether or not any particular data point was included in the training set, which is precisely the condition that defeats membership inference. The beauty of DP is that it provides a worst-case guarantee: the bound holds for every possible data point, every possible dataset, and every possible attack, no matter how computationally powerful.

The definition has an intuitive reading: for any possible output of the mechanism, the probability of that output changes by at most a factor of $e^{\epsilon }$ (plus a tiny $\delta$) when we add or remove a single data point. An attacker observing the output cannot confidently determine whether any particular data point was present.

Two technical notes on the definition are worth highlighting. First, the notion of “adjacent datasets” admits two common variants: add/remove adjacency (where $D^{\prime }$ is obtained from $D$ by adding or removing one record) and replace adjacency (where $D^{\prime }$ is obtained by replacing one record with another). The difference is a factor of two in $\epsilon$, so the distinction matters for precise statements but not for the qualitative picture. We use add/remove adjacency throughout this chapter. Second, the parameter $\delta$ should be thought of as a “failure probability”: with probability $1-\delta$, the mechanism provides $\epsilon$-DP, and with probability $\delta$, the guarantee may be violated. For meaningful privacy, $\delta$ should be cryptographically small, typically $\delta \ll 1/n$ where $n$ is the dataset size.

It is worth emphasising that DP is a property of the mechanism (the training algorithm, viewed as a randomised map from datasets to models), not of the output (the trained model). A single trained model cannot be “differentially private” on its own; the privacy guarantee applies to the process that produced it. This distinction matters because it means that DP must be built into the training procedure from the start; it cannot be retroactively applied to a model that was trained without privacy considerations.

The connection between differential privacy and membership inference advantage is direct and powerful.

The Gaussian Mechanism and DP-SGD

The definition of differential privacy tells us what to guarantee; we now describe how to achieve it in the context of deep learning. The primary tool is DP-SGD (Differentially Private Stochastic Gradient Descent), introduced by abadi2016deep, which modifies the standard SGD training loop to provide a provable $(\epsilon ,\delta )$-DP guarantee.

The building block is the Gaussian mechanism.

DP-SGD applies the Gaussian mechanism to the gradient computation at each training step. The key insight is that by clipping individual per-sample gradients to a fixed norm $C$ and adding Gaussian noise proportional to $C$, we can bound the sensitivity of each gradient step and accumulate a privacy budget across training.

The clipping step is essential: without it, a single unusual data point could produce a gradient of arbitrary magnitude, making the sensitivity infinite and rendering the Gaussian mechanism useless. By clipping each per-sample gradient to have ${\ell }_2$ norm at most $C$, we ensure that removing any single data point changes the aggregate gradient by at most $C/B$ (where $B$ is the batch size), giving a bounded sensitivity to which we can calibrate the noise.

When DP-SGD is applied for multiple steps, the privacy budget accumulates. The composition theorem describes how.

In practice, even tighter accounting is achieved using Rényi DP [2], which tracks privacy loss through the Rényi divergence of order $\alpha$ and provides tighter composition bounds. The moments accountant [1] implements this idea and is the standard tool for computing the privacy budget of DP-SGD training runs. Recent advances in privacy accounting (the “privacy loss distribution” framework and numerical composition methods) have further tightened the bounds, enabling useful models to be trained at lower $\epsilon$ values than the advanced composition theorem alone would suggest.

The practical implications of the advanced composition theorem are significant. For a training run of $T=\mathrm{10,000}$ steps with per-step privacy ${\epsilon }_0=0.01$, naïve composition gives a total budget of $\epsilon =100$ (useless), while advanced composition gives $\epsilon \approx 1.5$ (meaningful). This $\sqrt{T}$ improvement is what makes DP-SGD practical for deep learning, where models are trained for thousands or millions of steps.

tab:mia:dp-tradeoffs summarises the relationship between privacy parameters, noise levels, and model quality.

The Shadow Model Paradigm

While differential privacy provides the theoretical defence against membership inference, the shadow model paradigm, introduced by shokri2017membership in a landmark 2017 paper at the IEEE Symposium on Security and Privacy, provides the practical attack methodology. Their paper, which has been cited over $\mathrm{5,000}$ times, introduced a framework so general that nearly every subsequent membership inference attack can be understood as a variant. The idea is elegant and, in retrospect, almost obvious: if you want to know how a model behaves differently on training data versus non-training data, train your own models and observe the difference directly.

The key assumption of the shadow model paradigm is that the attacker has access to auxiliary data from a distribution similar to the target model's training distribution. This assumption is often realistic: if the target model was trained on natural images, the attacker can use publicly available image datasets; if the target model is a language model trained on web text, the attacker can scrape similar text from the internet. The assumption is weakest when the target model was trained on truly private, domain-specific data (e.g., internal corporate documents or classified government records) for which no public proxy exists. In such cases, the shadow model attack may still work if the attacker can obtain data from a related domain, but the transferability guarantee weakens.

The shadow model paradigm can also be understood as a form of meta-learning: the attacker is learning a task (“does this model treat members differently from non-members?”) by training on multiple related tasks (the shadow models). The attack classifier is essentially a meta-learner that generalises the member/non-member distinction across models.

The shadow model attack works because of a transferability assumption: the relationship between membership and model behaviour is similar across models trained on similar data. If shadow models exhibit lower loss on their training data than on held-out data (as they typically do, due to overfitting), and if this pattern is consistent across shadow models, then the attack classifier learns to detect this pattern and can apply it to the target model.

This transferability is not guaranteed. It depends on the similarity between the auxiliary data distribution and the target training distribution, the similarity in model architectures, and the similarity in training procedures. In practice, shadow model attacks are most effective when the attacker can closely mimic the target model's training setup. When the distributions or architectures differ significantly, the attack's performance degrades, though often by less than one might expect: the fundamental pattern (lower loss on training data) is remarkably universal across architectures and datasets [5].

fig:mia:shadow-pipeline illustrates the complete shadow model attack pipeline.

Evaluation Methodology

Evaluating membership inference attacks requires care. The choice of metric, the construction of the evaluation dataset, and the interpretation of results all involve subtleties that have led to misleading conclusions in published work. A careless evaluation can make a weak attack appear strong (by exploiting distributional differences between members and non-members) or a strong attack appear weak (by using metrics that are insensitive to the regime of practical interest). In this subsection we catalogue the primary evaluation metrics, highlight common pitfalls, and provide guidance for rigorous experimental methodology.

The gold standard for evaluation, advocated by carlini2022membership, is to construct a balanced evaluation set where members and non-members are drawn from the same distribution, differing only in whether they were selected for training, and to report the full ROC curve (or at minimum, TPR at several low FPR values) rather than a single threshold-dependent accuracy number.

tab:mia:eval-metrics summarises the primary metrics used in the membership inference literature.

Having established the foundations of membership inference, from the hypothesis testing framework through information-theoretic bounds to differential privacy and the shadow model paradigm, we are now equipped to tackle the specific challenges posed by modern generative models. The next sections specialise these tools to diffusion models, where the denoising loss trajectory provides a rich signal for membership detection, and to large language models, where per-token likelihoods and reference model comparisons form the basis of state-of-the-art attacks.

The key takeaways from this foundational material are:

1. Membership inference is a hypothesis test, and the optimal test is the likelihood ratio test (Theorem 1).

2. The information available to the attacker decreases through the chain: parameters $\to$ outputs $\to$ post-processed outputs (Theorem 2).

3. Overfitting is the root cause of vulnerability: the generalisation gap directly bounds the achievable advantage (Proposition 3).

4. Differential privacy provides the only known provable defence, at the cost of model utility (Proposition 4).

5. Shadow models provide a practical, transferable attack methodology that approximates the optimal test (Proposition 5).

With these tools in hand, we turn to the specific generative model architectures that dominate the current landscape.

Why Diffusion Models Leak

The forward diffusion process gradually corrupts an image with noise until it becomes indistinguishable from pure static. The reverse process learns to undo this corruption. But in learning to denoise, the model must remember something about each training image. How much does it remember, and can we detect that memory?

This section develops a mathematical framework for understanding why diffusion models leak membership information. We begin with the forward process and trace how information about a training sample persists through the noise schedule, then examine how the reverse process encodes memorisation signals in the learned score function. The analysis reveals that membership leakage is not an accident or a bug; it is a structural consequence of how denoising objectives work. Every diffusion model that learns to denoise well necessarily retains detectable traces of its training data.

Imagine you are a detective, and the diffusion model is a suspect who may have seen your client's photograph during training. You cannot inspect the model's weights directly (they are an inscrutable tangle of floating-point numbers), but you can ask the model to denoise corrupted versions of the photograph and observe how well it performs. The central question is: does the model denoise this particular image suspiciously well, better than it would denoise an image it has never seen?

Forward Process and Information Retention

Recall from ch:diffusion that the DDPM forward process defines a Markov chain that progressively adds Gaussian noise to a clean image ${𝒙}_0$. The marginal distribution at timestep $t$ is available in closed form: (Forward)$$\begin{array}{ccc}& q({𝒙}_t|{𝒙}_0)=\mathrm{Normal}({𝒙}_t;\sqrt{{\bar{\alpha }}_t}{𝒙}_0,(1-{\bar{\alpha }}_t)𝐈),& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${\bar{\alpha }}_t={\prod }_{s=1}^t{\alpha }_s$ and each ${\alpha }_s=1-{\beta }_s$ is determined by the noise schedule $\{{\beta }_1,\dots ,{\beta }_T\}$. At $t=0$ the noisy version is the clean image itself; at $t=T$ it is (approximately) an isotropic Gaussian, independent of ${𝒙}_0$.

The key quantity for privacy analysis is how much information about the original image ${𝒙}_0$ survives in the noisy version ${𝒙}_t$. We formalise this through mutual information.

The formula in (Retention) follows directly from the well-known expression for the mutual information between a signal and its noise-corrupted version in the Gaussian channel. For a real image with non-Gaussian statistics the exact value differs, but the qualitative behaviour, monotonic decay with increasing $t$, remains the same.

The following diagram illustrates the forward and reverse processes with privacy annotations showing how the mutual information between ${𝒙}_0$ and ${𝒙}_t$ evolves across timesteps.

Reverse Process and Memorisation Signals

The reverse process is where memorisation enters the picture. The model learns a score function ${𝒔}_{𝜽}({𝒙}_t,t)\approx {\nabla }_{{𝒙}_t}\log p_t({𝒙}_t)$, or equivalently a noise prediction network ${𝝐}_{𝜽}({𝒙}_t,t)$, that is trained to denoise corrupted samples from the training set. For a member ${𝒙}_0\in {𝒟}_{\text{train}}$, the model has directly optimised the objective (LOSS)$$\begin{array}{ccc}& {\ell }_t({𝒙}_0)={𝔼}_{𝝐\sim \mathrm{Normal}(0,𝐈)}\left[{\Vert 𝝐-{𝝐}_{𝜽}(\sqrt{{\bar{\alpha }}_t}{𝒙}_0+\sqrt{1-{\bar{\alpha }}_t}𝝐,t)\Vert }^2\right].& \text{<span class="tml-eqn"></span>}\end{array}$$ This per-sample denoising loss is the central object of study for membership inference in diffusion models.

To make this intuition precise, we define the excess denoising error and establish that it is negative for members on average.

The excess denoising error provides a clean theoretical signal, but its practical utility depends on how reliably it can be estimated from a finite number of noise samples. The following lemma quantifies the estimation variance.

Score Function Memorisation: Mathematical Characterisation

We now introduce a formal measure of how strongly a model memorises individual training samples, expressed as a ratio of denoising errors.

The memorisation index provides a per-timestep, per-sample view of the model's memory. To understand the aggregate behaviour, we establish an information-theoretic bound on the average memorisation index.

Connection to Overfitting in Score Matching

The analysis so far has focused on the denoising loss as a membership signal. We now connect this to the broader theory of score matching (see ch:diffusion for a comprehensive treatment) and show that the generalisation gap of the score matching objective directly controls the strength of membership inference attacks.

Loss-Based and Likelihood-Based Attacks

Armed with the insight that training members enjoy lower denoising loss, the simplest attack is embarrassingly direct: compute the loss for a query image and threshold it. If the loss falls below the threshold, declare the image a member; otherwise, declare it a non-member. This section develops this idea from its simplest form through increasingly sophisticated variants, culminating in likelihood ratio tests that approach the information-theoretic optimum.

The progression mirrors a detective's refinement of technique. The first approach (raw loss thresholding) is the equivalent of checking whether a suspect has an alibi. The second (ELBO decomposition) examines the alibi in granular detail, looking for inconsistencies in specific time intervals. The third (likelihood ratio tests) compares the suspect's behaviour to a baseline population, asking not just “is this loss low?” but “is this loss lower than expected for a random individual?”

Training Loss as Membership Signal

We begin with the most direct formulation. Given a trained diffusion model with noise prediction network ${𝝐}_{𝜽}$ and a query image $𝒙$, we compute the average denoising loss across all timesteps and use it as a membership score.

In practice, the expectation over $𝝐$ is approximated by Monte Carlo sampling with $K$ noise vectors per timestep, and the sum over all $T$ timesteps may be replaced by a sum over a subset of $T_{\text{eval}}$ uniformly sampled timesteps to reduce computation.

Statistical Separation

How large is the gap between member and non-member losses, and what determines its magnitude? The following theorem establishes that the separation is $\mathrm{\Omega }(1/n)$, which is small for large datasets but non-vanishing.

The following diagram illustrates the statistical separation between member and non-member loss distributions.

ELBO-Based Membership Inference

The loss-based attack in Training Loss as Membership Signal uses a single scalar (the average denoising loss) as its membership signal. But the ELBO of a diffusion model naturally decomposes into $T$ terms, each carrying potentially different amounts of membership information. By using the full vector of per-term losses rather than their average, we can construct a more powerful attack.

Recall from ch:vi that the evidence lower bound for a diffusion model decomposes as (ELBO)$$\begin{array}{cccc}& \log p(𝒙)\ge & \underset{L_0}{\underbrace{{𝔼}_q[\log p_{𝜽}({𝒙}_0|{𝒙}_1)]}}-\sum _{t=2}^T\underset{L_{t-1}}{\underbrace{{𝖣}_{\text{KL}}(q({𝒙}_{t-1}|{𝒙}_t,{𝒙}_0)\Vert p_{𝜽}({𝒙}_{t-1}|{𝒙}_t))}}& \\ & & -\underset{L_T}{\underbrace{{𝖣}_{\text{KL}}(q({𝒙}_T|{𝒙}_0)\Vert p({𝒙}_T))}}.& \text{<span class="tml-eqn"></span>}\end{array}$$ Each term $L_t$ measures how well the model's reverse process matches the true posterior at a specific timestep.

The power of the ELBO-based approach lies in its ability to weight different timesteps according to their informativeness. The following diagram illustrates which ELBO terms contribute most to the membership signal.

Likelihood Ratio Tests for Diffusion Models

The Neyman–Pearson lemma tells us that the most powerful test for distinguishing two hypotheses at any significance level is the likelihood ratio test. Applying this principle to membership inference, the optimal attack compares the likelihood of the observed model outputs under two hypotheses: “$𝒙$ was in the training set” versus “$𝒙$ was not in the training set.”

In practice, the likelihoods in (LRT) are intractable because they require marginalising over all possible training sets that include (or exclude) $𝒙$. The practical approximation, which is the diffusion-model analogue of the LiRA attack [4], uses reference models trained on datasets that do not contain $𝒙$.

The procedure is as follows. Train $R$ reference diffusion models on datasets sampled independently from the same distribution, each of size $n$, none containing the query $𝒙$. Compute the denoising loss $\ell (𝒙;{𝜽}_r)$ for each reference model $r=1,\dots ,R$ to form a reference distribution of losses. Then compare the target model's loss $\ell (𝒙;{𝜽}_{\text{target}})$ to this reference distribution. If the target loss is significantly lower than the reference losses, infer membership.

Comparison and Computational Considerations

Having introduced three families of loss-based and likelihood-based attacks, we now compare them systematically across several dimensions: the access required to the model, the computational cost, the empirical performance, and the practical advantages and limitations.

Several observations emerge from tab:mia:loss:comparison.

• Loss-based attacks are the simplest and cheapest, requiring only the ability to compute the denoising loss for a query image. They serve as a useful baseline and are effective when the model is significantly overfit, but they struggle on well-regularised models with large training sets.

• ELBO-based attacks improve upon the loss baseline by exploiting the per-timestep structure of the diffusion objective. The computational overhead is moderate (computing all $T$ ELBO terms rather than a subset), and the need for a labelled reference set to train the logistic classifier is a practical requirement that may not always be met.

• Likelihood ratio tests are the most powerful but also the most expensive, requiring the training of multiple reference models. For diffusion models, each reference model training run may take days on modern hardware, making this approach prohibitive for large-scale models. The payoff is substantially higher AUC, particularly on challenging settings [4].

Step-wise Error and Noise Schedule Vulnerability

Not all timesteps are created equal. A detective searching for evidence would focus on the most revealing moments, not examine every second of a suspect's day. Similarly, the most powerful membership inference attacks on diffusion models focus on specific timesteps where the model's memory is most exposed.

This section introduces SecMI (Step-wise Error Comparing Membership Inference), an attack that concentrates its effort on the timesteps where the gap between member and non-member denoising quality is largest [3]. We then analyse which timesteps are most vulnerable and why, connecting the vulnerability profile to the noise schedule. The section concludes with a broader discussion of the privacy-quality tradeoff that confronts every diffusion model developer.

SecMI: Step-wise Error Comparing Membership Inference

The key idea behind SecMI is to replace the average denoising loss (used by the loss-based attack in Training Loss as Membership Signal) with a more targeted measure: the step-wise estimation error, which quantifies how well the model can reconstruct the clean image from a single noisy observation.

The step-wise estimation error $e_t({𝒙}_0)$ is closely related to the denoising loss ${\ell }_t({𝒙}_0)$ from (LOSS), but it operates in the image space rather than the noise space. For members, the model has been optimised to predict $𝝐$ accurately when denoising corrupted versions of ${𝒙}_0$, which translates to a more accurate reconstruction ${\widehat{𝒙}}_0^{(t)}$ and hence a smaller estimation error $e_t$. This is the membership signal that SecMI exploits.

The SecMI algorithm evaluates the step-wise estimation error at a carefully chosen subset of timesteps, averages over multiple noise samples for stability, and produces a membership score.

The following diagram illustrates the complete SecMI pipeline.

Which Timesteps Are Most Vulnerable?

A natural question is: which timesteps should be included in ${𝒯}_{\text{eval}}$ to maximise the power of the SecMI attack? The answer, perhaps surprisingly, is neither the very early timesteps (low noise) nor the very late ones (high noise), but the intermediate timesteps where a moderate amount of noise has been added.

The following diagram visualises the vulnerability profile as a function of the normalised timestep.

Noise Schedule Design and Privacy Implications

The vulnerability profile in fig:mia:secmi:vulnerability depends on the noise schedule $\{{\beta }_1,\dots ,{\beta }_T\}$ through its effect on ${\bar{\alpha }}_t$. Different schedule choices lead to different vulnerability windows, and this observation has important implications for privacy-aware model design.

The Privacy–Quality Tradeoff

Throughout this chapter, we have encountered hints that better generation quality comes at a privacy cost. Models with lower training loss denoise members more accurately, making them more vulnerable to membership inference. We now state this tradeoff formally.

fig:mia:pareto illustrates the tension: as models improve in generation quality (lower FID), they tend to become more vulnerable to membership inference (higher AUC). The ideal operating point (low FID, low AUC) lies in the lower-left corner, but the Pareto frontier prevents reaching it with standard training. Achieving points below the frontier requires fundamentally different approaches, such as differential privacy (discussed in later sections) or privacy-preserving data augmentation.

The following table presents benchmark results across multiple datasets and models, quantifying the quality–privacy tradeoff.

We conclude this section with a systematic comparison of timestep vulnerability across different noise schedules.

tab:mia:schedule:vulnerability confirms the prediction of Proposition 13: linear schedules produce the widest vulnerability windows, while cosine schedules are the narrowest. Sigmoid schedules fall in between, offering a compromise. These differences, while meaningful, are modest in absolute terms; the choice of noise schedule alone is not sufficient to provide strong privacy guarantees. For that, one needs formal mechanisms such as differential privacy, which we address in later sections.

White-Box Gradient Attacks on Diffusion Models

So far, our detective has been peering through the keyhole: observing the model's denoising losses, querying its likelihood estimates, probing its behaviour at specific timesteps. These are powerful techniques, but they treat the model as a black box whose internal workings remain hidden. Now we break through the door entirely. In a white-box attack, the adversary has full access to the model's architecture and learned parameters $𝜽$. Every weight, every bias, every layer is laid bare for inspection. The question becomes: what additional membership signals can we extract when the model's internals are no longer secret?

The answer, as we shall see, is gradients. The gradient of the loss with respect to the model parameters, evaluated at a query sample, carries a rich fingerprint of whether that sample participated in training. For members, the gradient landscape is smoother, flatter, and more predictable than for non-members, because the optimisation process has sculpted the loss surface to accommodate training samples. This section develops the mathematical framework for gradient-based membership inference, culminating in the decomposition technique of pang2023whitebox that achieves state-of-the-art white-box attack performance on diffusion models.

The scenario is not hypothetical. Open-source diffusion models such as Stable Diffusion distribute their full weights publicly, and fine-tuned variants are routinely shared on model hubs. Any adversary who downloads these weights can mount a white-box attack against the training data. Understanding these attacks is therefore essential for assessing the privacy risks of open-weight release.

The White-Box Threat Model

We begin by precisely specifying the adversary's capabilities. The white-box threat model is the strongest standard model in the membership inference literature and subsumes all weaker settings.

The adversary may also have access to auxiliary data from the same distribution, which can be used to calibrate thresholds. However, unlike shadow-model attacks (see The Shadow Model Paradigm), the white-box adversary does not need to train shadow models, because the target model's own gradients provide a direct membership signal.

Gradient-Based Membership Signals

The fundamental insight of gradient-based membership inference is that the gradient of the loss with respect to the model parameters encodes information about the relationship between the query sample and the training set. Consider the per-sample denoising loss at timestep $t$: (LOSS)$$\begin{array}{ccc}& {\ell }_t({𝒙}_0;𝜽)={𝔼}_{𝝐\sim \mathrm{Normal}(0,𝐈)}\left[{\Vert 𝝐-{𝝐}_{𝜽}({𝒙}_t,t)\Vert }^2\right],& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${𝒙}_t=\sqrt{{\bar{\alpha }}_t}{𝒙}_0+\sqrt{1-{\bar{\alpha }}_t}𝝐$. The gradient of this loss with respect to $𝜽$ is (Gradient)$$\begin{array}{ccc}& {𝒈}_t({𝒙}_0;𝜽)={\nabla }_{𝜽}{\ell }_t({𝒙}_0;𝜽)=-2{𝔼}_{𝝐}\left[\right(𝝐-{𝝐}_{𝜽}({𝒙}_t,t){)}^{\top }{\nabla }_{𝜽}{𝝐}_{𝜽}({𝒙}_t,t)].& \text{<span class="tml-eqn"></span>}\end{array}$$

For a training sample ${𝒙}_0\in {𝒟}_{\text{train}}$, the optimisation process has (approximately) driven this gradient toward zero. The parameter vector $𝜽$ sits near a local minimum of ${\ell }_t({𝒙}_0;𝜽)$ for each member. For a non-member ${𝒙}_0\notin {𝒟}_{\text{train}}$, no such optimisation has occurred, so the gradient is generically non-zero and points in a direction that would reduce the loss if training were to continue.

The gradient norm statistic is simple and intuitive, but it discards directional information. Richer signals emerge when we consider the full geometry of the gradient vector.

Fisher Information and Membership

The connection between gradients and membership becomes precise through the lens of Fisher information. The Fisher information matrix captures the curvature of the loss landscape and determines how sensitively the model's predictions respond to parameter perturbations.

The Fisher information provides a natural decomposition of the membership signal into “magnitude” (trace) and “direction” (principal eigenvectors) components.

Gradient Decomposition Across Timesteps

The multi-step structure of diffusion models provides a unique opportunity for white-box attacks: the adversary can compute gradients at each timestep independently and combine them. This temporal decomposition, first exploited by pang2023whitebox, yields significantly more powerful attacks than aggregate gradient statistics.

The key observation is that different timesteps carry different membership signals (as we established in Step-wise Error and Noise Schedule Vulnerability), and the gradient at each timestep captures a distinct aspect of the model's memorisation. At early timesteps ($t$ near $0$), the gradient reflects the model's ability to reproduce fine details; at late timesteps ($t$ near $T$), it reflects the model's knowledge of global structure. By separately analysing the gradient at each timestep, the attacker obtains a multidimensional membership signal that is far more informative than any single-timestep statistic.

pang2023whitebox showed that this decomposition dramatically improves attack performance compared to single-timestep gradient statistics. The intuition is that the weight vector $𝒘$ learns to emphasise the most informative timesteps and suppress noisy ones, effectively performing an optimal linear combination of the per-timestep signals.

The Pang et al. White-Box Attack

We now describe the complete white-box attack methodology of pang2023whitebox, which combines the gradient decomposition framework with several practical innovations.

The attack proceeds in four stages:

Stage 1: Timestep selection.

Rather than using all $T$ timesteps (which would be computationally prohibitive for $T=1000$), the attacker selects a subset $𝒯=\{t_1,\dots ,t_K\}$ with $K\ll T$. pang2023whitebox recommend selecting timesteps that maximise the signal-to-noise ratio of the gradient norm statistic. In practice, they use $K=10-50$ timesteps uniformly spaced across the interval, with higher density in the intermediate range $t\in [0.2T,0.8T]$ where the memorisation signal is strongest (recall the analysis of Which Timesteps Are Most Vulnerable?).

Stage 2: Gradient computation.

For each selected timestep $t_k$ and each noise realisation ${𝝐}^{(j)}$ (using $J$ noise samples for Monte Carlo estimation), the attacker computes: (GRAD)$$\begin{array}{ccc}& {\hat{G}}_{t_k}({𝒙}_0)=\frac{1}{J}\sum _{j=1}^J{\Vert {\nabla }_{𝜽}{\Vert {𝝐}^{(j)}-{𝝐}_{𝜽}({𝒙}_{t_k}^{(j)},t_k)\Vert }^2\Vert }_2,& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${𝒙}_{t_k}^{(j)}=\sqrt{{\bar{\alpha }}_{t_k}}{𝒙}_0+\sqrt{1-{\bar{\alpha }}_{t_k}}{𝝐}^{(j)}$.

Stage 3: Layer-wise decomposition.

A further refinement decomposes the gradient norm by network layer. Let the denoiser have $L$ layers with parameter subsets $𝜽=({𝜽}_1,\dots ,{𝜽}_L)$. The layer-wise gradient norm at timestep $t_k$ is (Layer)$$\begin{array}{ccc}& G_{t_k}^{(\ell )}({𝒙}_0)={\Vert {\nabla }_{{𝜽}_{\ell }}{\ell }_{t_k}({𝒙}_0;𝜽)\Vert }_2,\ell =1,\dots ,L.& \text{<span class="tml-eqn"></span>}\end{array}$$ This creates a feature vector of dimension $K\times L$ per query sample, providing a fine-grained view of the membership signal across both time and depth.

Stage 4: Classification.

The feature vector is fed to a lightweight binary classifier (typically logistic regression or a small MLP) trained on a calibration set. The calibration set consists of samples with known membership status, which can be obtained from auxiliary data or by splitting the adversary's own data.

Connection to Influence Functions

The gradient-based membership signal has a deep connection to the classical theory of influence functions from robust statistics. Influence functions quantify the effect of adding or removing a single training point on the model's parameters and predictions. In the context of membership inference, they provide an alternative derivation of why gradients are informative about membership.

The self-influence ${ℐ}_{\text{loss}}({𝒙}_0^{(i)},{𝒙}_0^{(i)})$ measures how much removing sample $i$ from the training set would increase the loss on sample $i$ itself. This quantity is directly related to the membership signal.

The practical difficulty with influence functions is the need to compute or approximate ${𝐇}^{-1}$, which is intractable for modern diffusion models with millions of parameters. Stochastic approximations (such as the Arnoldi iteration or conjugate gradient methods) can estimate the Hessian-vector product ${𝐇}^{-1}𝒈$ without forming $𝐇$ explicitly, but these require many additional forward and backward passes. For this reason, the direct gradient norm approach of pang2023whitebox is preferred in practice, as it avoids the Hessian entirely while capturing most of the membership signal.

Comparison of White-Box Attack Variants

tab:mia:whitebox:compare summarises the main white-box attack variants for diffusion models, comparing their membership signals, computational cost, and empirical performance.

llccc@ Method	Signal	Cost	AUC Range	Ref.
Single-$t$ gradient norm	$G_t({𝒙}_0)$	$J$ backward	$0.55$–$0.70$	[5]
Multi-$t$ gradient norm	$(G_{t_1},\dots ,G_{t_K})$	$KJ$ backward	$0.65$–$0.82$	[6]
Layer-wise decomposition	$G_t^{(\ell )}$ for all $\ell ,t$	$KJ$ backward	$0.70$–$0.88$	[6]
Gradient alignment	$A_t({𝒙}_0)$	$KJ$ backward $+$ cal.	$0.60$–$0.75$	[5]
Influence function	${ℐ}_{\text{loss}}({𝒙}_0,{𝒙}_0)$	Hessian approx.	$0.68$–$0.80$	[6]
Fisher trace	$\mathrm{tr}({𝐅}_t({𝒙}_0))$	$KJ$ backward	$0.62$–$0.78$	[7]

The layer-wise decomposition of pang2023whitebox achieves the highest reported AUC, because it captures the finest-grained membership signal. However, its computational cost scales linearly with both the number of timesteps and noise samples, making it expensive for large models. The single-timestep gradient norm is the cheapest variant but sacrifices considerable discriminative power.

A natural question arises: is the additional power of white-box attacks worth the additional access requirement? The answer depends on the threat model. For open-source models, white-box access is free, and the additional $10$–$15$ percentage points of AUC can make the difference between a marginal and a decisive attack. For proprietary models, the gray-box loss-based attacks of Loss-Based and Likelihood-Based Attacks may be the strongest available option.

Exercises

Frequency-Domain and Noise-Based Attacks

The previous sections have pursued our suspect, the diffusion model, through the corridors of loss values, likelihood ratios, and gradient norms. Each approach interrogates the model in the spatial domain, asking how accurately it reconstructs an image pixel by pixel. But there is an entirely different line of questioning available to the detective. Rather than examining the pixels themselves, we can transform the denoising error into the frequency domain and ask: at which spatial frequencies does the model reveal its memory?

This change of perspective is not merely a mathematical curiosity. It turns out that memorisation in diffusion models leaves its most distinctive fingerprints in the high-frequency components of the denoising error. A model that has memorised a training image reconstructs its fine details, edges, textures, and subtle gradients, with uncanny precision, while it struggles with these same high-frequency components for unseen images. The frequency domain amplifies this difference, providing a membership signal that is often stronger and more robust than spatial-domain alternatives.

Alongside the frequency-domain perspective, this section also explores a complementary idea: using the model's response to carefully crafted noise patterns as a membership probe. The initial noise fed to the reverse process is not merely a random starting point; it is a key that unlocks different aspects of the model's memory. By choosing this key carefully, the attacker can extract membership information with remarkable efficiency.

Spectral Analysis of Denoising Errors

To analyse denoising errors in the frequency domain, we need the discrete cosine transform (DCT), which is the standard tool for decomposing images into spatial frequency components.

The power spectral density of the denoising error provides a frequency-resolved view of the model's performance.

The spectral power provides a natural decomposition of the total denoising error (which is Parseval's theorem applied to the DCT): (Parseval)$$\begin{array}{ccc}& {\Vert {𝒆}_t({𝒙}_0)\Vert }_2^2=\sum _{\omega =0}^{{\omega }_{\max }}|{𝒮}_{\omega }|\cdot P_t^{(\omega )}({𝒙}_0),& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${\omega }_{\max }=\lfloor \sqrt{{(H-1)}^2+{(W-1)}^2}\rfloor$. The total squared error is the same quantity used by loss-based attacks (Section 6); the frequency decomposition simply distributes it across frequency bands.

High-Frequency Components as Membership Signals

Why should the frequency domain help? The key insight, developed by liu2024frequency, is that memorisation manifests most strongly in the high-frequency components of the denoising error.

Consider what happens when the diffusion model encounters a training image. It has learned to denoise this specific image across all timesteps, including the fine high-frequency details: sharp edges, texture patterns, subtle colour gradients. The denoising error in the high-frequency bands is therefore small: the model gets these details right. For a non-member image, the model has only learned the distribution of high-frequency content, not the specific pattern for this particular image. It predicts reasonable high-frequency content on average, but the prediction error is systematically larger.

This proposition provides the theoretical foundation for the frequency-domain attack: by focusing on high-frequency bands, the attacker extracts a cleaner membership signal than by using the aggregate loss (which averages over all frequencies and dilutes the signal with uninformative low-frequency components).

The Liu et al. Frequency-Domain Attack

liu2024frequency operationalise the frequency-domain insight into a practical attack. Their method proceeds as follows:

Step 1: Compute denoising errors.

For a query image ${𝒙}_0$ and a set of timesteps $𝒯=\{t_1,\dots ,t_K\}$, compute the denoising error ${𝒆}_{t_k}({𝒙}_0)$ using $J$ independent noise realisations and average: (Error)$$\begin{array}{ccc}& {\overline{𝒆}}_{t_k}({𝒙}_0)=\frac{1}{J}\sum _{j=1}^J{𝒆}_{t_k}^{(j)}({𝒙}_0).& \text{<span class="tml-eqn"></span>}\end{array}$$

Step 2: Transform to frequency domain.

Apply the 2D DCT to each averaged error: (DCT)$$\begin{array}{ccc}& {\widehat{\bar{𝒆}}}_{t_k}({𝒙}_0)=\mathrm{DCT}\{{\overline{𝒆}}_{t_k}({𝒙}_0)\}.& \text{<span class="tml-eqn"></span>}\end{array}$$

Step 3: Extract high-frequency statistics.

Compute the spectral power in a high-frequency band ${\mathrm{\Omega }}_{\text{HF}}=\{\omega :\omega >{\omega }_0\}$ for a threshold frequency ${\omega }_0$: (HF)$$\begin{array}{ccc}& {\mathrm{HF}}_{t_k}({𝒙}_0)=\sum _{\omega \in {\mathrm{\Omega }}_{\text{HF}}}{P}_{t_k}^{(\omega )}({𝒙}_0).& \text{<span class="tml-eqn"></span>}\end{array}$$

Step 4: Aggregate across timesteps.

Form the final membership statistic by a weighted sum: (STAT)$$\begin{array}{ccc}& S_{\text{freq}}({𝒙}_0)=\sum _{k=1}^K{w}_k\cdot {\mathrm{HF}}_{t_k}({𝒙}_0),& \text{<span class="tml-eqn"></span>}\end{array}$$ where the weights $w_k$ are learned from a calibration set or set to be uniform.

Step 5: Threshold.

Declare ${𝒙}_0$ a member if $S_{\text{freq}}({𝒙}_0)<{\tau }_{\text{freq}}$, since members exhibit smaller high-frequency error.

Noise as a Probe: Initial Noise Techniques

A complementary line of attack, introduced by zhai2024noise, exploits the relationship between the initial noise vector and the model's reconstruction fidelity. The key idea is beautifully simple: rather than adding random noise to the query image, the attacker crafts the noise to maximise the membership signal.

In the standard forward process, the noisy image at timestep $t$ is (Forward)$$\begin{array}{ccc}& {𝒙}_t=\sqrt{{\bar{\alpha }}_t}{𝒙}_0+\sqrt{1-{\bar{\alpha }}_t}𝝐,𝝐\sim \mathrm{Normal}(0,𝐈).& \text{<span class="tml-eqn"></span>}\end{array}$$ For membership inference, the attacker is free to choose any $𝝐$; the standard Gaussian distribution is merely a convention. zhai2024noise propose choosing $𝝐$ to probe specific aspects of the model's memory.

In practice, zhai2024noise approximate the probe noise using the following heuristic. They run a single denoising step from a random $𝝐$ and compute the denoising direction: (Direction)$$\begin{array}{ccc}& 𝒅=𝝐-{𝝐}_{𝜽}({𝒙}_t,t).& \text{<span class="tml-eqn"></span>}\end{array}$$ The probe noise is then constructed by amplifying the denoising direction: (Practical)$$\begin{array}{ccc}& {𝝐}^{\ast }=𝝐+\lambda \frac{𝒅}{{\Vert 𝒅\Vert }_2}\sqrt{d},& \text{<span class="tml-eqn"></span>}\end{array}$$ where $\lambda >0$ is a scaling parameter. This construction pushes the noise in the direction where the model exhibits the largest denoising error, amplifying the difference between members (which the model handles well in all directions) and non-members (which exhibit larger errors in specific directions).

Noise Aggregation Analysis

A related but distinct technique, proposed by wu2024noise, analyses the model's response to small perturbations of the query image rather than crafted initial noise. The method, called noise aggregation analysis, injects small Gaussian noise into the query image and aggregates the model's responses.

The procedure is as follows. Given a query ${𝒙}_0$, the attacker generates $M$ perturbed copies: (Perturb)$$\begin{array}{ccc}& {\widetilde{𝒙}}_0^{(m)}={𝒙}_0+{\sigma }_{\text{small}}{𝜼}^{(m)},{𝜼}^{(m)}\sim \mathrm{Normal}(0,𝐈),m=1,\dots ,M,& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${\sigma }_{\text{small}}\ll 1$ is a small perturbation scale. Each perturbed copy is then passed through the forward process (to timestep $t$) and denoised, yielding a reconstructed image ${\widehat{𝒙}}_0^{(m)}$. The membership statistic is based on the variance of the reconstructions: (VAR)$$\begin{array}{ccc}& V_{\text{agg}}({𝒙}_0)=\frac{1}{M}\sum _{m=1}^M{\Vert {\widehat{𝒙}}_0^{(m)}-{\overline{\hat{𝒙}}}_0\Vert }_2^2,{\overline{\hat{𝒙}}}_0=\frac{1}{M}\sum _{m=1}^M{\widehat{𝒙}}_0^{(m)}.& \text{<span class="tml-eqn"></span>}\end{array}$$

Combining Frequency and Noise Perspectives

The frequency-domain and noise-based approaches are complementary. The frequency analysis identifies which components of the denoising error are most informative, while the noise probing technique controls how the model is interrogated to maximise the signal. A natural combination applies the frequency decomposition to the denoising errors obtained with probe noise: (Combined)$$\begin{array}{ccc}& S_{\text{combined}}({𝒙}_0)=\sum _{\omega \in {\mathrm{\Omega }}_{\text{HF}}}{P}_t^{(\omega )}({𝒙}_0;{𝝐}^{\ast }),& \text{<span class="tml-eqn"></span>}\end{array}$$ where $P_t^{(\omega )}({𝒙}_0;{𝝐}^{\ast })$ is the spectral power computed using the probe noise ${𝝐}^{\ast }$ instead of random Gaussian noise.

Empirically, the combined approach yields attack AUC values that exceed either method alone. liu2024frequency report that the frequency-domain attack on CIFAR-10 achieves AUC $\approx 0.78$, while the combination with probe noise reaches AUC $\approx 0.83$. The improvement is most pronounced for models trained with data augmentation, which suppresses spatial-domain membership signals but leaves frequency-domain signals relatively intact.

Exercises

Black-Box API Attacks on Diffusion Models

Our investigation now reaches its most constrained, and in many ways most compelling, scenario. The detective has been stripped of all special tools. No access to the model's weights (ruling out gradient attacks), no visibility into intermediate representations (ruling out frequency-domain analysis of denoising errors at specific timesteps), not even the ability to query the model with custom-crafted noise vectors. All that remains is the public API: the ability to submit a text prompt (or an image) and receive a generated image in return, perhaps accompanied by a numerical score. Can membership inference still succeed under these extreme constraints?

The answer, remarkably, is yes. Black-box attacks on diffusion models exploit the fact that even the final generated outputs carry statistical fingerprints of the training data. A model that has memorised a training image will produce outputs that are, in subtle but detectable ways, more similar to that image than to images it has never seen. Detecting this similarity from generated samples alone requires ingenuity, but as we shall see, several groups have developed effective techniques.

The black-box setting is the most practically relevant threat model for commercial diffusion model APIs (such as DALL-E, Midjourney, and Stable Diffusion via hosted services), where users interact only through a generation interface. Understanding black-box attacks is therefore critical for assessing the privacy risks of deployed systems.

The Black-Box Threat Model

Generation-Based Membership Signals

The most constrained black-box setting provides only generation access: the attacker can produce images from the model but cannot evaluate any score on a query image. How can one possibly infer membership from generated samples alone?

The insight of tang2024blackbox is that a model which has memorised a training image will, with non-negligible probability, generate images that are unusually close to that training image. This closeness can be measured using an external perceptual similarity metric (such as LPIPS, SSIM, or a feature-space distance from a pre-trained classifier).

The generation proximity statistic relies on the assumption that memorised images are more likely to appear as (approximate) outputs. This is justified by the empirical observation that diffusion models occasionally produce near-copies of training images, particularly for images that appear multiple times in the training set or are otherwise “memorable” (e.g., iconic photographs, distinctive compositions).

Score-Based Black-Box Attacks

When the API provides a scoring function, the attacker gains significant additional power. tang2024blackbox propose using the model's own score (e.g., a log-likelihood or a noise-prediction residual exposed through the API) as a membership signal, similar to the loss-based attacks of Loss-Based and Likelihood-Based Attacks but without access to the model's internals.

The score-based approach queries the API to evaluate $s({𝒙}_0)=-{\Vert 𝝐-\widehat{𝝐}({𝒙}_t,t)\Vert }_2^2$ for various timesteps $t$ and aggregates the results. In a true black-box setting, the attacker cannot control the timestep $t$ or the noise $𝝐$. However, some APIs expose an “encode” function that maps an image to a latent representation, which implicitly involves a forward diffusion step. The attacker can use the encode-decode round-trip error as a membership signal: (Roundtrip)$$\begin{array}{ccc}& R({𝒙}_0)=d_{\text{perc}}({𝒙}_0,\text{Decode}(\text{Encode}({𝒙}_0))).& \text{<span class="tml-eqn"></span>}\end{array}$$ Members typically exhibit smaller round-trip errors because the model's latent space has been optimised to represent them faithfully.

Realistic Attacks on Large Diffusion Models

A persistent criticism of membership inference research is that many attacks are evaluated on small models trained on small datasets, where overfitting is severe and membership signals are artificially strong. dubinski2024towards address this gap by evaluating membership inference attacks on large-scale diffusion models, including Stable Diffusion v1.5 and v2.1 trained on billions of images.

Their findings are sobering. On these large models, most existing attacks achieve AUC close to $0.50$ (random guessing) when applied naively. The reason is that large-scale training with diverse data reduces per-sample overfitting, shrinking the gap between member and non-member loss distributions.

However, dubinski2024towards identify two important exceptions. First, images that appear multiple times in the training set (near-duplicates are common in web-scraped datasets) exhibit significantly stronger membership signals. Second, images that are “atypical” for their class, those with unusual compositions, rare colour palettes, or distinctive features, are more vulnerable because the model must memorise rather than generalise to reconstruct them.

Query-Efficient Attacks

API calls to commercial diffusion models are expensive, both in monetary cost and in computational time. A practical black-box attack must minimise the number of queries while maximising the membership signal. Several strategies achieve this.

Proximal initialisation.

kong2023efficient propose an elegant approach called proximal initialisation. Instead of generating images from random noise, the attacker initialises the reverse diffusion process from a “proximal” starting point that is close to the query image in the noisy space. Specifically, the attacker constructs (Proximal)$$\begin{array}{ccc}& {𝒙}_T^{\text{prox}}=\sqrt{{\bar{\alpha }}_T}{𝒙}_0+\sqrt{1-{\bar{\alpha }}_T}𝝐,𝝐\sim \mathrm{Normal}(0,𝐈),& \text{<span class="tml-eqn"></span>}\end{array}$$ and submits ${𝒙}_T^{\text{prox}}$ as the starting point for generation (if the API supports it). The generated image ${\widehat{𝒙}}_0^{\text{prox}}$ is then compared to ${𝒙}_0$: (STAT)$$\begin{array}{ccc}& S_{\text{prox}}({𝒙}_0)=d_{\text{perc}}({𝒙}_0,{\widehat{𝒙}}_0^{\text{prox}}).& \text{<span class="tml-eqn"></span>}\end{array}$$ For members, the model's denoising trajectory “snaps back” to the memorised image, producing a reconstruction that is very close to ${𝒙}_0$. For non-members, the trajectory drifts, and the generated image deviates more from the query.

Multi-scale querying.

Another query-efficient strategy uses a coarse-to-fine approach. The attacker first generates a small number of low-resolution images (which are cheaper to produce) to identify promising candidates, then re-generates at full resolution only for the most proximal candidates. This hierarchical approach reduces the total computational cost while maintaining attack effectiveness.

Conditioning optimisation.

For text-conditioned models, the choice of text prompt significantly affects the probability of generating images close to the query. The attacker can optimise the prompt (e.g., by captioning the query image with an image captioning model) to maximise the generation proximity statistic. tang2024blackbox show that optimised prompts improve the attack AUC by $5$–$10$ percentage points compared to generic class-level prompts.

White-Box vs. Gray-Box vs. Black-Box: A Unified View

Having developed attacks across the full spectrum of access levels, we can now take a step back and compare them systematically. tab:mia:all:compare provides a comprehensive comparison of all diffusion-model MIA methods covered in Sections 6–10.

llllcc@ Section	Method	Access	Signal	Cost	AUC
Loss-Based and Likelihood-Based Attacks	Loss threshold	Gray-box	${\ell }_t({𝒙}_0)$	$KJ$ F	$0.55$–$0.72$
Loss-Based and Likelihood-Based Attacks	Likelihood ratio (LiRA)	Gray-box	$\mathrm{\Lambda }({𝒙}_0)$	$KJ$ F $+$ ref.	$0.60$–$0.78$
Step-wise Error and Noise Schedule Vulnerability	SecMI (step-wise)	Gray-box	$e_t({𝒙}_0)$	$KJ$ F	$0.62$–$0.80$
White-Box Gradient Attacks on Diffusion Models	Gradient norm	White-box	$G_t({𝒙}_0)$	$KJ$ B	$0.65$–$0.82$
White-Box Gradient Attacks on Diffusion Models	Layer-wise decomp.	White-box	$G_t^{(\ell )}$	$KJ$ B	$0.70$–$0.88$
Frequency-Domain and Noise-Based Attacks	Frequency-domain	Gray-box	$P_t^{(\omega )}$	$KJ$ F	$0.68$–$0.83$
Frequency-Domain and Noise-Based Attacks	Probe noise	Gray-box	${𝒆}_t({𝝐}^{\ast })$	$2KJ$ F	$0.65$–$0.80$
Frequency-Domain and Noise-Based Attacks	Noise aggregation	Gray-box	$V_{\text{agg}}$	$MKJ$ F	$0.63$–$0.78$
Black-Box API Attacks on Diffusion Models	Generation proximity	Black-box	$D_Q({𝒙}_0)$	$Q$ gen.	$0.52$–$0.65$
Black-Box API Attacks on Diffusion Models	Proximal init.	Black-box${}^{\ast }$	$S_{\text{prox}}$	$Q$ gen.	$0.58$–$0.72$
Black-Box API Attacks on Diffusion Models	Round-trip error	Gray-box	$R({𝒙}_0)$	$1$ enc/dec	$0.60$–$0.75$
6@l ${}^{\ast }$Requires API support for custom initial noise.

Several patterns emerge from tab:mia:all:compare:

1. More access yields more power. White-box attacks consistently achieve the highest AUC, followed by gray-box attacks, then black-box attacks. This ordering is expected: more information about the model enables more effective attacks.

2. Diminishing returns from additional access. The gap between white-box and gray-box is often smaller than the gap between gray-box and black-box. The gradient provides a moderate boost over loss-based signals, but the biggest challenge is operating without any direct model access.

3. Timestep decomposition helps universally. Whether using loss, gradient, or frequency signals, exploiting the multi-timestep structure of diffusion models consistently improves attack performance.

4. Computational cost varies widely. Black-box generation is orders of magnitude more expensive per query than a single forward pass, making black-box attacks practical only for targeted queries about specific images.

Practical Considerations and Limitations

Black-box attacks on diffusion models face several practical challenges that merit discussion.

The large-model problem.

As dubinski2024towards demonstrate, the membership signal weakens dramatically for models trained on large, diverse datasets. On Stable Diffusion trained on LAION-5B, most black-box attacks achieve AUC below $0.55$, which is barely above random guessing. The signal is detectable primarily for duplicate or near-duplicate images and for highly atypical samples.

Conditioning mismatch.

The generation proximity statistic depends critically on using an appropriate conditioning input $c$. If the attacker's caption does not accurately describe the query image, the generated images will be systematically different, masking the membership signal. This is a fundamental limitation of generation-based black-box attacks: they require an “inverse” model (image to text) that may itself introduce noise.

Perceptual metric choice.

The choice of perceptual distance metric $d_{\text{perc}}$ affects the attack's sensitivity to different types of memorisation. Pixel-level metrics (MSE, PSNR) are sensitive to exact memorisation but miss semantic memorisation. Feature-level metrics (LPIPS, FID-based distances) capture semantic similarity but may miss pixel-exact copies. An effective attack may need to combine multiple metrics.

Ethical and legal considerations.

Black-box attacks are the most likely to be deployed in practice, because they require no special access to the model. This raises important ethical questions: is it acceptable to probe a commercial API to determine if a specific image was used for training? From the perspective of a data subject exercising their right to know (under GDPR's Article 15, for instance), the answer may be yes. From the perspective of the model provider, such probing may violate terms of service. We discuss these tensions further in The Defense-Attack Arms Race.

Exercises

Differential Privacy for Diffusion Models

If membership inference attacks are the prosecution, differential privacy is the defense attorney. DP provides the only mathematically provable guarantee that a model's outputs reveal limited information about any individual training sample. But in the world of diffusion models, this defense comes at a steep price. The high dimensionality of image generation, the millions of parameters in a typical U-Net denoiser, and the iterative nature of score matching training all conspire to make DP training extraordinarily expensive, both in computational cost and in generation quality. This section investigates whether that price is worth paying and, if so, how to minimise it.

Recall from The Gaussian Mechanism and DP-SGD that differential privacy formalises the intuition that a model trained on a dataset $𝒟$ should produce nearly the same outputs as a model trained on a neighbouring dataset ${𝒟}^{\prime }$ differing in a single record. The mechanism for achieving this during stochastic gradient descent is DP-SGD: clip each per-sample gradient to a fixed norm and add calibrated Gaussian noise. We now adapt this machinery to the specific structure of diffusion model training.

DP-SGD Adapted to Score Matching

The standard diffusion training objective asks the denoiser ${𝝐}_{𝜽}$ to predict the noise $𝝐$ added at timestep $t$: (Score LOSS)$$\begin{array}{ccc}& ℒ(𝜽)={𝔼}_{t,{𝒙}_0,𝝐}\left[{\Vert {𝝐}_{𝜽}({𝒙}_t,t)-𝝐\Vert }^2\right],& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${𝒙}_t=\sqrt{{\bar{\alpha }}_t}{𝒙}_0+\sqrt{1-{\bar{\alpha }}_t}𝝐$ and $𝝐\sim \mathrm{Normal}(0,𝐈)$. Each training sample ${𝒙}_0$ contributes a per-sample loss ${\ell }_t({𝒙}_0)={\Vert {𝝐}_{𝜽}({𝒙}_t,t)-𝝐\Vert }^2$. The key observation is that this per-sample loss has bounded sensitivity (for bounded data), making it amenable to gradient clipping.

The clipping operation ensures that no single training sample can exert disproportionate influence on the gradient update, while the Gaussian noise masks the contribution of any individual record. Together, these two mechanisms provide the formal DP guarantee.

We can now assemble these ideas into a complete training procedure.

Privacy Accounting for Multi-Step Training

Diffusion models require far more training iterations than typical discriminative models; DDPM on CIFAR-10 trains for 800K iterations, and large-scale models like Stable Diffusion train for millions. Each iteration consumes a portion of the privacy budget, so accurate accounting is critical for achieving reasonable utility at a given $\epsilon$.

The amplification factor $q=B/n$ is the key to practical DP training: by using small batch fractions, we can dramatically reduce the per-step privacy cost, allowing more training iterations before exhausting the budget.

Quality-Privacy Tradeoff: FID versus Epsilon

The central question for any practitioner is: how much generation quality must I sacrifice for a given level of privacy? The answer, for current diffusion models, is sobering.

The table reveals a stark tradeoff. Moving from $\epsilon =\infty$ (no privacy) to $\epsilon =10$ increases FID by nearly an order of magnitude, from 5 to 48. At $\epsilon =1$, widely considered a “strong” privacy guarantee, the FID exceeds 180, rendering the generated images essentially useless for high-fidelity applications. On the other hand, the MIA AUC drops dramatically, approaching the random-guess baseline of 0.50.

This lower bound explains why DP training is particularly painful for image generation. For CIFAR-10 at $32\times 32\times 3$, $d=\mathrm{3,072}$; for $256\times 256$ images, $d=\mathrm{196,608}$. The quadratic scaling with $d$ means that higher-resolution generation faces exponentially steeper privacy-quality tradeoffs.

To make the scaling concrete, consider the minimum FID achievable at $\epsilon =10$ for various resolutions:

Scalability Challenges

Beyond the quality degradation, DP-SGD imposes severe computational burdens that make it challenging to apply to large-scale diffusion models.

Per-sample gradient computation.

Standard SGD computes a single gradient for the entire minibatch using efficient vectorised operations, at a cost of $O(|𝜽|)$. DP-SGD requires per-sample gradients so that each can be individually clipped. The naive approach computes $B$ separate forward-backward passes, increasing the cost to $O(B\cdot |𝜽|)$. For a minibatch of size $B=64$ and a U-Net with $|𝜽|\approx {10}^8$ parameters, this represents a $64\times$ increase in memory and a substantial increase in wall-clock time.

Gradient clipping bias.

Clipping introduces a systematic bias: $𝔼[\text{clip}(𝒈,C)]\ne 𝔼[𝒈]$ whenever $\mathrm{Pr}(\Vert 𝒈\Vert >C)>0$. Larger clip norms reduce this bias (since fewer gradients are clipped) but require proportionally more noise ($\sigma C$) to maintain the same privacy level. This creates a bias-variance tradeoff that must be carefully tuned.

Memory overhead.

Storing $B$ per-sample gradient vectors of dimension $|𝜽|$ requires $B\cdot |𝜽|$ floats. For a Stable Diffusion U-Net with $\sim 860$M parameters and $B=32$, this amounts to roughly 100,GB in float32, far exceeding typical GPU memory.

The scalability challenges are real but not insurmountable. The field is actively developing more efficient DP training algorithms, and the gap between private and non-private training costs continues to narrow.

Higher-Order Dynamics and Regularization Defenses

What if the diffusion process itself could be modified to inherently resist privacy attacks? Higher-order Langevin dynamics offers precisely this: by introducing auxiliary variables that mix external randomness, the model's memory of individual training samples is naturally diluted. Unlike DP, which adds noise to the training procedure, this family of defenses modifies the structure of the diffusion process, achieving privacy through physics-inspired dynamics rather than cryptographic noise.

HOLD++: Higher-Order Langevin Dynamics Defense

The standard diffusion SDE evolves the data through a first-order stochastic process: (Standard SDE)$$\begin{array}{ccc}& \mathrm{d}𝒙=𝒇(𝒙,t)\mathrm{d}t+g(t)\mathrm{d}𝒘,& \text{<span class="tml-eqn"></span>}\end{array}$$ where $𝒇$ is the drift coefficient, $g(t)$ is the diffusion coefficient, and $𝒘$ is a standard Wiener process. HOLD++ extends this to a second-order system by introducing an auxiliary momentum variable $𝒚$.

The physical analogy is illuminating. In the standard (overdamped) diffusion, particles move through a viscous medium where inertia is negligible: they stop as soon as the driving force vanishes. In HOLD++, particles have mass and momentum. When the score function points toward a training sample, the particle does not simply drift there and settle; its momentum carries it through and beyond, eventually settling into the broader basin of the data distribution rather than the precise location of any individual sample.

To see this mathematically, consider the effective “memory” of the system at time $t$. In the standard SDE, the position ${𝒙}_t$ is a Markov process: its future depends only on its current state. In HOLD++, the joint state $({𝒙}_t,{𝒚}_t)$ is Markov, but the marginal ${𝒙}_t$ is not Markov because its future depends on the momentum ${𝒚}_t$. This non-Markovian structure in the position variable acts as a form of “controlled forgetting”: the model cannot simply memorise and replay a training point's exact position, because the momentum term continuously perturbs the trajectory.

Mathematical Analysis of Convergence

The joint density $p(𝒙,𝒚,t)$ of the HOLD++ process satisfies the Kramers equation (the Fokker-Planck equation for the underdamped system): (Kramers)$$\begin{array}{ccc}& \frac{\partial p}{\partial t}=-{\nabla }_{𝒙}\cdot (𝒚p)-{\nabla }_{𝒚}\cdot ([𝒇(𝒙,t)-\gamma 𝒚]p)+\frac{g(t{)}^2}{2}{\mathrm{\Delta }}_{𝒚}p,& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${\mathrm{\Delta }}_{𝒚}$ denotes the Laplacian with respect to $𝒚$. The stationary distribution of this equation, when $𝒇$ is derived from the data score, takes the form $p_{\infty }(𝒙,𝒚)\propto p_{\text{data}}(𝒙)\cdot \mathrm{Normal}(𝒚;0,{\gamma }^{-1}𝐈)$, so the marginal over $𝒙$ recovers the target data distribution exactly.

Regularisation Techniques for Score Matching

Beyond modifying the dynamics, we can directly regularise the score-matching objective to discourage memorisation. The intuition is that overly sharp score functions, those that assign dramatically different scores to training points versus nearby non-training points, are both a sign of overfitting and the primary signal exploited by MIA.

Data Augmentation as Implicit Privacy

A surprisingly effective and computationally cheap defense against membership inference is aggressive data augmentation. The mechanism is intuitive: if the model sees $K$ distinct augmented versions of each training sample, its “memory” of any specific version is diluted by a factor of $K$.

The Defense-Attack Arms Race

Every lock eventually encounters a skilled lockpick. The history of membership inference attacks on diffusion models is a story of escalation: each new defense inspires a more sophisticated attack, which in turn motivates a stronger defense. Understanding this arms race is essential for knowing where we truly stand. In this section, we formalise the adversarial interplay between attackers and defenders, establish minimax bounds on what is achievable, and survey the current state of the field.

Adaptive Attacks against Defenses

A defense that withstands only the attacks it was designed against provides a false sense of security. The true test of any defense is its resilience against adaptive attacks, those specifically crafted to exploit the defense mechanism itself.

Minimax Formulation

To reason rigorously about the arms race, we need a mathematical framework that accounts for both the attacker's and the defender's strategies simultaneously. The minimax formulation provides exactly this.

Current State and Open Questions

As of 2025, the arms race between attackers and defenders of diffusion model privacy remains unresolved. Several key observations characterise the current landscape.

The table reveals a striking pattern: HOLD++ achieves comparable MIA AUC reduction to DP-SGD at $\epsilon =5\text{–}10$ with dramatically less FID degradation and computational overhead. However, HOLD++ lacks the formal worst-case guarantees of DP. The choice between them depends on the threat model and the stakes involved.

This question represents perhaps the most important open direction in diffusion model privacy. A positive answer would revolutionise the field by eliminating the painful quality-privacy tradeoff. A negative answer, proving that such models cannot exist, would be equally valuable by establishing fundamental limits.

Attacks on Diffusion Language Models

Diffusion language models represent a fascinating hybrid: they apply the denoising diffusion framework to discrete text, using bidirectional masking rather than unidirectional generation. This bidirectionality, while powerful for generation, opens an exponentially larger attack surface for membership inference. Where autoregressive language models expose one prediction per token position (the next token given all previous tokens), diffusion language models expose predictions for any subset of tokens given any other subset. As we shall see, this exponential expansion of the attack surface is not merely theoretical; it translates directly into dramatically more powerful attacks.

For the foundations of diffusion language models, including the masking process, the denoising objective, and the connection to discrete diffusion, we refer the reader to sec:diffusion_language.

SAMA: Subset-Aggregated Membership Attack

The key insight behind SAMA (Subset-Aggregated Membership Attack) [10] is that each masking configuration provides an independent “view” of the model's memorisation. By aggregating many such views, the attack achieves far greater statistical power than any single-mask probe.

Each score $s_{𝒎}(𝒙)$ measures how surprised the model is when asked to predict the masked tokens from the unmasked context. For training members, the model has seen the complete sequence during training and should predict the masked tokens more confidently (lower $s_{𝒎}$) than for non-members.

The innovation of SAMA is to aggregate scores across many different masking patterns, exploiting the fact that different masks probe different aspects of memorisation.

The sign-based aggregation is critical. Individual masked prediction scores can be highly variable (some masks may cover uninformative regions of the text), but the direction of the score relative to the threshold is a much more stable signal. This is analogous to the use of sign-based gradient aggregation in distributed optimisation for robustness to Byzantine failures.

Why Diffusion LMs Are Doubly Vulnerable

The power of SAMA stems from a fundamental structural property of diffusion language models: bidirectionality. While autoregressive models process text in a single direction (left to right), diffusion language models can condition on any subset of positions and predict any other subset. This flexibility, which is a strength for generation, is a catastrophic weakness for privacy.

Evaluation Benchmarks for Diffusion Model MIA

A fair comparison of MIA methods requires standardised evaluation. Unfortunately, the current literature is heterogeneous, with different papers using different datasets, model architectures, train/test splits, and metrics. We compile the most comprehensive available results in tab:mia:diff:lang:benchmark-summary.

Several patterns emerge from the benchmark table. First, attack power increases with access level: loss-based attacks (black-box) are weakest, followed by intermediate-access attacks (SecMI, PIA), with white-box gradient attacks being strongest. Second, larger and more diverse training sets (LAION-5B) are harder to attack than smaller curated sets (CIFAR-10, CelebA), consistent with the intuition that less memorisation occurs when data is abundant. Third, diffusion language models are dramatically more vulnerable than image diffusion models, with SAMA achieving AUC above 0.80 compared to the best image attacks at 0.78.

Summary of the Diffusion Model Half

Let us step back and survey the terrain we have covered since Differential Privacy for Diffusion Models. The investigation of membership inference attacks on diffusion models reveals a consistent narrative with several recurring themes.

We now turn to large language models, where the memorisation mechanisms are different but equally concerning. Why Large Language Models Memorize begins this investigation by examining how autoregressive training creates memorisation in the text domain and why the resulting privacy vulnerabilities, while structurally different from those in diffusion models, are no less severe.

We have spent the first half of this chapter in the visual world of diffusion models, where membership leaks through denoising errors, gradient patterns, and frequency signatures. We now cross into the linguistic realm. Large language models process sequences of tokens, predict next words, and generate fluent text. Their attack surface is fundamentally different: instead of pixel-level memorization, LLMs exhibit token-level memorization, where verbatim sequences from training data can be extracted under the right conditions. The detective's toolkit must adapt accordingly.

Where diffusion models reveal membership through the geometry of the denoising trajectory, language models reveal membership through probability assignments: the confidence with which the model predicts the next token in a sequence. A training example leaves a distinctive probabilistic fingerprint, one that persists even after billions of gradient updates on millions of other examples. The attacks we develop in the next four sections exploit this fingerprint with increasing sophistication, from simple perplexity thresholds to elegant outlier-token statistics that require no reference models at all.

The stakes, if anything, are higher. Modern LLMs are trained on vast corpora scraped from the internet, encompassing personal emails, medical records, copyrighted books, and private conversations. A successful membership inference attack can confirm that a specific individual's data was used in training, with all the legal and ethical implications that follow. Let us begin by understanding why these models memorize in the first place.

Why Large Language Models Memorize

In January 2021, Carlini et al. demonstrated something remarkable and troubling: by prompting GPT-2 with specific prefixes, they could extract verbatim passages from its training data, including personal phone numbers, email addresses, and unique text sequences that appeared only once in the training corpus [12]. If a 1.5 billion parameter model could memorize and regurgitate individual training examples, what about models with hundreds of billions of parameters?

This section investigates the mechanisms behind memorization in large language models. We establish scaling laws that relate model size, dataset size, and training duration to the fraction of data memorized (Scaling Laws and Memorization); examine how data duplication and long-tail distributions influence memorization risk (Data Deduplication and Long-Tail Distributions); provide formal definitions of memorization at multiple levels of fidelity (Formal Definitions of Memorization); and describe the extraction attacks that first revealed the scope of the problem (Extracting Memorized Content: The Carlini Attack).

Scaling Laws and Memorization

The relationship between model scale and memorization is not accidental; it follows predictable scaling laws that mirror the well-known scaling laws for language modelling loss.

The role of training duration.

The epoch exponent $\beta \approx 0.7$ indicates a sub-linear relationship between training duration and memorization. The first epoch produces the most dramatic increase in memorization; subsequent epochs provide diminishing returns. This is because early training steps learn the most distinctive features of each example (establishing initial memorization), while later steps refine the model's distributional knowledge with smaller, more incremental updates.

Data Deduplication and Long-Tail Distributions

Not all training examples are equally vulnerable to memorization. The frequency with which a sequence appears in the training corpus is a primary determinant of whether it will be memorized.

The empirical evidence is compelling. Carlini et al. [11] found that sequences appearing 10 or more times in The Pile were memorized at rates 10 to 100 times higher than unique sequences. A Wikipedia paragraph duplicated across multiple web pages may appear hundreds of times, making it almost certain to be memorized. A personal email, by contrast, appears at most once.

Formal Definitions of Memorization

To reason precisely about memorization, we require formal definitions that capture the phenomenon at different levels of fidelity. The definitions below, drawn from the work of Carlini et al. [12][11], formalize the intuitive notion that a model “remembers” a training example. Each definition progressively relaxes the requirements, encompassing a broader class of memorized content.

Extracting Memorized Content: The Carlini Attack

The extraction attack proceeds in three stages: diverse generation, memorization scoring, and verification.

The memorization score $s_j$ is the perplexity ratio: the difference in log-likelihood between the target model and a reference model. A sequence that the target model assigns high probability to, but the reference model does not, is likely memorized from the target's specific training data.

Why high temperature helps.

The use of high temperature ($T\ge 1.0$) during generation is counterintuitive at first: wouldn't we want the model to generate its most confident predictions? The answer reveals a subtle point about memorization. At low temperature, the model generates generic, high-probability text that resembles training data in distribution but not in specific content. At high temperature, the model explores a wider region of the output space, occasionally stumbling into the exact token sequences it has memorized. The diversity of high-temperature sampling acts as a probe, sweeping across the model's memory to find the sequences most deeply encoded in its parameters.

The role of prompt design.

The choice of prompts $𝒫$ significantly affects extraction success. Three strategies have proven effective:

1. Context prompts: Using known context that precedes the target content (e.g., the first sentence of an article). This is the most direct approach but requires partial knowledge of the training data.

2. Topic prompts: Using topical keywords or domain-specific language to steer generation toward regions of the training data likely to contain sensitive content.

3. Empty or random prompts: Using minimal or random context to elicit the model's “default” memorized content. Surprisingly effective for highly memorized sequences (e.g., famous quotes, common code snippets).

Perplexity and Loss-Based Attacks

The simplest membership signal for a language model is perplexity: how surprised is the model by a given text? Training data should, on average, be less surprising. This intuition is the foundation of the most widely used LLM membership inference attacks.

The idea is deceptively simple, yet its formal analysis reveals subtleties involving token-level signal distribution, the interplay between text difficulty and memorization, and the need for careful threshold calibration. We develop the full theory in this section, progressing from the basic perplexity score (Perplexity as a Membership Signal), through per-token analysis that reveals where the signal concentrates (Per-Token Loss Analysis), to the formal characterization of member vs. non-member loss distributions (Member vs. Non-Member Loss Distributions), and finally to the practical question of threshold calibration (Threshold Selection and Calibration).

Perplexity as a Membership Signal

Black-box access requirements.

A crucial practical advantage of perplexity-based attacks is their minimal access requirement. The attacker needs only the ability to query the model for per-token log-probabilities, which many API-based LLM services provide (including the OpenAI API's logprobs parameter). No access to model weights, gradients, or internal activations is required. This makes the perplexity-based attack applicable to commercial, closed-source models, a significant practical concern for organizations deploying proprietary LLMs.

Even when per-token log-probabilities are not directly available, the attacker can approximate them by querying the model's top-$k$ token predictions at each position. If the true next token appears among the top-$k$ predictions, its probability can be read directly; otherwise, a lower bound can be used. This approximation introduces some noise but preserves the attack's effectiveness for large $k$ (e.g., $k=100$).

Per-Token Loss Analysis

The average perplexity treats all tokens equally, but not all tokens carry equal membership signal.

Member vs. Non-Member Loss Distributions

The perplexity-based attack succeeds when the loss distributions of members and non-members are sufficiently separated. We now quantify this separation.

Implications for fine-tuned models.

The loss separation theorem has particularly strong implications for fine-tuned language models. When a pretrained model is fine-tuned on a small, specialized dataset ($n$ is small), the gap $c\eta E/n$ becomes large, making membership inference significantly easier. This is a critical concern for organizations that fine-tune LLMs on proprietary or sensitive data (medical records, legal documents, customer support transcripts): the fine-tuned model leaks substantially more membership information than the base model.

Threshold Selection and Calibration

Selecting the optimal threshold $\tau$ is critical. A threshold that is too permissive yields many false positives; one that is too conservative misses true members.

Calibration procedure.

In the absence of a pre-specified threshold, the attacker can use the following calibration strategy:

1. Collect a set of $m$ known members ${𝒟}_{\text{in}}$ and $m$ known non-members ${𝒟}_{\text{out}}$.

2. Compute scores $\{s_{\text{PPL}}(𝒙):𝒙\in {𝒟}_{\text{in}}\cup {𝒟}_{\text{out}}\}$.

3. Fit Gaussian distributions to the two score sets.

4. Set $\tau ={\tau }^{\ast }$ from Proposition 38, or select $\tau$ to maximize the desired metric (e.g., balanced accuracy, F1, or TPR at fixed FPR) on the calibration set.

Reference-Based Attacks

A detective doesn't just ask “does the suspect know the victim?” They ask “does the suspect know the victim better than a random person would?” Reference-based attacks apply this comparative logic: they measure not the absolute loss on a sample, but the relative loss compared to a reference model that definitely did not see the sample during training.

This comparative approach eliminates the confound between text difficulty and memorization that plagues pure perplexity attacks. A medical journal article has high perplexity under any language model because it contains rare terminology; but if the target model assigns lower perplexity than a reference model of similar capability, the difference is attributable to memorization.

LiRA: Likelihood Ratio Attack

The Likelihood Ratio Attack (LiRA), introduced by Carlini et al. [4], is the gold standard for membership inference. It is a direct instantiation of the Neyman-Pearson hypothesis testing framework.

The full LiRA procedure is summarized in the following algorithm.

Gaussian Score Assumption and Justification

The power of LiRA rests on the Gaussian assumption for loss distributions. We now justify this assumption and establish the optimality of the resulting test.

When Gaussianity fails.

The Gaussian assumption can break down in specific scenarios:

1. Very short sequences ($L<50$ tokens): the central limit theorem approximation is poor, and the loss distribution may exhibit skewness or heavy tails.

2. Highly structured text (e.g., code, formulae): the per-token losses have strong dependencies, violating the approximate independence needed for the CLT.

3. Mixed-domain text: a sequence containing both common and highly specialized content may produce a bimodal loss distribution.

In these cases, non-parametric density estimation (kernel density estimation or histogram-based methods) can replace the Gaussian fit, at the cost of requiring more reference models ($k\ge 128$) to achieve smooth density estimates. Carlini et al. [4] found that the Gaussian approximation remains near-optimal for the vast majority of evaluation sequences, making it the default choice in practice.

Reference Model Selection and Training

The practical effectiveness of LiRA depends critically on the quality and quantity of reference models.

Architecture mismatch effects.

When exact architecture matching is infeasible (the target model's architecture may be proprietary), the attacker can use a proxy architecture. The degradation in attack performance depends on the representation similarity between the proxy and the target. Models from the same family (e.g., different sizes of GPT-Neo) show high representation similarity and yield effective reference models. Models from different families (e.g., using LLaMA as a reference for GPT-4) show lower similarity, but the attack still outperforms perplexity-only baselines because the comparative approach removes most of the text-difficulty confound even with imperfect references.

Data overlap considerations.

The reference models should be trained on data drawn from the same distribution as the target model's training data. If the reference data is drawn from a very different distribution (e.g., using only Wikipedia to reference a model trained on diverse web text), the loss comparison becomes unreliable. In practice, using a large public corpus (C4, The Pile, RedPajama) as auxiliary data works well for most web-trained LLMs, since these corpora overlap substantially with the training data of commercial models.

Shadow Model Computational Cost

The main limitation of LiRA is computational: training $2k$ reference models can be prohibitively expensive for large language models.

This enormous computational burden motivates the search for attacks that achieve comparable performance without reference models. Two prominent alternatives are:

• Self-calibration methods that use the target model itself as its own reference, normalizing by text difficulty without external models.

• Outlier-token methods (such as Min-K% Prob, Outlier Token Detection: Min-K% Prob) that exploit distributional properties of per-token probabilities without any reference.

We close this section with a comprehensive comparison of the attacks discussed so far and those that follow.

Outlier Token Detection: Min-K% Prob

In 2024, Shi et al. [14] proposed an elegant insight: if you have never read a particular book, you are likely to stumble on a few unusual words that catch you off guard. A language model behaves similarly; for an unseen text, there will be a handful of tokens that receive surprisingly low probability. This observation, formalized as Min-K% Prob, provides one of the most practical and efficient membership inference attacks on large language models.

Unlike LiRA, which requires training dozens of reference models, Min-K% Prob requires only a single forward pass through the target model. Unlike raw perplexity, which averages over all tokens (including uninformative function words), Min-K% Prob focuses on the tokens that carry the strongest membership signal. The result is a method that is both computationally efficient and surprisingly powerful.

Min-K% Prob: The Key Idea

The intuition is best understood through analogy. Consider a student who has read a textbook cover-to-cover versus one who has not. If you ask both to fill in missing words in a passage from the textbook, both will handle common words (“the”, “and”, “is”) with ease. The difference emerges on the unusual words: technical terms, specific names, and rare phrases. The student who has read the book will handle even these unusual words with relative confidence, while the one who has not will stumble. Min-K% Prob measures this “stumbling” by examining the worst-predicted tokens.

The logic is as follows: for a training member, the model has been optimized to predict even the difficult tokens, so the bottom $k\%$ log-probabilities are relatively high. For a non-member, the model has never directly optimized for these specific tokens, so the bottom $k\%$ log-probabilities include some genuinely surprising tokens with very low probability. The score for non-members is therefore lower (more negative), creating a separation between the two classes.

Order Statistics and Outlier Detection

The theoretical foundation of Min-K% Prob rests on order statistics, the study of sorted random variables.

Consistency and Convergence

We now establish that Min-K% Prob is a theoretically sound statistical test with well-characterized convergence properties.

Practical Implementation and the WikiMIA Benchmark

We now turn to the practical deployment of Min-K% Prob and its evaluation on standardized benchmarks.

Practical considerations.

Several implementation details affect Min-K% Prob's real-world performance:

1. Choice of $k$. The default recommendation is $k=20\%$, but this should be tuned on a validation set if available. For very long sequences ($L>500$), smaller $k$ values ($10\%$ or even $5\%$) can be effective because there are still enough tokens to average over.

2. Tokenization effects. The attack operates on the model's native tokenization. Subword tokenizers (BPE, SentencePiece) split rare words into multiple tokens, which can distribute the “surprise” of a rare word across several subtokens. This does not eliminate the signal but may dilute it slightly.

3. Prompt formatting. If the model uses a chat template or system prompt, the query text should be formatted consistently. Including system tokens in the probability computation can introduce noise.

4. Sequence length. Min-K% Prob becomes unreliable for very short sequences ($L<30$), where $\lfloor kL/100\rfloor$ may be only a few tokens and the variance of the score is high. For best results, use passages of at least 100 tokens.

5. Document-level vs. sentence-level. Meeus et al. [16] showed that document-level membership inference (querying an entire document rather than a single sentence) significantly improves attack performance. The Min-K% score computed over an entire document benefits from more tokens and captures cross-sentence memorization patterns that sentence-level analysis misses.

Comparison with perplexity on different text types.

The advantage of Min-K% Prob over perplexity is not uniform across text types. The improvement is largest for:

• Mixed-register text (e.g., news articles, academic papers) where the overall perplexity is dominated by common words, but key facts and names provide strong outlier signals.

• Long documents where the law of large numbers makes average perplexity similar for members and non-members, but the extreme order statistics remain discriminative.

• Texts with distinctive vocabulary (medical records, legal documents, scientific papers) where rare terminology creates strong outlier tokens.

The advantage is smallest for:

• Very short texts ($L<50$) where both methods suffer from high variance.

• Homogeneous text (e.g., simple conversational language) where all tokens have similar probability and there are no clear outliers.

Self-Prompted Verification Attacks

The attacks of the preceding sections share a common architectural dependence: they require either external reference models (as in LiRA, LiRA: Likelihood Ratio Attack) or carefully tuned percentile statistics (as in Min-K% Prob, Outlier Token Detection: Min-K% Prob). Reference-based methods demand the computational expense of training dozens of shadow models; percentile methods, while cheaper, sacrifice statistical power by discarding calibration information entirely. Is there a middle path-a method that achieves the calibration benefits of reference models without actually training any?

In 2024, fu2024membership proposed a startling answer: let the target model be its own reference. The resulting attack, Self-Prompted Verification MIA (SPV-MIA), exploits a simple but profound observation: if a text passage $𝒙$ is a training member, then the target model $f_{𝜽}$ should be able to regenerate text similar to $𝒙$ when prompted appropriately. The likelihood of $𝒙$ can then be compared not against an external baseline, but against the likelihood of the model's own regeneration. The detective, in essence, asks the suspect to repeat their story and checks for consistency.

This section develops the mathematical framework of SPV-MIA, analyses its statistical properties, and compares it against the reference-based and reference-free methods from sec:mia:llm:reference:lira,sec:mia:llm:minkprob.

The Self-Calibration Principle

The fundamental challenge of membership inference is calibration. Observing that $\log p_{𝜽}(𝒙)$ is “high” is meaningless without a reference point: high compared to what? LiRA answers this question by training IN and OUT reference models and comparing their loss distributions. Min-K% Prob sidesteps the question by focusing on outlier tokens whose probabilities are informative regardless of calibration. SPV-MIA offers a third answer: compare the target text against text that the model itself generates as a surrogate.

The intuition is as follows. If $𝒙$ is a training member, then $f_{𝜽}$ has been optimized to assign high probability to the specific token sequence in $𝒙$. The model's own continuation $\widetilde{𝒙}$ will be a plausible completion but will lack the precise memorization fingerprint that distinguishes the original. Consequently, the likelihood gap $\log p_{𝜽}(𝒙)-\log p_{𝜽}(\widetilde{𝒙})$ should be positive for members (the original is more likely than the regeneration) and near zero for non-members (both are equally unfamiliar).

The score $S_{\text{SPV}}$ measures how much better the model fits the original text compared to its own regenerations. This is a self-calibrated likelihood ratio: instead of comparing against an external reference model, we compare against the model's own generative distribution.

Mathematical Framework

We now formalize the statistical properties of the SPV-MIA score. Let $𝒙$ be a text sequence of length $L$, and let $m$ be the prefix length used for self-prompting.

Prompt Design Strategies

The effectiveness of SPV-MIA depends critically on how the prefix ${𝒙}_{1:m}$ is used to prompt the model. fu2024membership identify three principal strategies, each with distinct advantages.

Direct prefix prompting.

The simplest approach uses the first $m$ tokens of $𝒙$ as the prompt and samples continuations directly. This works well when the prefix is sufficiently informative to constrain the continuation's topic and style. The prefix length $m$ creates a bias-variance trade-off: longer prefixes produce continuations more similar to the original (reducing variance) but also reduce the portion of $𝒙$ available for comparison (reducing signal).

Paraphrase prompting.

Instead of using the raw prefix, the attacker constructs a prompt that paraphrases or summarizes ${𝒙}_{1:m}$, then asks the model to continue. This strategy is useful when the exact prefix might trigger memorized verbatim recall in both the member and non-member cases, which would reduce discriminative power. By paraphrasing the prefix, the attacker forces the model to generate from semantic understanding rather than surface-level pattern matching.

Topic-guided prompting.

The attacker extracts the topic or domain of $𝒙$ (e.g., “a news article about climate change policy in the European Union”) and prompts the model to generate text on the same topic. This produces reference texts that share the domain and vocabulary of $𝒙$ but differ in specific content. Topic-guided prompting is particularly effective for documents in specialized domains where the baseline loss varies significantly across topics.

The SPV-MIA Pipeline

The full SPV-MIA procedure, illustrated in fig:mia:spv:pipeline, operates in four stages:

1. Prefix extraction. Given the target text $𝒙$ of length $L$, extract a prefix of length $m$ tokens (typically $m\approx L/4$ to $L/3$).

2. Self-prompted generation. Using the prefix ${𝒙}_{1:m}$ (possibly paraphrased), generate $K$ continuations from $f_{𝜽}$ using nucleus sampling with $p=0.95$ and temperature $T=1.0$.

3. Loss computation. Compute the per-token cross-entropy loss ${\ell }_{𝜽}(𝒙)$ for the original text and ${\ell }_{𝜽}({\widetilde{𝒙}}^{(j)})$ for each reference text, both evaluated only over positions $m+1$ through $L$.

4. Score and threshold. Compute $S_{\text{SPV}}(𝒙)$ via (Score) and classify as member if $S_{\text{SPV}}>\tau$.

Decoding Strategy and Temperature Effects

The quality of the self-prompted references depends critically on the decoding strategy used to generate them. Different strategies trade off diversity against fidelity, with direct consequences for the SPV-MIA score's variance and discriminative power.

Greedy decoding.

Selecting the most probable token at each step ($x_{t+1}=\arg {\max }_v{p}_{𝜽}(v|{𝒙}_{1:t})$) produces deterministic continuations. While this eliminates sampling variance, it also eliminates diversity: all $K$ self-prompted references are identical. The SPV-MIA score reduces to a single-sample comparison $S_{\text{SPV}}={\ell }_{𝜽}({\widetilde{𝒙}}_{\text{greedy}})-{\ell }_{𝜽}(𝒙)$, which has no internal variance estimate and cannot benefit from averaging. More fundamentally, greedy decoding produces continuations that are maximally biased toward the model's mode, which may itself carry memorization artifacts.

Temperature scaling.

Sampling with temperature $T$ modifies the softmax distribution: $p_T(v)\propto p_{𝜽}{(v)}^{1/T}$. Low temperatures ($T<1$) sharpen the distribution, producing continuations similar to greedy decoding; high temperatures ($T>1$) flatten it, producing more diverse but potentially less coherent continuations. The optimal temperature for SPV-MIA balances two considerations:

• Diversity: Higher $T$ increases the variance of ${\ell }_{𝜽}({\widetilde{𝒙}}^{(j)})$ across samples, providing better coverage of the model's generative distribution.

• Relevance: Lower $T$ keeps the references closer to the target text's topic and style, ensuring that the loss comparison is meaningful.

Nucleus sampling.

Nucleus (top-$p$) sampling [29] restricts sampling to the smallest set of tokens whose cumulative probability exceeds $p$, then renormalises. This produces diverse yet coherent text by excluding the low-probability tail of the distribution. fu2024membership recommend nucleus sampling with $p=0.95$ as the default strategy, finding that it provides the best balance between diversity and relevance for SPV-MIA.

Comparison with LiRA and Min-K%

How does SPV-MIA compare with the established baselines? The answer depends on the threat model and computational budget.

Computational cost.

LiRA requires training $2K$ reference models (typically $K=64$ to $K=256$), each comparable in size to the target model. This makes LiRA prohibitively expensive for models with billions of parameters. Min-K% Prob requires a single forward pass through the target model: $O(L)$ computation. SPV-MIA requires $K+1$ forward passes (one for the target text and $K$ for the self-prompted references), where $K$ is typically 10 to 50. SPV-MIA is thus orders of magnitude cheaper than LiRA but moderately more expensive than Min-K%.

Statistical power.

LiRA achieves the highest statistical power because it directly estimates the likelihood ratio between the IN and OUT distributions. SPV-MIA approximates this ratio using the model's own generative distribution as a proxy for the OUT distribution, which is imperfect (the model's generations are drawn from the learned distribution, not from models that excluded $𝒙$). Min-K% Prob captures a different signal entirely (outlier token probabilities) and can be either stronger or weaker than SPV-MIA depending on the text characteristics.

Access requirements.

All three methods require only black-box access to the target model's token probabilities. Min-K% requires probabilities for each token in $𝒙$; SPV-MIA additionally requires the ability to sample completions from the model; LiRA requires the ability to train reference models on subsets of the training data. In practice, SPV-MIA's access requirements are the easiest to satisfy among the calibrated methods, since it only needs API access to the target model (no training data access required).

The key insight from Proposition 47 is that SPV-MIA is a faithful approximation to LiRA when the distributional shift term is small-that is, when the model's generative distribution is a good proxy for the OUT distribution. This condition holds approximately for large language models trained on diverse corpora, where the omission of any single document from the training set has a negligible effect on the model's overall distribution.

Exercises

Sampling-Based and Pseudo-Likelihood Attacks

The attacks presented so far exploit the autoregressive factorisation of language model probabilities: $p_{𝜽}(𝒙)={\prod }_{t=1}^L{p}_{𝜽}(x_t|{𝒙}_{1:t-1})$. This factorisation gives us the conditional probability of each token given its left context, and the product yields the joint probability of the entire sequence. But this product is not the only way to measure how well a model “knows” a text. In fact, the autoregressive joint probability can be a misleading membership signal: a long text with consistently moderate per-token probabilities may have a very low joint probability simply because the chain of conditionals multiplies many values below one.

A different perspective comes from pseudo-likelihood estimation, a technique with roots in spatial statistics and undirected graphical models. Instead of measuring how well the model predicts each token from its left context, pseudo-likelihood measures how well the model predicts each token from its entire surrounding context-left and right. For masked language models like BERT, this is natural: the model is trained to predict masked tokens from bidirectional context. For autoregressive models, computing pseudo-likelihood requires a clever workaround.

zhang2024samia introduced SaMIA (Sampling-based Membership Inference Attack), which uses Monte Carlo sampling to estimate a pseudo-likelihood score for autoregressive models. The key idea is to repeatedly mask random tokens in the target text, sample replacements from the model, and measure how often the model's samples match the original tokens. This transforms the unidirectional autoregressive model into an approximate bidirectional scorer.

The Pseudo-Likelihood Problem

For a masked language model (MLM) such as BERT, computing $p_{𝜽}(x_t|{𝒙}_{\setminus t})$ is straightforward: mask position $t$ and read off the softmax probability. For an autoregressive model, this conditional is not directly available. The model computes $p_{𝜽}(x_t|{𝒙}_{1:t-1})$, conditioning only on the left context. To condition on both left and right context, we need either a different model architecture or an approximation scheme.

The numerator in (Estimator) measures the joint probability of the original text when token $t$ is fixed at $x_t$, and the denominator normalizes over all possible tokens at position $t$. This ratio is exactly $p_{𝜽}(x_t|{𝒙}_{\setminus t})$ if the model were bidirectional; for an autoregressive model, the right-context term $p_{𝜽}({𝒙}_{t+1:L}|{𝒙}_{1:t-1},v)$ captures the influence of the right context on the probability of $x_t$.

Monte Carlo Estimation and Importance Sampling

Computing the exact denominator in (Estimator) requires summing over the entire vocabulary $𝒱$ (typically $|𝒱|\ge \mathrm{32,000}$), with each term requiring a full forward pass through the model to evaluate the right-context probability. This is clearly intractable. SaMIA resolves this through importance sampling.

Variance Reduction Techniques

The importance sampling estimator in (IS) can have high variance when the right-context probability $w(v)$ varies dramatically across vocabulary tokens. This occurs when the right context is highly constraining: certain tokens at position $t$ make the continuation ${𝒙}_{t+1:L}$ plausible, while others render it nearly impossible. Several variance reduction strategies mitigate this issue.

Stratified sampling.

Partition the vocabulary $𝒱$ into strata based on the proposal probability $q(v)$: high-probability tokens, medium-probability tokens, and low-probability tokens. Draw a fixed fraction of samples from each stratum. This ensures that the estimator is not dominated by a few high-probability tokens and captures the contribution of the long tail.

Control variates.

Use the left-context log-probability $\log p_{𝜽}(x_t|{𝒙}_{1:t-1})$ as a control variate. Define the adjusted estimator (CV)$$\begin{array}{ccc}& {\hat{S}}_{\text{SaMIA}}^{\text{CV}}(𝒙)=S_{\text{SaMIA}}(𝒙)-\beta (\frac{1}{L}\sum _{t=1}^L\log p_{𝜽}(x_t|{𝒙}_{1:t-1})-{\mu }_{\text{AR}}),& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${\mu }_{\text{AR}}$ is the expected autoregressive log-probability (estimated from a held-out calibration set) and $\beta$ is chosen to minimise variance. Since the autoregressive and pseudo-likelihood scores are highly correlated, this control variate can reduce variance substantially.

Rao-Blackwellisation.

For positions where the top-$k$ tokens account for most of the proposal mass, compute the importance weights exactly for the top-$k$ tokens and use Monte Carlo only for the remaining tail: (RB)$$\begin{array}{ccc}& {𝔼}_q[w(v)]\approx \sum _{v\in \text{top-}k}q(v)w(v)+(1-\sum _{v\in \text{top-}k}q(v))\cdot \frac{1}{N^{\prime }}\sum _{i=1}^{N^{\prime }}w(v_i^{\prime }),& \text{<span class="tml-eqn"></span>}\end{array}$$ where $v_1^{\prime },\dots ,v_{N^{\prime }}^{\prime }\sim q(\cdot |v\notin \text{top-}k)$. This Rao-Blackwellised estimator is guaranteed to have variance no greater than the pure Monte Carlo estimator.

Why Pseudo-Likelihood Outperforms Joint Likelihood

A natural question arises: why go through the trouble of estimating pseudo-likelihood when the autoregressive factorisation already gives us the exact joint likelihood $\log p_{𝜽}(𝒙)={\sum }_t\log p_{𝜽}(x_t|{𝒙}_{1:t-1})$? The answer lies in the directionality of the membership signal.

The joint likelihood is dominated by the left-to-right conditional probabilities. If a token $x_t$ is highly predictable from its left context alone (e.g., a common function word following a strong collocate), its contribution to the joint likelihood is high regardless of membership. The membership signal in such tokens is weak. Conversely, a token that is predictable from its right context but surprising from the left carries membership information that the autoregressive likelihood entirely misses.

This result explains why SaMIA can outperform simple perplexity-based attacks on certain texts: the pseudo-likelihood captures bidirectional memorization effects that the autoregressive factorisation misses. The improvement is most pronounced for texts where the model has memorized specific long-range dependencies that manifest more strongly in the right-to-left direction.

Connection to Masked Language Models

The pseudo-likelihood framework illuminates a deep connection between autoregressive and masked language models in the context of membership inference. For a masked language model such as BERT or RoBERTa, the pseudo-likelihood ${\text{PLL}}_{𝜽}(𝒙)={\sum }_t\log p_{𝜽}(x_t|{𝒙}_{\setminus t})$ can be computed directly by masking each position in turn and reading off the softmax probability. No Monte Carlo sampling is needed.

This observation suggests a complementary attack strategy: if one has access to both an autoregressive model and a masked language model trained on the same (or similar) data, the MLM's pseudo-likelihood can serve as a free reference signal. The membership score becomes (Cross)$$\begin{array}{ccc}& S_{\text{cross}}(𝒙)={\text{PLL}}_{\text{MLM}}(𝒙)-{\text{PLL}}_{\text{AR}}(𝒙),& \text{<span class="tml-eqn"></span>}\end{array}$$ where the difference controls for text difficulty (both models assign high PLL to easy text) and isolates the memorization signal specific to the target model. This cross-model calibration is related in spirit to LiRA but uses architectural diversity rather than data diversity for calibration.

Sampling Process for Pseudo-Likelihood

Practical Considerations and Computational Budget

Deploying SaMIA in practice requires careful navigation of its computational demands. The method's $O(L\cdot N)$ forward-pass budget can be substantial, but several practical optimizations make it feasible.

Position sub-sampling.

Rather than evaluating all $L$ positions, randomly select a subset of $L^{\prime }\ll L$ positions. The sub-sampled SaMIA score ${\hat{S}}_{\text{SaMIA}}^{(L^{\prime })}=\frac{1}{L^{\prime }}{\sum }_{t\in 𝒮}\log {\hat{p}}_{𝜽}^{\text{IS}}(x_t|{𝒙}_{\setminus t})$, where $|𝒮|=L^{\prime }$, is an unbiased estimator of the full score with additional variance $\frac{L-L^{\prime }}{L^{\prime }(L-1)}\cdot {\sigma }_{\text{pos}}^2$, where ${\sigma }_{\text{pos}}^2$ is the variance of per-position pseudo-log-likelihoods.

Batch importance sampling.

Instead of drawing independent importance samples for each position, reuse the same set of samples across nearby positions. If positions $t$ and $t+1$ share the same left context up to position $t-1$, the importance weights $w(v)$ change only through the shift in right context, and partial computation can be shared.

Early stopping.

For positions where the importance sampling estimate converges quickly (i.e., the variance of $w(v)$ is low), terminate sampling early. Allocate the saved budget to positions with high variance, where the estimator benefits most from additional samples. This adaptive allocation can reduce total computation by 30–50% without degrading the membership score.

Exercises

Label-Only and Attention-Based Attacks

Every attack we have studied so far relies on one critical assumption: the adversary can observe the model's output probabilities. Whether computing perplexity, likelihood ratios, Min-K% statistics, or pseudo-likelihoods, each method requires access to the numerical values of $p_{𝜽}(x_t|{𝒙}_{1:t-1})$ for each token. But what if the model API returns only the generated text, with no logits, no probabilities, no confidence scores? This is increasingly the reality: commercial APIs like ChatGPT and Claude often return only the text response, hiding the underlying probability distribution behind a black curtain.

This section explores two complementary approaches to membership inference in these restricted settings. First, PETAL (Privacy Evaluation Through Analysis of Language) extracts membership signals from the text itself, without any numerical output beyond the words on the screen. Second, AttenMIA (tang2024attenmia) exploits a different information channel-the model's attention patterns-which are sometimes accessible through interpretability APIs or white-box access. Together, these methods demonstrate that membership information leaks through channels far more diverse than the obvious probability vector.

PETAL: Label-Only Membership Inference

The label-only setting is the most restrictive threat model for membership inference. The attacker queries the model with a prompt related to the target text $𝒙$ and observes only the generated response $𝒚$-a sequence of tokens with no attached probabilities. Can membership be inferred from the text alone?

The answer is yes, and the key insight is that language models generate text about training data differently than text about unfamiliar topics. When asked to continue or discuss a training member, the model produces text that is more specific, more detailed, and more consistent with the original than when discussing a non-member. These qualitative differences can be quantified.

PETAL constructs the score $S_{\text{LO}}$ through three complementary channels:

Textual similarity.

The attacker prompts the model to generate text on the same topic as $𝒙$ (e.g., “Write a paragraph about [topic extracted from $𝒙$]”) and measures the semantic similarity between the generation and $𝒙$ using an external embedding model. Training members yield generations with higher cosine similarity to the original because the model has internalised their specific content.

Factual consistency.

The attacker extracts factual claims from $𝒙$ and queries the model about each claim. For training members, the model's responses are more likely to confirm and elaborate on these claims, since the specific information has been absorbed during training. The consistency score is the fraction of claims that the model affirms.

Completion fidelity.

The attacker provides the first half of $𝒙$ as a prompt and measures the $n$-gram overlap (e.g., ROUGE-L or BLEU) between the model's continuation and the actual second half. For heavily memorized training members, the model may reproduce substantial portions of the original text.

This proposition formalises the intuition that label-only attacks pay an information-theoretic price for their restricted access. The gap $I_{\text{logit}}-I_{\text{text}}^{(Q)}$ quantifies how much membership signal is lost when the attacker observes only generated text. In practice, PETAL compensates for this loss by using diverse prompting strategies that extract different facets of the membership signal from each query.

AttenMIA: Attention Patterns as Membership Signals

While PETAL operates with text-only access, AttenMIA (tang2024attenmia) exploits a different information channel: the model's internal attention patterns. In transformer-based language models, the self-attention mechanism computes a matrix of attention weights ${𝐀}^{(h,l)}\in {ℝ}^{L\times L}$ for each head $h$ in each layer $l$, where $A_{ij}^{(h,l)}$ represents the attention weight from token $i$ to token $j$. These attention matrices encode the model's internal reasoning about relationships between tokens and, crucially, carry information about whether the model has seen the input during training.

The key observation behind AttenMIA is that training members produce attention patterns with lower entropy than non-members. When the model has been trained on a specific text, its attention mechanism has learned to focus on the relevant tokens with high confidence, producing sharp, concentrated attention distributions. For unfamiliar text, the model is less certain about which tokens are important, resulting in more diffuse attention patterns.

Layer selection.

Not all layers contribute equally to the membership signal. tang2024attenmia find that early layers (which handle local syntactic patterns) show minimal membership signal, while middle and late layers (which encode semantic and memorization-related features) show the strongest signal. A practical strategy selects the top-$k$ layers by their discriminative power on a small calibration set.

Head-level analysis.

Within each layer, different attention heads specialise in different functions. Some heads attend to syntactic structure, others to semantic content, and a few appear to specialise in “retrieval” patterns that are particularly informative for membership. AttenMIA can weight heads by their discriminative contribution: (Weighted)$$\begin{array}{ccc}& C_{\text{weighted}}(𝒙)=\sum _{l\in ℒ}\sum _{h=1}^H{w}_{h,l}\cdot \frac{1}{L}\sum _{i=1}^L(\log L-{\mathscr{H}}_i^{(h,l)}(𝒙)),& \text{<span class="tml-eqn"></span>}\end{array}$$ where the weights $w_{h,l}\ge 0$ are learned on a calibration set to maximise the separation between member and non-member attention profiles.

The AttenMIA Score Function

AttenMIA combines multiple attention-based statistics into a single membership score. Beyond the concentration score $C(𝒙)$ defined in Definition 53, tang2024attenmia identify two additional informative statistics.

Attention stability.

For a training member, the attention patterns are stable across different prompting contexts. Specifically, if the same text $𝒙$ is presented with different system prompts or prepended contexts, the attention matrices for members vary less than for non-members. The attention stability score is (Stability)$$\begin{array}{ccc}& \text{Stab}(𝒙)=1-\frac{1}{|𝒫|(|𝒫|-1)}\sum _{\begin{array}{c}p,q\in 𝒫\\ p\ne q\end{array}}{\Vert {\overline{𝐀}}^{(p)}(𝒙)-{\overline{𝐀}}^{(q)}(𝒙)\Vert }_F,& \text{<span class="tml-eqn"></span>}\end{array}$$ where $𝒫$ is a set of different prompting contexts, ${\overline{𝐀}}^{(p)}(𝒙)$ is the attention matrix averaged across heads and layers when $𝒙$ is presented with context $p$, and $\Vert \cdot {\Vert }_F$ is the Frobenius norm.

Attention self-similarity.

The model's attention to a training member exhibits higher self-similarity across layers. Define the layer-wise attention profile as the vector of per-position entropies for layer $l$: ${𝒉}^{(l)}=({\bar{\mathscr{H}}}_1^{(l)},\dots ,{\bar{\mathscr{H}}}_L^{(l)})\in {ℝ}^L$, where the bar denotes averaging over heads. The self-similarity score is (Selfsim)$$\begin{array}{ccc}& \text{SelfSim}(𝒙)=\frac{1}{|ℒ|(|ℒ|-1)}\sum _{\begin{array}{c}l,l^{\prime }\in ℒ\\ l\ne l^{\prime }\end{array}}\frac{{𝒉}^{(l)}\cdot {𝒉}^{(l^{\prime })}}{\Vert {𝒉}^{(l)}\Vert \cdot \Vert {𝒉}^{(l^{\prime })}\Vert },& \text{<span class="tml-eqn"></span>}\end{array}$$ which measures the average cosine similarity of entropy profiles across layers.

Visualising Attention Patterns

fig:mia:attenmia:patterns illustrates the characteristic difference between member and non-member attention patterns. For training members, the attention heads learn to focus on specific token relationships that the model has internalised during training. This creates a pattern with clear structure: strong diagonal elements (self-attention and local context) plus a few strong off-diagonal connections (long-range dependencies that the model has learned to exploit for this specific text). For non-members, the model has no pre-learned attention template and distributes attention more uniformly, resulting in a higher-entropy, less structured pattern.

When Different Modalities Excel

We have now surveyed a rich landscape of LLM membership inference attacks spanning Perplexity and Loss-Based Attacks through the present section. Each attack exploits a different information channel, and no single method dominates in all scenarios.

Attack	Access	Cost	Ref. Models?	Typical AUC	Reference
Perplexity	L/T$k$	$1$	No	0.55–0.65	[13]
LiRA	L/T$k$	$1+2K$ train	Yes ($2K$)	0.70–0.85	[4]
Min-K% Prob	L/T$k$	$1$	No	0.60–0.75	[14]
SPV-MIA	L/T$k$	$K+1$	No (self-ref)	0.65–0.78	[17]
SaMIA	L/T$k$	$L\cdot N$	No	0.62–0.76	[18]
AttenMIA	A	$1$	No	0.58–0.72	[19]
PETAL	T	$Q$ queries	No	0.52–0.65	Label-only

tab:mia:llm:comparison summarises the key characteristics of each attack. Several patterns emerge:

• More information yields better attacks. Full logit access enables the strongest attacks (LiRA), while text-only access limits the attacker to weaker signals.

• Calibration matters more than raw information. SPV-MIA with self-calibration substantially outperforms raw perplexity despite using the same logit access, because it controls for the distributional component of the loss.

• Reference-free methods can approach reference-based performance. Min-K% Prob and SPV-MIA achieve AUC within 5–10 percentage points of LiRA, at a fraction of the cost.

• Attention provides an orthogonal signal. AttenMIA can be combined with probability-based attacks for improved performance, since the attention and probability channels carry partially independent membership information.

Exercises

Attacks on Alignment and Preference Data

The story of membership inference in language models has, until now, focused on a single chapter of the model's life: pretraining. We have asked whether a specific text was among the billions of tokens used to train the base model from scratch. But modern language models lead a second life after pretraining. They undergo alignment: a process of fine-tuning that reshapes the model's behaviour to be helpful, harmless, and honest. This alignment typically involves reinforcement learning from human feedback (RLHF) or direct preference optimisation (DPO), both of which require a new kind of data-not raw text, but preference pairs: tuples of the form (prompt, chosen response, rejected response), where human annotators have judged which response is better.

This preference data represents a fresh attack surface. shetty2024premia introduced PREMIA (Preference Data Membership Inference Attack), demonstrating that the alignment process creates distinctive membership fingerprints that are, in some respects, easier to detect than pretraining membership. The reason is subtle but important: while pretraining distributes its gradient updates across billions of tokens, alignment concentrates its updates on a much smaller dataset of carefully curated preference pairs. Each pair leaves a deeper imprint on the model's parameters, making membership inference correspondingly easier.

This section develops the mathematical framework for preference-based membership inference, analyses why alignment creates new privacy vulnerabilities, and examines the broader implications for the safety of aligned models.

The Alignment Process and Its Data

To understand PREMIA, we must first understand the data that alignment consumes. The two dominant alignment paradigms, RLHF and DPO, both rely on human preference data, but use it in fundamentally different ways.

RLHF.

In reinforcement learning from human feedback, the alignment process has two stages. First, a reward model $r_{\varphi }(y|x)$ is trained on preference pairs $(x,y_w,y_l)$ to predict which response a human would prefer, where $y_w$ is the chosen (“winning”) response and $y_l$ is the rejected (“losing”) response. The reward model is trained with the Bradley-Terry loss: (BT)$$\begin{array}{ccc}& {ℒ}_{\text{BT}}(\varphi )=-{𝔼}_{(x,y_w,y_l)}\left[\log \sigma \right(r_{\varphi }(y_w|x)-r_{\varphi }(y_l|x)\left)\right],& \text{<span class="tml-eqn"></span>}\end{array}$$ where $\sigma$ is the sigmoid function. Second, the language model is fine-tuned using proximal policy optimisation (PPO) to maximise the expected reward while staying close to the pretrained model (measured by a KL divergence penalty).

DPO.

Direct preference optimisation [30] bypasses the reward model entirely. The DPO loss directly optimises the language model on preference pairs: (DPO)$$\begin{array}{ccc}& {ℒ}_{\text{DPO}}(𝜽)=-{𝔼}_{(x,y_w,y_l)}\left[\log \sigma (\beta \log \frac{p_{𝜽}(y_w|x)}{p_{\text{ref}}(y_w|x)}-\beta \log \frac{p_{𝜽}(y_l|x)}{p_{\text{ref}}(y_l|x)})\right],& \text{<span class="tml-eqn"></span>}\end{array}$$ where $p_{\text{ref}}$ is the reference (pretrained) model and $\beta >0$ controls the strength of the KL constraint. The DPO objective directly encodes preference information into the model's probability assignments, creating a clear channel for membership leakage.

The crucial difference from pretraining MIA is the structure of the data. A preference triple contains three informative components: the prompt $x$, the chosen response $y_w$, and the rejected response $y_l$. The alignment process modifies the model to increase $p_{𝜽}(y_w|x)$ and decrease $p_{𝜽}(y_l|x)$ relative to the reference model. This bidirectional push creates a distinctive fingerprint.

DPO Membership Leakage

The DPO objective in (DPO) provides a transparent window into why alignment leaks membership information. Consider what happens to a preference pair $(x,y_w,y_l)$ that is in the training set. The DPO gradient pushes the model to increase the log-probability ratio (Ratio)$$\begin{array}{ccc}& R_{𝜽}(x,y_w,y_l)=\log \frac{p_{𝜽}(y_w|x)}{p_{\text{ref}}(y_w|x)}-\log \frac{p_{𝜽}(y_l|x)}{p_{\text{ref}}(y_l|x)},& \text{<span class="tml-eqn"></span>}\end{array}$$ which we call the preference margin. For training members, this margin is pushed to be large and positive; for non-members, the margin remains near zero (since the model has not been optimised on them).

The PREMIA Attack Framework

Armed with the membership signal from Theorem 18, we can construct the PREMIA attack. The attack exploits both the preference margin $R_{𝜽}$ and several auxiliary signals that capture different facets of the alignment fingerprint.

Note that the preference margin $R_{𝜽}={\mathrm{\Delta }}_w+{\mathrm{\Delta }}_l$, so the three-term score in (Score) allows the attack to weight the chosen-response and rejected-response signals differently. This flexibility is important because the strength of each signal depends on the alignment algorithm and hyperparameters.

The PREMIA Pipeline

The PREMIA attack, illustrated in fig:mia:premia:pipeline, requires access to two models: the aligned model $f_{𝜽}$ and its pretrained reference $f_{\text{ref}}$. In many practical settings, both models are available: the reference model is often the base model (e.g., LLaMA-2 base), while the aligned model is the instruction-tuned variant (e.g., LLaMA-2-Chat). The attack proceeds as follows:

1. Query both models. For the candidate triple $(x,y_w,y_l)$, compute $\log p_{𝜽}(y_w|x)$, $\log p_{𝜽}(y_l|x)$, $\log p_{\text{ref}}(y_w|x)$, and $\log p_{\text{ref}}(y_l|x)$. This requires four forward passes total (two per model).

2. Compute signals. Calculate the preference margin $R_{𝜽}$, chosen uplift ${\mathrm{\Delta }}_w$, and rejected suppression ${\mathrm{\Delta }}_l$.

3. Score and threshold. Combine the signals using (Score) with pre-calibrated weights, and classify as member if $S_{\text{PREMIA}}>\tau$.

Why Alignment Creates New Vulnerabilities

The alignment phase creates privacy vulnerabilities that are, in several respects, more severe than those of pretraining. Three factors contribute to this heightened risk.

Dataset size disparity.

Pretraining corpora contain trillions of tokens, diluting the influence of any single document. Alignment datasets typically contain tens of thousands to hundreds of thousands of preference pairs. By Theorem 18(c), the membership gap $\mathrm{\Delta }R\propto \eta T/n$ grows inversely with dataset size $n$. For $n=\mathrm{50,000}$ alignment pairs versus $n={10}^{12}$ pretraining tokens (with effective documents in the millions), the per-example influence during alignment is orders of magnitude larger.

Structured supervision.

Preference pairs encode explicit human judgements about response quality. This structured supervision creates a sharper gradient signal than the generic next-token prediction of pretraining. The model must learn to distinguish between the chosen and rejected responses for each specific prompt, creating prompt-specific memorization that is easier to detect.

Bidirectional modification.

Unlike pretraining, which only pushes the model to increase the probability of the training text, alignment creates a push-pull dynamic: the chosen response is upweighted and the rejected response is downweighted. This bidirectional modification doubles the membership signal, since the attacker can detect both the uplift on $y_w$ and the suppression of $y_l$.

Distinguishing RLHF from DPO Fingerprints

An intriguing extension of PREMIA arises when the attacker does not know which alignment algorithm was used. RLHF and DPO leave qualitatively different membership fingerprints, and understanding these differences is essential for robust attacks.

DPO fingerprint.

DPO directly modifies the language model's probabilities to satisfy the preference constraint. The resulting fingerprint is sharp and localised: the preference margin $R_{𝜽}$ increases primarily for the specific token sequences in $(y_w,y_l)$, with limited spillover to semantically similar but textually distinct responses. This makes DPO membership easier to detect but also easier to confuse with domain-level effects.

RLHF fingerprint.

In RLHF, the preference data trains a reward model, and the reward model then guides PPO fine-tuning. The membership fingerprint is more diffuse: the reward model learns a generalised preference function that affects the language model's behaviour across many prompts and responses, not just the specific training triples. This indirection makes RLHF membership harder to detect for individual preference pairs but creates a broader “distributional shift” that can be detected through aggregate statistics.

The practical implication is that PREMIA is most effective against DPO-aligned models, where the direct optimisation creates a clear, per-pair fingerprint. For RLHF-aligned models, the attacker may need to shift from individual-pair detection to a dataset-level test that aggregates weak signals across many candidate pairs.

Broader Implications for Alignment Safety

PREMIA raises fundamental questions about the tension between alignment and privacy:

Privacy of annotators.

Human annotators who provide preference judgements may have their personal biases and values encoded in the alignment data. If PREMIA can identify which preference pairs were used, it can potentially reconstruct aspects of individual annotators' judgement patterns, raising concerns about annotator privacy.

Alignment data auditing.

On the positive side, PREMIA-style attacks can serve as an auditing tool. If a model provider claims to have aligned their model on ethically sourced preference data, a regulator could use PREMIA to verify whether specific (known) datasets were used. This “alignment auditing” mirrors the copyright auditing applications of pretraining MIA but operates in the more structured preference domain.

Differential privacy for alignment.

The vulnerability exposed by PREMIA motivates the application of differential privacy (DP) to the alignment process. DP-SGD can be applied during DPO training to bound the influence of any single preference pair. However, the small dataset sizes typical of alignment make DP particularly costly in terms of utility: the noise required for meaningful privacy guarantees ($ϵ\le 10$) can substantially degrade alignment quality.

Defence through preference aggregation.

An alternative defence aggregates multiple annotators' preferences before training, so that no single annotator's judgement is decisive. If each preference pair is evaluated by $k$ annotators and the majority vote determines $(y_w,y_l)$, the model's gradient is influenced by the aggregate preference rather than any individual's judgement. This reduces the PREMIA signal for individual annotator-level triples while preserving alignment quality, at the cost of requiring $k$ times more annotation effort.

Model merging and alignment recycling.

Recent approaches to alignment involve merging multiple independently aligned models (e.g., via weight averaging or task arithmetic). These merged models inherit partial membership fingerprints from each constituent alignment dataset, creating a more complex membership signal that may be harder to attribute to any single preference pair. The theoretical analysis of PREMIA under model merging remains an open problem, as the interaction between merged fingerprints is not well understood.

Exercises

Attacks on Vision-Language Models

Vision-language models (VLMs) like GPT-4V, LLaVA, and Gemini can see and reason about images while generating text. They are trained on vast multimodal datasets that include private photographs, medical scans, and copyrighted artwork. The question of membership inference becomes doubly challenging and doubly important: can an attacker determine whether a specific image-text pair was used during training? The answer, as we shall discover, is a resounding yes, and the multimodal nature of these models actually creates more attack surfaces than their unimodal counterparts. Where a language model offers a single channel of information leakage (token probabilities), a VLM offers three: the visual encoder, the language decoder, and the cross-modal bridge that connects them. Each channel tells a different part of the membership story, and the cleverest attacks listen to all three simultaneously.

The stakes are substantial. VLMs are trained on datasets scraped from the internet that contain personal photographs, medical imagery shared in clinical forums, copyrighted illustrations, and private documents that were inadvertently indexed. A membership inference attack against a VLM could reveal that a specific patient's retinal scan was used to train a diagnostic model, that a photographer's portfolio was incorporated without consent, or that confidential corporate documents were included in a multimodal training corpus. As VLMs become increasingly central to commercial products (from image search to medical diagnostics), the privacy implications of their training data grow ever more consequential.

VLM Architectures and Attack Surfaces

Before mounting an attack, the detective must understand the building. Modern VLMs share a common architectural template, though the details vary across model families. Understanding this architecture reveals where membership information can leak.

A VLM combines three principal components. First, a vision encoder, typically a Vision Transformer (ViT) pre-trained with contrastive learning (e.g., CLIP), converts an input image into a sequence of visual tokens or patch embeddings. Second, a projection layer (sometimes called a connector or adapter) maps the visual token representations into the embedding space of the language model. This projection can be as simple as a linear layer or as complex as a cross-attention module with learnable queries (as in the Q-Former architecture of BLIP-2 and InstructBLIP). Third, a language model (typically a large autoregressive transformer such as LLaMA or Vicuna) processes the projected visual tokens alongside text tokens to generate responses.

The three-surface structure of VLMs creates both opportunities and challenges for the attacker. On one hand, more attack surfaces mean more information to exploit. On the other hand, it is not immediately obvious how to optimally combine signals from different modalities. The visual encoder operates in a continuous feature space, while the language model operates over a discrete token vocabulary; the projection layer bridges these fundamentally different representations. An effective multimodal membership inference attack must fuse information across these heterogeneous channels.

MaxRenyi-K%: Extending Outlier Detection to Multimodal

The Min-K% attack, which we studied in earlier sections for text-only models, exploits the fact that training data produces systematically lower per-token loss. Extending this idea to multimodal models requires two innovations: (1) defining per-token confidence for both visual and textual tokens, and (2) determining how to weight and aggregate these heterogeneous scores. The MaxRenyi-K% method [20] addresses both challenges by introducing a principled weighting scheme based on Rényi divergence.

The Rényi divergence provides a family of divergence measures parameterised by $\alpha$, offering a smooth interpolation between the KL divergence (which emphasises the average-case difference between distributions) and the max-divergence (which emphasises the worst-case difference). For membership inference, the choice of $\alpha$ controls the balance between sensitivity to broadly distributed memorisation signals and sensitivity to sharply localised memorisation (individual tokens with extreme confidence).

The intuition behind this construction is elegant. Training data produces tokens with anomalously high confidence (as we know from loss-based attacks), but not all tokens are equally informative. Function words like “the” and “is” receive high confidence regardless of membership, contributing noise rather than signal. The Rényi divergence weight $w_k$ measures how much the model's prediction at position $k$ deviates from the reference distribution; tokens where the model is much more confident than the reference (indicating memorisation) receive large weights, while tokens where the model and reference agree (indicating generic language patterns) receive small weights. Selecting only the top $k\%$ of weighted tokens further concentrates the score on the most informative positions.

Cross-Modal Membership Signals

The most intriguing finding from VLM membership inference research is that the relationship between an image and its paired text carries stronger membership information than either modality alone. This phenomenon, which we call cross-modal amplification, arises because VLMs are explicitly trained to associate specific images with specific textual descriptions.

Visual Tokens vs. Text Tokens

Not all tokens are created equal in the membership inference game. A natural question is whether visual tokens or text tokens carry more membership information, and the answer turns out to depend on the nature of the training data.

The intuition is straightforward. When the distinctive information in a training pair resides primarily in the image (a unique photograph, a specific medical scan), the visual tokens encode that distinctiveness more directly. When the distinctive information resides in the text (a detailed medical report, a long caption with specific names and dates), the text tokens carry the membership fingerprint. In both cases, combining the two modalities recovers information that either alone would miss, because the cross-modal association provides complementary evidence.

Several patterns emerge from these results. First, InstructBLIP consistently shows slightly higher vulnerability than the other models, likely because its Q-Former architecture creates a richer cross-modal representation that carries more membership information. Second, medical data shows the largest text-to-visual gap, reflecting the relative uniformity of medical images compared to the specificity of medical reports. Third, document data shows the largest text dominance, because the visual representation of a document page carries less unique information than the actual text content.

Token-Level Membership Detection

Sequence-level membership inference asks: “was this entire passage in the training data?” But a finer-grained question is often more relevant: “which specific tokens in this text were memorized?” Token-level detection enables applications like copyright attribution, where we need to identify exactly which parts of a generated text were copied from training data. It is the difference between a detective knowing that evidence exists somewhere in a room and knowing precisely which drawer contains the incriminating document.

This granularity matters enormously in practice. A model might generate a paragraph that is 90% original composition but contains a 10% verbatim quotation from a copyrighted source. Sequence-level MIA would flag the entire paragraph as “possibly memorised,” which is unhelpful for legal proceedings or content filtering. Token-level detection, by contrast, can highlight the exact span that was reproduced from training data, enabling precise attribution, targeted redaction, and quantitative measurement of the degree of memorisation.

From Sequence-Level to Token-Level

The transition from sequence-level to token-level membership detection represents a fundamental shift in granularity, analogous to moving from a binary blood test (“infection present or absent”) to a microscope image showing which specific cells are infected. This direction builds on the detection framework of [14], which demonstrated that per-token statistics carry rich membership information, and connects to the broader memorisation analysis of [4].

Statistical Framework for Token-Level Detection

We now develop the mathematical framework for token-level membership detection. The key idea is to compare the model's confidence at each token position against a reference expectation: memorised tokens show anomalously high confidence (low loss), while non-memorised tokens show confidence consistent with the reference.

The score $m_i$ has a natural interpretation. When the model assigns very high probability to the actual token $x_i$ (indicating memorisation), the loss ${\ell }_i$ is near zero, so $m_i\approx 1$. When the model's confidence is typical of what it would assign to any plausible token at that position (indicating no memorisation), the loss matches the reference expectation, giving $m_i\approx 0$. Negative values of $m_i$ indicate tokens where the model is less confident than expected, which can occur for tokens that are surprising in context regardless of membership status.

Practical Algorithms and Threshold Selection

Translating the theoretical framework into a practical algorithm requires addressing two challenges: efficiently estimating the reference loss ${𝔼}_{{\hat{x}}_i}[{\ell }_i(\cdot )]$, and selecting appropriate per-token thresholds.

The reference loss estimation (Step 2) deserves particular attention. Two strategies are available. The first uses a reference model $f_{{𝜽}_{\text{ref}}}$ trained on a separate dataset (as in the sequence-level reference model approach of Statistical Framework for Token-Level Detection). The reference model's loss at each position provides an estimate of what a non-memorising model would assign. The second strategy uses dropout sampling: by running $K$ forward passes through the target model with different dropout masks, the variance in per-token loss estimates the model's uncertainty, and the mean provides a smoothed reference. The dropout approach is attractive because it requires only the target model (no reference), but it is noisier and less reliable for heavily memorised tokens.

Applications: Copyright Detection and Data Attribution

Token-level membership detection finds its most compelling applications in the domains of copyright enforcement and data provenance, where the precise identification of memorised content is legally and practically essential.

Attacks on Fine-Tuned LLMs

Fine-tuning is the alchemist's art of modern AI: take a general-purpose model and transform it into a specialist. But this specialization comes with a privacy cost. When you fine-tune on a small, domain-specific dataset, the model memorises that data with remarkable fidelity. The smaller the dataset, the greater the vulnerability. If membership inference against pre-trained LLMs is like searching for a needle in a haystack, inference against fine-tuned models is like searching for a red-hot needle in a small pile of straw: the signal is intense and the search space is limited.

The practical implications are profound. Fine-tuning is the dominant paradigm for adapting large language models to specific tasks: medical diagnosis, legal document analysis, customer service, code generation for specific codebases, and countless other applications. In each case, the fine-tuning data is often proprietary, sensitive, or both. A hospital fine-tunes a model on clinical notes. A law firm fine-tunes on confidential case files. A company fine-tunes on internal communications. If membership inference can detect individual samples in these fine-tuning datasets, the privacy consequences are immediate and severe.

Why Fine-Tuning Amplifies Membership Leakage

The fundamental reason fine-tuning amplifies membership leakage is arithmetic: a model that was pre-trained on trillions of tokens and fine-tuned on thousands of examples allocates a disproportionate share of its capacity to the fine-tuning data. Each fine-tuning example occupies, in a precise sense, a million times more of the model's “memory” than each pre-training example.

LoRA, QLoRA, and Parameter-Efficient Fine-Tuning

A natural question is whether parameter-efficient fine-tuning (PEFT) methods, which update only a small fraction of the model's parameters, provide privacy benefits. Low-Rank Adaptation (LoRA) is the most popular PEFT method; it freezes the pre-trained weights and adds trainable low-rank decompositions to selected weight matrices: ${𝐖}^{\prime }={𝐖}_0+𝐀𝐁$, where $𝐀\in {\mathrm{ℝ}}^{d\times r}$ and $𝐁\in {\mathrm{ℝ}}^{r\times d}$ with $r\ll d$.

Few-Shot Fine-Tuning: Extreme Vulnerability

The privacy landscape becomes particularly dire in the few-shot regime, where models are fine-tuned on fewer than 100 examples. Few-shot fine-tuning is increasingly popular for task-specific adaptation (instruction tuning with a handful of examples, domain adaptation with limited data), but it creates the most extreme membership vulnerability.

The intuition is straightforward. When a model with billions of parameters is fine-tuned on, say, 50 examples, it must drastically alter its behaviour to accommodate those specific examples. The per-sample influence is enormous: each example accounts for 2% of the fine-tuning signal. The model essentially memorises the fine-tuning set in its entirety, creating a gap between member and non-member loss that is trivially detectable by any reasonable membership inference method.

Pre-Training vs. Fine-Tuning Data Membership

The contrast between attacking pre-training data and fine-tuning data illuminates the fundamental relationship between dataset size and privacy vulnerability.

The table reveals two important trends. First, the pre-training AUC increases only slowly with model size (from $0.57$ to $0.64$), reflecting the fact that larger models are trained on proportionally more data, approximately balancing the increased capacity. Second, the fine-tuning AUC increases more rapidly (from $0.79$ to $0.95$), because larger models have more capacity available for memorising the fixed-size fine-tuning set. The gap between pre-training and fine-tuning AUC widens from $0.22$ for the smallest model to $0.31$ for the largest, confirming the amplification effect of Theorem 21.

Machine Unlearning Verification via MIA

The European Union's General Data Protection Regulation (GDPR) enshrines the “right to be forgotten”: individuals can demand that organisations delete their data. But for AI models, deletion is not straightforward. You cannot simply remove a data point from a trained model's parameters; the influence permeates the entire network like dye dissolved in water. Machine unlearning attempts to solve this problem, and membership inference attacks are the natural tool to verify whether unlearning was successful. The relationship between these two concepts is one of the most elegant dualities in modern machine learning: MIA is to unlearning what a litmus test is to chemistry, the definitive verification of whether the desired transformation has occurred.

The stakes extend beyond abstract privacy principles. Under the GDPR, organisations face fines of up to 4% of global annual revenue for non-compliance. The California Consumer Privacy Act (CCPA) grants similar deletion rights. As AI systems become more prevalent in sectors that handle sensitive data (healthcare, finance, education), the ability to verifiably “forget” specific training data becomes a legal and operational necessity, not merely a theoretical curiosity.

The Right to Be Forgotten and Machine Unlearning

This definition mirrors the definition of differential privacy, and the parallel is not coincidental. Both concepts formalise the idea that a dataset perturbation (adding a point for DP, removing a point for unlearning) should be undetectable to any observer. The parameter $\epsilon$ quantifies the residual detectability: an $\epsilon$ close to zero means the unlearned model is nearly indistinguishable from one that never saw the forgotten data.

MIA as the Natural Verification Tool

The duality between unlearning and membership inference is not merely philosophical; it can be made precise. If a model has truly forgotten a data point, then no membership inference attack should be able to detect that the point was ever in the training set. Conversely, if any MIA succeeds, the data point has not been fully forgotten.

This theorem provides the theoretical foundation for using MIA as an unlearning verification tool. In practice, we cannot test against all possible attacks, so we use a battery of the strongest known attacks as a practical approximation.

The TOFU Benchmark

Evaluating machine unlearning requires standardised benchmarks with known ground truth. The TOFU (Task of Fictitious Unlearning) benchmark [21] addresses this need by constructing a controlled environment using fictitious data that avoids the ethical complications of experimenting with real personal information.

The TOFU benchmark elegantly captures the three-way tension in machine unlearning. An unlearning method must simultaneously (a) erase the influence of forgotten data, (b) preserve knowledge of retained data, and (c) maintain general capabilities. A trivial “unlearning” method that simply reinitialises the model to random weights would achieve perfect forget quality but catastrophic retain quality and model utility. Conversely, doing nothing (returning the original model) preserves retain quality and utility but fails the forget quality check. The challenge is to find methods that satisfy all three criteria simultaneously.

The table paints a sobering picture. No approximate method matches the gold standard of exact retraining across all three metrics simultaneously. Gradient ascent with 200 steps comes closest on forget quality ($0.53$) but sacrifices substantial retain accuracy ($0.74$ vs. $0.89$). Influence functions preserve retain accuracy well ($0.88$) but leave a significant residual membership signal ($0.57$). The KL minimisation approach, which regularises the unlearned model to match a reference model's predictions on retained data while diverging on forgotten data, offers a reasonable compromise ($0.55$ forget AUC, $0.85$ retain accuracy), but is still detectably imperfect by MIA standards.

Exploring the Limits of Strong Attacks

The results in the previous subsection used relatively simple MIA methods (loss-based, Min-K%) for unlearning verification. A critical question is: do current approximate unlearning methods withstand scrutiny from stronger attacks? Recent work [22] answers this question with a troubling “no.”

The interplay between MIA and unlearning generates a rich landscape of open problems. What is the minimum computational cost of $(\epsilon ,\delta )$-complete unlearning? Can we construct unlearning methods that are certifiably robust against all polynomial-time MIA attacks? How do we handle the case where the forget set is large (e.g., an entire demographic group requests deletion)? These questions sit at the intersection of privacy, learning theory, and computational complexity, and their resolution will shape the future of trustworthy AI.

This proposition reveals an elegant connection: differential privacy is a form of pre-emptive unlearning. A DP-trained model has already limited the influence of any individual data point to at most ${\epsilon }_{\text{DP}}$, so “forgetting” that point requires no additional effort. The trade-off is that achieving strong DP guarantees during training imposes a significant utility cost (typically 5–15% accuracy degradation for ${\epsilon }_{\text{DP}}\le 8$), which must be paid upfront regardless of whether any deletion requests materialise.

Differential Privacy for LLMs

Applying differential privacy to large language models is like fitting armor to a giant: the protection is invaluable, but the weight is staggering. The sheer scale of modern LLMs, with billions of parameters trained on trillions of tokens, makes DP-SGD orders of magnitude more expensive than standard training. Yet the privacy guarantees are irreplaceable. In this section, we confront the engineering and theoretical challenges of making differential privacy work at the scale of modern language models, and we discover that clever algorithmic design can make the seemingly impossible merely difficult.

DP-SGD at LLM Scale

The core obstacle is straightforward to state but formidable to overcome. DP-SGD requires computing and clipping per-sample gradients before adding calibrated noise. For a model with $|𝜽|$ parameters and a minibatch of $B$ sequences, the naive approach materializes all $B$ individual gradient vectors, consuming $O(B\cdot |𝜽|)$ memory. When $|𝜽|$ is measured in billions, even a single gradient vector strains modern GPU memory; storing $B$ of them simultaneously is infeasible.

The memory bottleneck is severe. A model with $|𝜽|=7\times {10}^9$ parameters requires approximately 28GB to store a single gradient vector in FP32. With a batch size of $B=64$, naive per-sample gradient computation would require 1.8TB of GPU memory, far exceeding the capacity of any single accelerator. Even with gradient checkpointing and other memory optimization techniques, the $O(B\cdot |𝜽|)$ scaling is fundamentally prohibitive.

The memory bottleneck of per-sample gradient computation motivated the development of ghost clipping, a technique that computes per-sample gradient norms without ever materializing the full per-sample gradient vectors [23].

Privacy Accounting for Next-Token Prediction

A subtle complication arises from the autoregressive structure of language models. Unlike image classifiers, where each training sample contributes a single scalar loss, a language model computes $L$ loss terms per sequence (one per token position), and the gradient is the sum of these $L$ contributions. This structure has important implications for privacy accounting, because the gradient norm of a sequence depends on $L$, and longer sequences contribute larger gradients that require more aggressive clipping.

DP Fine-Tuning vs. DP Pre-Training

The calculation in Example 42 paints an optimistic picture for DP pre-training on massive datasets. In practice, however, the computational overhead of ghost clipping (requiring multiple backward passes) and the degraded convergence from noisy gradients make DP pre-training prohibitively expensive for the largest models. This has led the community to focus on a more practical alternative: DP fine-tuning.

Scaling Laws under Differential Privacy

The celebrated Chinchilla scaling laws describe how model performance improves as a function of model size $N$ and dataset size $D$: roughly, ${\text{PPL}}_0(N,D)\approx a{N}^{-\alpha }+b{D}^{-\beta }+c_0$ for architecture-dependent constants $a,b,c_0$ and exponents $\alpha \approx 0.34$, $\beta \approx 0.28$. Under differential privacy, these scaling laws must be modified to account for the additional noise injected during training. The modification introduces a third axis, the privacy budget $\epsilon$, that fundamentally alters the compute-optimal tradeoff between model size and data size.

Token-Level Defenses and Watermarking

If the attack operates at the token level, perhaps the defense should too. The previous sections revealed that membership inference against LLMs exploits the fine-grained signal carried by individual token probabilities: memorized tokens have anomalously low loss, and this signal persists even when the overall sequence loss is unremarkable. A natural counterstrategy is to intervene at the same granularity, treating tokens differently based on their memorization risk. The DuoLearn framework [24] embodies this philosophy, training the model with a dual-purpose objective that simultaneously learns useful patterns and unlearns privacy-risky memorization.

DuoLearn: Dual-Purpose Token-Level Defense

Output Perturbation and Confidence Masking

Not every organization can afford to retrain or fine-tune its models with privacy-aware objectives. The computational cost of retraining a billion-parameter model with DP-SGD can be prohibitive, and many organizations deploy third-party models over which they have no training control. For deployed models, post-hoc defenses that modify the model's output without touching its parameters offer a pragmatic alternative. These defenses operate at inference time, introducing controlled perturbations to the model's output distribution that reduce the membership signal without requiring access to the training pipeline.

Watermarking for Provenance

Watermarking is a defense mechanism that serves a different, but complementary, purpose to membership inference defenses. While MIA defenses aim to prevent an attacker from determining whether specific data was used for training, watermarking aims to establish provenance: proving that a given text was generated by a specific model. The distinction is important: MIA concerns the input side of the model (was this data used for training?), while watermarking concerns the output side (was this text produced by this model?). Both are essential components of a comprehensive model governance strategy, but they address orthogonal threats and require different technical approaches.

Defense Evaluation Methodology

Agentic AI Privacy Challenges

We are entering the age of AI agents: systems that autonomously browse the web, send emails, execute code, and manage databases on behalf of users. These agents have access to vast amounts of private data, and they interact with external systems in ways that create novel privacy attack surfaces. The membership inference threat extends beyond the model itself to the entire agentic pipeline. Where previous sections examined the model as an isolated oracle, we now confront the reality that modern AI systems are embedded in complex environments, with tools, APIs, and other agents forming an interconnected web of information flow. Each connection is a potential channel through which private data can leak.

The shift from model-as-oracle to model-as-agent represents a qualitative change in the threat landscape. An oracle processes queries and returns answers; the privacy risk is limited to the information conveyed in those answers. An agent, by contrast, takes actions in the world: it can read files, send messages, make API calls, and modify databases. Each action both consumes and produces information, creating a rich web of data flows that an attacker can exploit. The privacy analysis must therefore consider not only what the model “knows” but what it does with that knowledge.

Data Exfiltration in Agentic Pipelines

Prompt Injection and Indirect Membership Inference

Shadow AI: Unauthorized Model Deployment

Multi-Agent System Vulnerabilities

Open Problems and Future Directions

As we conclude this chapter, we look ahead to the most pressing unsolved problems. The field of membership inference and model privacy is young but rapidly evolving. Each answer raises new questions, and the stakes grow higher as AI systems become more powerful and more deeply integrated into society. The problems below range from fundamental theoretical questions to urgent practical challenges, and we hope they will inspire the next generation of privacy researchers.

The rapid pace of advancement in generative AI has created a widening gap between the capabilities of deployed systems and the maturity of privacy defenses. Models that were considered large five years ago are now routine; techniques developed for models with millions of parameters must be scaled to models with hundreds of billions. Simultaneously, the deployment context has shifted from controlled research environments to consumer-facing products used by billions of people, raising the stakes of any privacy failure. The problems we highlight reflect this reality: they are not merely academic curiosities but pressing challenges with real-world consequences.

Fundamental Tradeoffs

MIA Beyond Binary Classification

Regulatory Implications

Chapter Synthesis and Connections

We have traveled a long arc through this chapter, from the hypothesis-testing foundations that formalize membership inference as a statistical decision problem, through the specific manifestations of MIA in diffusion models (exploiting denoising loss distributions, intermediate reconstructions, gradient signals, and frequency-domain artifacts), to the LLM-specific attacks that leverage token-level probabilities and the structural properties of autoregressive generation. We examined vision-language models, where the multimodal attack surface creates new vulnerabilities, and we studied the landscape of defenses: differential privacy (both model-level and output-level), machine unlearning (exact and approximate), and token-level methods like DuoLearn. Finally, we confronted the emerging challenges of agentic AI, where the attack surface extends far beyond the model itself to encompass entire systems of tools, APIs, and autonomous agents.

Several unifying themes connect these diverse topics. First, the overfitting hypothesis: membership inference succeeds precisely to the extent that the model treats training data differently from unseen data, and this differential treatment is the fundamental signal exploited by every attack in this chapter, whether it manifests as lower denoising loss, higher token probabilities, smaller gradient norms, or distinctive frequency-domain signatures. Second, the privacy-utility tension: every defense we examined involves some cost in model quality, whether through DP noise, selective unlearning, output perturbation, or access restriction. The quest for defenses that are “free” (improving both privacy and utility) is the holy grail, and DuoLearn's surprising demonstration that selective unlearning can improve generalization offers a tantalizing hint that such defenses may exist more broadly. Third, the expanding attack surface: as AI systems evolve from isolated models to integrated agents, the privacy threat evolves in tandem, requiring defenses that span the entire system architecture rather than addressing the model alone.

The detective story we set out to tell at the beginning of this chapter is, in truth, a story without a final chapter. The attacker develops new probes; the defender deploys new shields; the regulator demands new guarantees; and the technology itself evolves, creating new vulnerabilities faster than old ones can be patched. What remains constant is the underlying mathematics: hypothesis testing, information theory, and differential privacy provide the enduring framework within which this arms race unfolds. The reader who has mastered these foundations is equipped to engage with whatever developments the field produces next.

Exercises

The following exercises range from foundational derivations that reinforce the theoretical framework of this chapter to challenging open-ended problems that connect to active research frontiers. Several exercises require multi-step reasoning and build from simpler to more demanding parts; we encourage readers to attempt each part before moving on, as later parts often depend on insights from earlier ones. Exercises marked with a “Bonus” part offer opportunities for deeper exploration suitable for advanced students and researchers.