33 Generative Models for Structured Data

Introduction and Motivation

Consider the following thought experiment. You are a hospital administrator in charge of a research programme on diabetic retinopathy. Your institution holds thousands of patient records: ages, blood glucose levels, comorbidity codes, imaging results, treatment histories, and outcomes. A team of machine learning researchers asks you for access to these records. They want to train a classifier that predicts which patients are at highest risk of vision loss. The model could save sight. The data could save lives.

You say no.

The reasons are familiar to anyone who has worked with sensitive data. Patient privacy regulations (HIPAA in the United States, GDPR in Europe) impose strict limits on data sharing [36]. De-identification is fragile: even removing names and social security numbers leaves quasi-identifiers (age, zip code, admission date) that can be linked to external datasets to re-identify individuals [37]. Institutional review boards move slowly. Data use agreements are labyrinthine. And underneath all the bureaucracy lies a genuine moral concern: these are real people, with real vulnerabilities, and sharing their data, however noble the purpose, carries real risk.

Now imagine a different scenario. Instead of sharing the raw patient records, you share synthetic records: a new dataset, generated by a machine learning model, that preserves the statistical properties of the original data (the correlations, the distributions, the predictive signal) without corresponding to any real individual. The researchers train their classifier on this synthetic dataset and achieve performance comparable to what they would have obtained with the real data. No patient was exposed. No regulation was violated. The model still saves sight.

This is the promise of generative models for structured data, and it is the subject of this chapter. But the promise, as we shall see, is far easier to state than to fulfil.

The Access Problem in Practice

The hospital scenario is not hypothetical. Variants of this dilemma play out daily across industries:

Beyond Images and Text

The past decade has witnessed a revolution in generative modelling. Generative adversarial networks (GANs) produce photorealistic faces [38]. Diffusion models generate artwork that wins competitions [24]. Large language models write poetry, code, and legal briefs [39]. These achievements are genuinely impressive. They are also, from the perspective of data types, remarkably narrow.

Images are regular grids of pixels. Each pixel is a tuple of continuous intensity values (red, green, blue) arranged in a fixed two-dimensional spatial layout. The data type is homogeneous: every element is a real number in $[0,255]$. The spatial structure is regular: every pixel has the same neighbourhood geometry. Text is a sequence of tokens drawn from a fixed vocabulary. The data type is again homogeneous: every element is a discrete symbol from a known alphabet. The sequential structure is regular: one token follows another, left to right.

Structured data is neither of these things.

The fundamental difficulty is that structured data combines heterogeneous types in complex, constraint-laden relationships. There is no spatial grid, no fixed vocabulary, no universal tokenization. Each column lives in its own space, with its own distribution, its own semantics, and its own quirks. And the columns are not independent: income correlates with education, hours worked correlates with occupation, and all of them correlate with age in ways that are subtle, nonlinear, and often domain-specific.

Why Structured Data Matters

If structured data is so difficult, why bother? The answer is that structured data is, by a wide margin, the most common form of data in the world. Estimates vary, but enterprise data warehouses, government databases, financial systems, and healthcare records collectively contain orders of magnitude more structured data than the entire corpus of images or text on the internet. When a business analyst queries a database, when a clinician reviews a patient chart, when a regulator audits a bank, they are working with structured data.

The applications of synthetic structured data are vast:

1. Privacy-preserving data sharing. As in our hospital example, synthetic data enables sharing across institutional boundaries without exposing individuals. This is critical in healthcare, finance, and government, where privacy regulations are strictest.

2. Data augmentation for rare events. Fraud detection models struggle because fraudulent transactions are rare (often less than 0.1% of all transactions). Generating synthetic fraud cases can balance the training set and improve classifier performance.

3. Software testing and development. Software teams need realistic test data to validate database schemas, query performance, and application logic. Synthetic data provides this without copying production databases into development environments.

4. Fairness auditing. To audit a model for bias, one needs to test it on populations that may be underrepresented in the training data. Synthetic data can fill these gaps, enabling counterfactual fairness analyses.

5. Simulation and planning. Urban planners, epidemiologists, and supply chain managers all need realistic synthetic populations to run simulations. The US Census Bureau itself produces synthetic data [1] for precisely this purpose.

6. Education and reproducible research. Researchers who cannot share their original data can share synthetic versions, enabling others to reproduce analyses and verify results.

A Roadmap for this Chapter

This chapter traces the arc of structured data generation from its earliest challenges to its most recent breakthroughs. We begin here, in Introduction and Motivation, with motivation and context. In The Landscape of Structured Data, we survey the landscape of structured data types. In Challenges in Structured Data Generation, we catalogue the fundamental challenges that make structured data generation hard. Subsequent sections then address each challenge in turn, presenting the key models and methods:

• CTGAN (CTGAN – The Foundation): the conditional tabular GAN that introduced mode-specific normalisation and training-by-sampling.

• CTAB-GAN+ (CTAB-GAN+ – The Next Generation): the next-generation tabular GAN with auxiliary classifiers and enhanced privacy.

• TabDDPM (TabDDPM – Diffusion Models for Tabular Data): diffusion models adapted for mixed-type tabular data.

• GOGGLE (GOGGLE – Graph-Optimized Generation): graph-based generation that explicitly models relational structure.

• LLM-based approaches: TabuLa, Tabby, and DeLTa, which leverage large language models for tabular generation.

• Harpoon (Harpoon – Manifold-Guided Diffusion for Tabular Data): a recent method addressing multi-table relational data.

Along the way, we will encounter deep connections to differential privacy, optimal transport, score matching, and graph neural networks. By the end of this chapter, you will have a comprehensive understanding of why structured data generation is hard, what the current best solutions are, and where the open problems lie.

The Landscape of Structured Data

Before we can generate structured data, we must understand what it is. The term “structured data” is used loosely in the literature; different communities mean different things by it. In this chapter, we adopt a broad definition that encompasses the major forms of non-image, non-text data encountered in machine learning practice.

This definition is deliberately broad. It includes tabular data (the most common form), graph-structured data, time series, and relational databases. Each of these presents its own generative challenges, and each has spawned its own family of methods. Let us survey them in turn.

Tabular Data

Tabular data is the bread and butter of data science. It is the format of spreadsheets, SQL databases, CSV files, and Pandas DataFrames. A tabular dataset consists of $N$ rows (records, observations, samples) and $D$ columns (features, attributes, variables), arranged in a matrix-like structure where each row represents an independent entity and each column represents a measured property.

The i.i.d. assumption is a simplification; real tabular data sometimes exhibits temporal drift or hierarchical structure. But it is the standard assumption in the tabular generation literature, and it is where most methods begin.

Column types in the wild.

The clean partition into continuous, categorical, and ordinal columns masks considerable practical complexity. Real-world columns exhibit a rich variety of quirks:

• Multimodal continuous columns. Income is not Gaussian: it has a spike at zero (for unemployed or retired individuals), a broad mode around the median, and a heavy right tail. Age has spikes at round numbers (people report “30” more often than “31”).

• High-cardinality categorical columns. ZIP codes in the US have over 40,000 unique values. ICD-10 medical diagnosis codes have over 70,000. Standard one-hot encoding becomes impractical at this scale.

• Mixed-type columns. A “dose” column might be “None” (categorical), “Low”, “Medium”, “High” (ordinal), or a precise numerical value like 2.5 mg (continuous). How should a generative model represent this?

• Missing values. Real tabular data is riddled with missing values, often with non-random missingness patterns. A “missing” income might mean unemployment, refusal to report, or data entry error, each with different statistical properties.

• Implicit constraints. If a patient's “pregnant” column is “yes”, their “sex” column must be “female”. If “employment status” is “retired”, “age” is unlikely to be 25. These logical and statistical constraints are nowhere in the schema but everywhere in the data.

The geometry of tabular data.

It is instructive to think about what tabular data looks like geometrically. Each row is a point in the product space $𝒳={𝒳}_1\times \cdots \times {𝒳}_D$. For a table with three continuous columns, this is simply ${\mathrm{ℝ}}^3$, and we can visualise the data as a point cloud. But for a table with a mix of continuous and categorical columns, the space is more exotic. Consider a table with one continuous column (income) and one categorical column (state, with 51 values). The data space is $\mathrm{ℝ}\times \{1,\dots ,51\}$, which can be visualised as 51 parallel real lines. The data forms clusters on these lines, and the distribution on each line may be very different (income distributions vary dramatically across states). A generative model must capture both the within-line distributions and the relative densities across lines.

For a table with $D_c$ continuous columns and $D_d$ categorical columns (with cardinalities $K_1,\dots ,K_{D_d}$), the data space is ${\mathrm{ℝ}}^{D_c}\times {\prod }_{j=1}^{D_d}\{1,\dots ,K_j\}$. This is a disjoint union of ${\prod }_{j=1}^{D_d}{K}_j$ copies of ${\mathrm{ℝ}}^{D_c}$, one for each combination of categorical values. The number of copies grows exponentially with the number of categorical columns, which is why high-dimensional tabular generation is so challenging: the model must learn a separate continuous distribution for each categorical combination, but most combinations are rarely (or never) observed.

Graph-Structured Data

Not all structured data fits neatly into rows and columns. Consider a social network: each person is a node with attributes (age, location, interests), and friendships are edges with their own attributes (since when, interaction frequency). A molecular graph represents atoms as nodes and chemical bonds as edges. A knowledge graph represents entities and their relationships. In each case, the topology of the graph (which nodes are connected to which) is itself part of the data that must be generated.

Formally, a graph $G=(V,E,{𝐗}^V,{𝐗}^E)$ consists of a node set $V$, an edge set $E\subseteq V\times V$, node feature matrices ${𝐗}^V\in {\mathrm{ℝ}}^{|V|\times d_V}$, and edge feature matrices ${𝐗}^E\in {\mathrm{ℝ}}^{|E|\times d_E}$. Generating a realistic graph thus requires simultaneously generating the topology $(V,E)$ and the attributes $({𝐗}^V,{𝐗}^E)$. This is harder than tabular generation for at least two reasons:

1. Variable size. The number of nodes and edges varies across graphs. A generative model must decide how large each generated graph should be.

2. Permutation invariance. The same graph can be represented by $|V|!$ different adjacency matrices (by permuting node labels). A generative model should ideally be invariant to this relabelling.

Graph generation connects deeply to the molecular design literature (where generating valid drug candidates is the goal), the social network literature (where realistic synthetic networks are needed for simulation), and the combinatorial optimisation literature (where graph generation is a form of constructive heuristic) [7].

Time Series Data

A time series is a sequence of observations indexed by time: ${𝒙}_1,{𝒙}_2,\dots ,{𝒙}_T$, where each ${𝒙}_t\in {\mathrm{ℝ}}^d$ is a $d$-dimensional measurement at time $t$. Unlike tabular data, where rows are (assumed) independent, time series observations are temporally dependent: the value at time $t$ depends on preceding values. This temporal structure is the defining characteristic and the principal generative challenge.

• Autocorrelation. A realistic synthetic stock price must exhibit the same autocorrelation structure as real prices: volatility clustering, mean reversion, and fat-tailed returns.

• Irregular sampling. Medical time series (vital signs, lab tests) are recorded at irregular intervals. The sampling pattern itself carries information (a patient monitored every hour is sicker than one monitored daily).

• Non-stationarity. Many real-world time series (economic indicators, climate data) have statistical properties that change over time. A generative model must capture these drifts.

Time series generation is related to, but distinct from, time series forecasting. Forecasting extends an observed prefix; generation creates entire sequences from scratch. The methods overlap (both use recurrent networks, transformers, and diffusion models) but the evaluation criteria differ.

Relational Data

Perhaps the most challenging form of structured data is relational data: multiple tables connected by foreign key relationships, as in a relational database. Consider an e-commerce database with tables for customers, orders, products, and order_items. A single customer has zero or more orders, each order has one or more items, and each item references a product. Generating synthetic relational data requires not only generating realistic rows in each table but also preserving the referential integrity (every foreign key points to a valid primary key) and the cross-table correlations (a customer's demographics should be consistent with their purchasing patterns).

Challenges in Structured Data Generation

Having surveyed the landscape, we now turn to the heart of the matter: why is generating structured data so hard? The difficulties are numerous, interconnected, and often domain-specific. In this section, we organise them into a taxonomy of challenges. Each challenge will be addressed by one or more of the methods we present in subsequent sections; our goal here is to lay out the full scope of the problem before reaching for solutions.

The Mixed-Type Challenge

The most fundamental challenge in tabular data generation is that columns have different types, and no single representation works well for all of them.

Consider a generator based on a neural network. The network produces a vector of real numbers as output. How should we interpret this output? For a continuous column like “income”, the output can be used directly (after appropriate denormalisation). For a categorical column like “state” (with 51 categories), we need to convert a real-valued vector into a discrete choice. For a binary column like “insured”, we need a probability and a threshold. For a discrete numerical column like “hours worked per week” (with spikes at 0, 20, 35, and 40), we need something that is neither purely continuous nor purely categorical.

The mathematical crux of the problem is that the joint distribution $p(𝒙)=p(x_1,x_2,\dots ,x_D)$ lives in a product space: (Product Space)$$\begin{array}{ccc}& 𝒙\in 𝒳={𝒳}_1\times {𝒳}_2\times \cdots \times {𝒳}_D,& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${𝒳}_j=\mathrm{ℝ}$ for continuous columns, ${𝒳}_j=\{1,\dots ,K_j\}$ for categorical columns, and so on. This product space is neither continuous (because of the categorical dimensions) nor discrete (because of the continuous dimensions). It is a hybrid space, and probability distributions over hybrid spaces do not have a single natural parameterisation.

The key insight, which we will formalise when we study CTGAN in CTGAN – The Foundation, is that each column type requires its own representation and its own loss function:

• Continuous columns $\to$ mode-specific normalisation $+$ mean squared error.

• Categorical columns $\to$ one-hot encoding $+$ cross-entropy loss.

• Discrete numerical columns $\to$ hybrid strategies that combine aspects of both.

The Correlation Challenge

Even if we solve the mixed-type problem for each column individually, we face a deeper challenge: the columns are not independent. The joint distribution $p(x_1,x_2,\dots ,x_D)$ is not the product of its marginals ${\prod }_{j=1}^{D}p(x_j)$. The correlations between columns carry the information that makes data useful, and destroying them renders the synthetic data worthless.

The correlation structure of a tabular dataset can be conceptualised at multiple levels:

1. Pairwise (marginal) correlations. The correlation between any two columns. Pearson's $r$ for continuous–continuous pairs, Cramér's $V$ for categorical–categorical pairs, and various mixed measures for continuous–categorical pairs.

2. Conditional correlations. The correlation between two columns given a third. For instance, the correlation between income and education may differ across age groups.

3. Higher-order interactions. Patterns that involve three or more columns simultaneously. For example, the combination (male, age $>60$, smoker) may have a specific risk profile that is not captured by any pairwise relationship.

4. Functional dependencies. Deterministic relationships between columns. Zip code determines city and state. BMI is exactly weight/(height${}^2$). These are not statistical correlations but logical constraints.

A particularly insidious failure mode is what we might call the “marginal trap”. A generative model produces synthetic data where each column, viewed in isolation, looks realistic: the distribution of ages matches, the distribution of incomes matches, the distribution of states matches. But the joint distribution is wrong: the correlations between columns have been destroyed. The synthetic data passes marginal tests with flying colours but fails joint tests miserably.

The difficulty of preserving correlations grows rapidly with the dimensionality of the data. For a table with $D$ columns, there are $\left(\genfrac{}{}{0px}{}{D}{2}\right)$ pairwise correlations, $\left(\genfrac{}{}{0px}{}{D}{3}\right)$ three-way interactions, and so on. Capturing all of these with finite data and finite model capacity is a fundamentally hard problem, closely related to the curse of dimensionality in density estimation.

In practice, most methods rely on the neural network's capacity to implicitly learn the relevant correlations from data, rather than explicitly modelling each interaction. Whether this implicit approach suffices, or whether explicit correlation modelling (as in GOGGLE's graph-based approach [7]) is necessary, remains an active debate.

The Constraint Satisfaction Challenge

Beyond statistical correlations, real-world structured data satisfies a host of hard constraints: logical, physical, regulatory, and domain-specific. Violating these constraints produces synthetic records that are not merely statistically implausible but logically impossible.

Constraints can be categorised along two axes:

1. Hard vs. soft. Hard constraints must be satisfied exactly (e.g., SpO${}_2\le 100$). Soft constraints are statistical regularities that should be approximately respected (e.g., most people over 65 are retired).

2. Explicit vs. implicit. Explicit constraints are documented in the schema or domain knowledge. Implicit constraints are discovered only by inspecting the data (e.g., no one in the dataset has both income $>\$\mathrm{500,000}$ and education $=$ “No schooling”, but this is not a logical impossibility).

Handling constraints in generative models is an active area of research. Current approaches fall into three categories:

• Post-hoc rejection/correction. Generate unconstrained samples and either reject those that violate constraints or correct them. Simple but wasteful, and correction can distort the learned distribution.

• Architectural enforcement. Design the generator architecture so that constraint violations are impossible (e.g., using softmax to ensure probabilities sum to 1, or clipping outputs to valid ranges). Effective for simple constraints but brittle for complex ones.

• Loss-based guidance. Add penalty terms to the generator's loss function that discourage constraint violations. Flexible but does not guarantee satisfaction.

The Evaluation Challenge

Suppose you have generated a synthetic dataset. How do you know it is “good”? For images, the answer is relatively well established: Fréchet Inception Distance (FID) measures distributional similarity using features from a pre-trained network [40]. For text, perplexity and human evaluation provide useful signals. For structured data, the evaluation landscape is far more fragmented, and there is no single universally accepted metric.

Fidelity is necessary, but it is only one dimension of quality. A comprehensive evaluation considers multiple axes:

Axis 1: Statistical fidelity.

Does the synthetic data have the same distributions, correlations, and higher-order statistics as the real data? Common metrics include:

• Column-wise distributional tests (Kolmogorov–Smirnov for continuous, chi-squared for categorical).

• Pairwise correlation matrices (Pearson's $r$, Cramér's $V$).

• Maximum mean discrepancy (MMD) between real and synthetic feature distributions.

Axis 2: Machine learning utility.

Can a model trained on synthetic data perform comparably to one trained on real data? The standard protocol is “train on synthetic, test on real” (TSTR): train a classifier or regressor on ${𝒯}^{\prime }$, evaluate on a held-out portion of $𝒯$, and compare against a model trained on $𝒯$ itself [4].

Axis 3: Privacy.

Does the synthetic data protect the privacy of individuals in the training set? Key metrics include:

• Distance to closest record (DCR): the minimum distance from each synthetic record to the nearest real record.

• Membership inference attack accuracy: can an adversary determine whether a specific individual was in the training data?

• Attribute inference risk: given partial knowledge of a record, can an adversary infer sensitive attributes?

Axis 4: Diversity.

Does the synthetic data exhibit the full diversity of the real data, or has the generator collapsed to a subset of modes? This is related to the mode collapse problem in GANs, and is measured by coverage metrics and support size estimates.

Axis 5: Constraint satisfaction.

Does the synthetic data satisfy known domain constraints? This axis is specific to structured data and is absent in the image generation literature.

The Privacy–Utility Tradeoff

The most fundamental tension in synthetic data generation is between privacy and utility. These two goals are inherently at odds: the more faithfully the synthetic data reproduces the real data, the more information it leaks about the individuals in the real data. Conversely, the more aggressively we protect privacy, the more we degrade the statistical properties that make the data useful.

This tradeoff is formalised by differential privacy [41], which provides a mathematically rigorous framework for quantifying privacy loss.

The key implication for synthetic data is this: if we train a generative model $G$ on real data $𝒯$ using a differentially private optimiser (e.g., DP-SGD [17]), then the trained model $G_{𝜽}$ is itself differentially private, and any synthetic data sampled from $G_{𝜽}$ inherits the privacy guarantee (by the post-processing theorem of differential privacy).

The tradeoff manifests as follows:

• Low $ϵ$ (strong privacy). Noise added during training is large. The model learns a blurred version of the real distribution. Synthetic data has poor fidelity and low ML utility.

• High $ϵ$ (weak privacy). Noise is small. The model learns the real distribution accurately. Synthetic data has high fidelity and high ML utility, but the privacy guarantee is weak.

• Intermediate $ϵ$. The practical sweet spot, where the synthetic data is “good enough” for the intended purpose while providing “strong enough” privacy for the regulatory context.

The Scalability Challenge

Production tabular datasets can be large along multiple dimensions: millions of rows, hundreds or thousands of columns, and complex schemas with dozens of tables. Scalability challenges arise in several forms:

1. Column-count scalability. One-hot encoding a categorical column with $K$ categories adds $K$ dimensions. A table with 50 categorical columns averaging 100 categories each becomes a 5,000-dimensional one-hot vector. Training a GAN in 5,000 dimensions is non-trivial.

2. Row-count scalability. Training generative models on millions of rows is computationally expensive, particularly for diffusion models that require multiple denoising steps per sample.

3. Schema complexity. Multi-table datasets with complex foreign key relationships require models that scale with the number of tables and the depth of the join graph.

4. Conditional generation. Generating data conditioned on specific constraints (e.g., “generate 1,000 records of female patients over 65 with diabetes”) requires the model to navigate high-dimensional conditional distributions efficiently.

The Temporal Challenge

When tabular data has a temporal dimension (e.g., financial transactions with timestamps, patient visits over time), the i.i.d. assumption breaks down. Rows are no longer exchangeable: their order matters, and the dependencies between them carry crucial information.

A sequence of credit card transactions for a single customer exhibits patterns (regular purchases at a grocery store every Tuesday, a rent payment on the first of each month) that a generative model must capture. Merely generating i.i.d. transactions with the right marginal distributions would produce a customer whose spending pattern is random noise.

The temporal challenge connects structured data generation to the broader field of time series generation, but with the additional complication that each “time step” is a mixed-type tabular record, not a simple numerical vector.

A Unified View of the Challenges

Let us step back and view the challenges holistically. At the highest level of abstraction, generating structured data requires learning a joint distribution $p(𝒙)$ over a product of heterogeneous spaces, subject to hard constraints, with limited data, while providing privacy guarantees and preserving the correlations that make the data useful. Each challenge corresponds to an aspect of this problem that is trivial for images or text but difficult for structured data:

The Historical Arc

The field of structured data generation has evolved rapidly. In the span of about a decade, it has progressed from simple Bayesian networks and copula models to sophisticated deep generative architectures that rival or exceed classical methods on standard benchmarks. Understanding this trajectory helps situate the methods we will study.

The historical arc reveals several important trends:

1. From general-purpose to specialised. Early attempts simply applied standard GANs to tabular data. Success came from specialising the architecture to respect the structure of the data: mode-specific normalisation for multimodal continuous columns, training-by-sampling for imbalanced categories, and hybrid noise processes for mixed-type columns.

2. From GANs to diversity. The GAN framework dominated tabular generation from 2019 to 2022, but the post-2022 era has seen a rapid diversification: diffusion models (TabDDPM), graph-based methods (GOGGLE), LLM-based approaches (TabuLa, GReaT, Tabby), and multi-table methods (Harpoon). This suggests that no single framework will dominate.

3. From generation to evaluation. Early papers focused on generating data and reported limited metrics. Recent work increasingly focuses on rigorous evaluation, with standardised benchmarks, multiple metrics, and privacy–utility analyses becoming the norm.

4. From single-table to multi-table. The field has gradually moved from single flat tables to the richer setting of relational databases. This mirrors the shift in the broader ML community from simple to complex data structures.

Formalising the Goal

Having catalogued the challenges, let us formalise what we are trying to achieve. The structured data generation problem can be stated as follows.

This formalisation makes explicit what we want and reveals the fundamental tension: requirements 1 and 2 are inherently at odds (the privacy–utility tradeoff), while requirements 1 and 3 may conflict if the learned distribution assigns mass to constraint-violating regions. The art of structured data generation lies in navigating these tensions, and the methods in the following sections represent different strategies for doing so.

What Lies Ahead

With the challenges laid out, we are ready to confront them. The structure of our attack follows the historical arc of the field. We begin with the breakthrough that made tabular data generation practical: CTGAN and its solution to the mixed-type and mode collapse problems. We then study the refinements of CTAB-GAN+, the diffusion-based approach of TabDDPM, the graph-based perspective of GOGGLE, and the LLM-based methods that leverage the power of foundation models. Finally, we address the frontier of multi-table relational generation with Harpoon.

At each stage, we will return to the challenges catalogued in this section and ask: which challenges does this method solve, which does it mitigate, and which does it leave open? By the end of the chapter, you will have not only a toolkit of methods but also a deep understanding of why each method works, where it succeeds, and where it fails.

CTGAN – The Foundation

Before 2019, the dream of generating realistic tabular data with generative adversarial networks was, to put it politely, a dream deferred. Researchers had tried the obvious approach: take a GAN, feed it rows of tabular data, train, and hope. The results were consistently disappointing. Generated tables exhibited bizarre artefacts – continuous columns collapsed to a single mode, rare categories vanished entirely, and the correlations between columns that make real data useful dissolved into noise. The problem was not that GANs were incapable of modelling structured data; the problem was that nobody had figured out how to speak the language of tabular data in a form that GANs could understand.

The breakthrough came from Lei Xu, Maria Skoularidou, Alfredo Cuesta-Infante, and Kalyan Veeramachaneni at MIT, who published CTGAN – the Conditional Tabular GAN – in 2019 [4]. Their insight was deceptively simple: the failure of previous tabular GANs was not a failure of architecture but a failure of representation. Continuous columns in real data are often multimodal; categorical columns are often imbalanced. Standard GAN training, which assumes that the data distribution is smooth and that all modes are equally important, is catastrophically ill-suited to these realities.

CTGAN introduced three innovations that, together, cracked the code:

1. Mode-specific normalisation: fitting a variational Gaussian mixture model (VGM) to each continuous column and representing values as (mode indicator, normalised residual) pairs.

2. Training-by-sampling: a conditional generation strategy that forces the generator to produce samples from every category, including rare ones, thereby combating mode collapse on discrete columns.

3. PacGAN integration: packing multiple generated samples together before feeding them to the discriminator, providing a structural defence against mode collapse.

Each of these ideas is individually straightforward; their power lies in the combination. In the sections that follow, we dissect each component with full mathematical detail, build up the architecture piece by piece, and prove the key properties that make CTGAN work.

The Architecture

Let us begin with the overall architecture. CTGAN uses a generator–discriminator pair, both fully connected, but with crucial modifications that distinguish it from a vanilla GAN.

Generator.

The generator $G_{𝜽}$ takes two inputs: a noise vector $𝒛\sim 𝒩(0,{𝐈}_d)$ drawn from a $d$-dimensional standard Gaussian, and a conditional vector $𝒎\in {\{\mathrm{0,1}\}}^{|𝒞|}$ that specifies which category (and which column) the generated row should exhibit. The concatenated input $[𝒛;𝒎]$ passes through a stack of fully connected layers with batch normalisation and ReLU activations: (Layer)$$\begin{array}{ccc}& {𝒉}^{(\ell +1)}=\mathrm{ReLU}\left(\mathrm{BN}\right({𝐖}^{(\ell )}{𝒉}^{(\ell )}+{𝒃}^{(\ell )}\left)\right),\ell =0,1,\dots ,L-1,& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${𝒉}^{(0)}=[𝒛;𝒎]$, and ${𝐖}^{(\ell )},{𝒃}^{(\ell )}$ are the weight matrix and bias vector of layer $\ell$. The final layer uses a combination of activation functions matched to the output representation:

• Tanh for the normalised continuous values (which lie in $[-\mathrm{1,1}]$ after mode-specific normalisation).

• Softmax for the mode indicator one-hot vectors of each continuous column.

• Gumbel-softmax for the one-hot representations of categorical columns.

Discriminator.

The discriminator $D_{𝝓}$ takes a (possibly packed) tabular row and outputs a scalar score. Its architecture is also fully connected but uses leaky ReLU activations and dropout: (Layer)$$\begin{array}{ccc}& {𝒈}^{(\ell +1)}=\mathrm{Dropout}\left(\mathrm{LeakyReLU}\right({𝐔}^{(\ell )}{𝒈}^{(\ell )}+{𝒄}^{(\ell )}),p_{\mathrm{drop}}),\ell =0,1,\dots ,L^{\prime }-1,& \text{<span class="tml-eqn"></span>}\end{array}$$ where $p_{\mathrm{drop}}\in (\mathrm{0,1})$ is the dropout probability (typically $p_{\mathrm{drop}}=0.5$). The use of leaky ReLU rather than ReLU avoids the dying neuron problem, while dropout acts as a regulariser that prevents the discriminator from memorising specific real samples.

Now we turn to PacGAN, one of CTGAN's most important defences against mode collapse.

The following TikZ diagram shows the full CTGAN architecture.

The next diagram illustrates the PacGAN packing concept in detail.

Mode-Specific Normalisation in Detail

We now turn to the heart of CTGAN's preprocessing strategy: mode-specific normalisation (MSN). This technique addresses a fundamental mismatch between the assumptions of neural networks and the reality of tabular continuous columns.

Standard neural network training assumes that input features are roughly Gaussian or at least unimodal. Batch normalisation, weight initialisation schemes, and the geometry of gradient descent all work best when the data is centred and symmetric. But continuous columns in real-world tabular data are often wildly non-Gaussian. A salary column might have one mode around $40,000 for entry-level employees and another around $120,000 for senior staff. An age column might be roughly uniform. A transaction amount column might follow a power law with a massive spike at zero.

If we naively min–max normalise such a column to $[\mathrm{0,1}]$, we compress the multimodal structure into a range where the generator must learn to reproduce multiple peaks – a task that standard generators handle poorly. Mode-specific normalisation solves this by decomposing each continuous column into a mixture of Gaussians, then representing each value relative to the Gaussian mode it most likely belongs to.

The procedure has three steps.

Step 1: Fit a Variational Gaussian Mixture.

For each continuous column $c_j$ in the dataset, we fit a variational Gaussian mixture model (VGM) with $M_j$ components. Let the fitted parameters be $$\{(w_{j,1},{\mu }_{j,1},{\sigma }_{j,1}),\dots ,(w_{j,M_j},{\mu }_{j,M_j},{\sigma }_{j,M_j})\},$$ where $w_{j,m}$ is the weight, ${\mu }_{j,m}$ the mean, and ${\sigma }_{j,m}$ the standard deviation of component $m$. The VGM is fitted using variational Bayesian inference, which automatically prunes components with negligible weight, effectively performing model selection.

Step 2: Assign Mode and Normalise.

For a value $x$ in column $c_j$, we compute the probability that $x$ belongs to each mode: (Assignment)$$\begin{array}{ccc}& {\rho }_{j,m}(x)=\frac{w_{j,m}\varphi ((x-{\mu }_{j,m})/{\sigma }_{j,m})}{{\sum }_{m^{\prime }=1}^{M_j}{w}_{j,m^{\prime }}\varphi ((x-{\mu }_{j,m^{\prime }})/{\sigma }_{j,m^{\prime }})},& \text{<span class="tml-eqn"></span>}\end{array}$$ where $\varphi (\cdot )$ is the standard normal density. We then sample a mode: (Sample)$$\begin{array}{ccc}& m^{\ast }\sim \mathrm{Categorical}({\rho }_{j,1}(x),\dots ,{\rho }_{j,M_j}(x)),& \text{<span class="tml-eqn"></span>}\end{array}$$ and compute the normalised value within that mode: (Normalize)$$\begin{array}{ccc}& {\alpha }_j(x)=\frac{x-{\mu }_{j,m^{\ast }}}{4{\sigma }_{j,m^{\ast }}}.& \text{<span class="tml-eqn"></span>}\end{array}$$ The factor of 4 ensures that approximately 99.99% of the values within a given mode fall in the range $[-1,1]$, which matches the range of the tanh activation in the generator output layer.

Step 3: Encode as (Mode One-Hot, Normalised Value).

The final representation of value $x$ in column $c_j$ is the pair $$({𝜷}_j(x),{\alpha }_j(x))\in \{\mathrm{0,1}{\}}^{M_j}\times [-\mathrm{1,1}],$$ where ${𝜷}_j(x)$ is a one-hot vector with a 1 in position $m^{\ast }$ and zeros elsewhere, and ${\alpha }_j(x)$ is the normalised scalar. The generator must produce both ${𝜷}_j$ (via softmax) and ${\alpha }_j$ (via tanh) for each continuous column.

The following figure visualises the GMM fitting and mode assignment process.

Training-by-Sampling

Mode-specific normalisation handles the representation of continuous columns. But what about discrete columns? Here, a different demon lurks: class imbalance.

Consider a dataset with a binary column “Fraud” where 99% of rows are non-fraudulent and 1% are fraudulent. If we train a GAN on this data with standard uniform sampling, the generator sees fraudulent examples extremely rarely. The discriminator, meanwhile, can achieve high accuracy by simply memorising the dominant class. The generator's rational response is to ignore the rare class entirely – it can fool the discriminator most of the time by generating only non-fraudulent rows. This is mode collapse on discrete columns, and it is even more pernicious than mode collapse on continuous columns because the lost modes correspond to semantically meaningful categories.

Training-by-sampling is CTGAN's elegant solution. The idea is to condition each training step on a randomly selected category, ensuring that every category – no matter how rare – receives equal attention during training.

The following diagram illustrates the conditional vector construction.

The Loss Function

CTGAN combines two loss components: the adversarial loss from WGAN-GP and a conditional loss that enforces the training-by-sampling constraint.

WGAN-GP Adversarial Loss.

The adversarial training follows the Wasserstein GAN with gradient penalty framework [11]. The discriminator loss is: (DISC)$$\begin{array}{ccc}& {ℒ}_D=\underset{\text{Wasserstein estimate}}{\underbrace{{𝔼}_{\widehat{𝒙}\sim p_G}[D(\widehat{𝒙})]-{𝔼}_{𝒙\sim p_{\mathrm{data}}}[D(𝒙)]}}+\underset{\text{gradient penalty}}{\underbrace{{\lambda }_{\mathrm{gp}}{𝔼}_{\widetilde{𝒙}}\left[\right(\Vert {\nabla }_{\widetilde{𝒙}}D(\widetilde{𝒙}){\Vert }_2-1{)}^2]}},& \text{<span class="tml-eqn"></span>}\end{array}$$ where $\widetilde{𝒙}=ϵ𝒙+(1-ϵ)\widehat{𝒙}$ with $ϵ\sim \mathrm{Uniform}(\mathrm{0,1})$ is a random interpolation between real and fake samples, and ${\lambda }_{\mathrm{gp}}=10$ is the gradient penalty coefficient.

Note that when PacGAN is used, the discriminator $D$ operates on packs of samples, so the Wasserstein estimate is computed over packed inputs rather than individual rows.

The generator loss consists of the adversarial component plus the conditional loss: (GEN)$$\begin{array}{ccc}& {ℒ}_G=-{𝔼}_{𝒛,𝒎}\left[D\right(G(𝒛,𝒎)\left)\right]+{\lambda }_{\mathrm{cond}}{ℒ}_{\mathrm{cond}},& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${ℒ}_{\mathrm{cond}}$ is the cross-entropy loss on the conditional vector.

Conditional Cross-Entropy Loss.

Let $𝒎$ be the conditional vector specifying that category $c_{k,i}$ should appear in column $d_k$ of the generated row. Let ${\widehat{𝒚}}_k$ be the softmax output of the generator for column $d_k$. Then: (COND)$$\begin{array}{ccc}& {ℒ}_{\mathrm{cond}}=-\sum _{i^{\prime }=1}^{|{𝒞}_k|}{m}_{k,i^{\prime }}\log {\hat{y}}_{k,i^{\prime }},& \text{<span class="tml-eqn"></span>}\end{array}$$ where $m_{k,i^{\prime }}$ is the component of $𝒎$ corresponding to category $c_{k,i^{\prime }}$. Since $𝒎$ is one-hot with 1 at position $i$, this simplifies to $${ℒ}_{\mathrm{cond}}=-\log {\hat{y}}_{k,i},$$ i.e., the negative log-probability that the generated row contains the specified category.

Limitations and What Comes Next

CTGAN was a watershed moment for tabular data generation, but it is not without limitations. Understanding these limitations is essential for appreciating the innovations that followed.

Limited inter-column dependency modelling.

CTGAN's generator is a fully connected network. While fully connected layers can theoretically model any function, they have no inductive bias toward capturing structured dependencies between columns. Real-world tabular data often has rich correlations: a person's income correlates with their education, which correlates with their age, which correlates with their health status. CTGAN learns these correlations implicitly through the adversarial loss, but it has no mechanism for explicitly encoding the relational structure between features.

No privacy guarantees.

CTGAN provides no formal privacy guarantees. Synthetic data generated by CTGAN may inadvertently memorise and reproduce identifiable patterns from the training data. In domains like healthcare and finance, where privacy is paramount, this is a significant limitation.

Single-table limitation.

CTGAN operates on a single flat table. Real-world databases are relational, with multiple tables linked by foreign keys. Generating coherent synthetic data across multiple related tables requires fundamentally different approaches.

Scalability.

The mode-specific normalisation preprocessing is performed once and is efficient, but the training procedure – particularly with PacGAN and WGAN-GP's gradient penalty – can be slow. The gradient penalty requires computing second-order derivatives, which is computationally expensive.

CTAB-GAN+ – The Next Generation

If CTGAN cracked the code of representing tabular data for GANs, CTAB-GAN+ asked a bolder question: what if we could represent tabular data not just for GANs, but as something GANs already know how to generate brilliantly – images?

The idea, proposed by Zhao et al. [42], sounds almost too simple to work. Take a tabular row, reshape it into a 2D matrix, and apply the convolutional architectures that have proven spectacularly successful in image generation. Yet this seemingly naive trick, combined with careful auxiliary losses and an optional differential privacy mechanism, produces synthetic data that consistently outperforms CTGAN across a wide range of benchmarks.

The key insight is that convolutional layers have a built-in inductive bias that fully connected layers lack: they excel at capturing local interactions. When we arrange tabular features into a 2D grid, nearby features in the grid interact through convolutional kernels. If we are thoughtful about the arrangement – placing correlated features near each other – the convolutional architecture naturally captures the feature interactions that CTGAN's fully connected layers must learn from scratch.

What CTAB-GAN+ Fixes

Before diving into the architecture, let us understand the core ideas that distinguish CTAB-GAN+ from CTGAN.

The CNN Insight.

A tabular row $𝒙=(x_1,x_2,\dots ,x_n)$ is naturally a 1D vector. CTGAN treats it as such and processes it with fully connected layers. CTAB-GAN+ makes a creative leap: wrap the 1D vector into a 2D matrix and process it with convolutional layers.

Why would this help? Consider a convolutional filter of size $3\times 3$ operating on a 2D matrix formed from a tabular row. This filter sees a local neighbourhood of features – it sees feature $x_i$ together with its neighbours $x_{i-1}$, $x_{i+1}$, and the features in adjacent rows of the matrix. If correlated features are placed near each other in the grid, the convolutional filter automatically captures their interactions. A fully connected layer, by contrast, treats all feature pairs symmetrically – it has no notion of locality.

This is the same principle that makes CNNs superior to fully connected networks for image processing: the inductive bias of locality. In images, nearby pixels are correlated; in tabular data, after careful arrangement, nearby features can also be correlated.

The following figure illustrates the tabular-to-image encoding.

The Architecture

CTAB-GAN+ consists of four components: an encoder that converts tabular rows to images, a generator that produces synthetic images from noise, a discriminator that distinguishes real from fake images, and a decoder that converts generated images back to tabular rows. The generator and discriminator follow the DCGAN architecture.

Encoder.

The encoder $E$ applies mode-specific normalisation to continuous columns and one-hot encoding to categorical columns (exactly as in CTGAN), then reshapes the resulting vector into a 2D matrix using the tabular-to-image encoding (Definition Definition 11). The encoder is not a neural network; it is a deterministic, fixed transformation computed once during preprocessing.

Generator.

The generator $G_{𝜽}$ follows the DCGAN transposed convolution architecture. It takes a noise vector $𝒛\sim 𝒩(0,𝐈)$ and produces an $h\times w$ single-channel image: (GEN)$$\begin{array}{ccc}& G_{𝜽}(𝒛)\in {\mathrm{ℝ}}^{h\times w\times 1}.& \text{<span class="tml-eqn"></span>}\end{array}$$ The architecture consists of a fully connected layer that maps $𝒛$ to a small spatial feature map (e.g., $\lfloor h/8\rfloor \times \lfloor w/8\rfloor \times 256$), followed by a series of transposed convolution layers with batch normalisation and ReLU activations, progressively upsampling to the target resolution: (Layer)$$\begin{array}{ccc}& {𝒉}^{(\ell +1)}=\mathrm{ReLU}\left(\mathrm{BN}\right(\mathrm{ConvTranspose2D}({𝒉}^{(\ell )})\left)\right).& \text{<span class="tml-eqn"></span>}\end{array}$$ The final layer uses a tanh activation.

Discriminator.

The discriminator $D_{𝝓}$ is a CNN with strided convolutions, batch normalisation, and leaky ReLU activations. Critically, it has two heads:

1. A real/fake head that outputs a scalar score (as in standard GAN training).

2. An auxiliary classifier head that predicts the class label of the input (for datasets with a designated target variable).

The discriminator architecture mirrors the generator in reverse: (Layer)$$\begin{array}{ccc}& {𝒈}^{(\ell +1)}=\mathrm{LeakyReLU}\left(\mathrm{BN}\right(\mathrm{Conv2D}({𝒈}^{(\ell )})),0.2).& \text{<span class="tml-eqn"></span>}\end{array}$$

Decoder.

The decoder $D_{\mathrm{dec}}$ reverses the encoder: it takes the generated $h\times w$ image, flattens it, strips padding, and applies the inverse of mode-specific normalisation to recover tabular feature values. Like the encoder, the decoder is a deterministic function, not a neural network.

The full architecture is shown in Figure fig:ctabganp:arch.

Auxiliary Classifier and Information Loss

CTAB-GAN+ introduces two loss terms beyond the standard adversarial loss: the auxiliary classifier loss and the information loss. Each addresses a specific weakness of vanilla GAN training for tabular data.

Auxiliary Classifier Loss.

Many tabular datasets have a designated target variable $y$ (e.g., “Income $>$ $50K” in the Adult dataset, “Fraud” in transaction data). Downstream machine learning tasks typically involve predicting $y$ from the other features. If the synthetic data does not preserve the conditional distribution $p(𝒙|y)$, then classifiers trained on synthetic data will perform poorly on real test data.

The auxiliary classifier ensures class-conditional fidelity. The discriminator's auxiliary head predicts the class label $y$ from the input image (whether real or fake). This yields two loss components:

For real data: (REAL)$$\begin{array}{ccc}& {ℒ}_{\mathrm{cls}}^D=-{𝔼}_{(𝐗,y)\sim p_{\mathrm{data}}}[\log D_{\mathrm{aux}}(𝐗)[y]],& \text{<span class="tml-eqn"></span>}\end{array}$$ where $D_{\mathrm{aux}}(𝐗)[y]$ is the predicted probability of class $y$.

For generated data, the generator is also trained to produce images that the auxiliary classifier labels correctly: (GEN)$$\begin{array}{ccc}& {ℒ}_{\mathrm{cls}}^G=-{𝔼}_{𝒛,y}\left[\log D_{\mathrm{aux}}\right(G_{𝜽}(𝒛))[y]].& \text{<span class="tml-eqn"></span>}\end{array}$$

The auxiliary classifier serves a dual purpose. For the discriminator, it provides a richer training signal: the discriminator must not only distinguish real from fake but also correctly classify the data, forcing it to learn semantically meaningful features. For the generator, it ensures that generated samples are not only realistic but also class-consistent.

Information Loss.

The information loss is a reconstruction penalty that ensures the generated images can be meaningfully decoded back to tabular data. Without this loss, the generator might produce images that fool the discriminator but, when decoded, yield nonsensical tabular values.

Let $D_{\mathrm{dec}}$ denote the deterministic decoder that converts a generated image back to a tabular row. The information loss has three components, one for each type of column:

For continuous columns (using mean squared error): (CONT)$$\begin{array}{ccc}& {ℒ}_{\mathrm{info}}^{\mathrm{cont}}=\frac{1}{J}\sum _{j=1}^J{𝔼}_{𝒛}\left[\right({\alpha }_j-{\hat{\alpha }}_j{)}^2],& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${\alpha }_j$ is the target normalised value and ${\hat{\alpha }}_j$ is the value extracted from the generated image.

For categorical columns (using cross-entropy): (CAT)$$\begin{array}{ccc}& {ℒ}_{\mathrm{info}}^{\mathrm{cat}}=-\frac{1}{K}\sum _{k=1}^K{𝔼}_{𝒛}\left[\sum _{i=1}^{|{𝒞}_k|}{c}_{k,i}\log {\hat{c}}_{k,i}\right],& \text{<span class="tml-eqn"></span>}\end{array}$$ where $c_{k,i}$ is the target one-hot encoding and ${\hat{c}}_{k,i}$ is the generated softmax output for category $i$ of column $k$.

For mode indicators (also cross-entropy): (MODE)$$\begin{array}{ccc}& {ℒ}_{\mathrm{info}}^{\mathrm{mode}}=-\frac{1}{J}\sum _{j=1}^J{𝔼}_{𝒛}\left[\sum _{m=1}^{M_j}{\beta }_{j,m}\log {\hat{\beta }}_{j,m}\right].& \text{<span class="tml-eqn"></span>}\end{array}$$

The total information loss is: (Total)$$\begin{array}{ccc}& {ℒ}_{\mathrm{info}}={ℒ}_{\mathrm{info}}^{\mathrm{cont}}+{ℒ}_{\mathrm{info}}^{\mathrm{cat}}+{ℒ}_{\mathrm{info}}^{\mathrm{mode}}.& \text{<span class="tml-eqn"></span>}\end{array}$$

Differential Privacy Integration (DP-SGD)

One of CTAB-GAN+'s most important features is its integration with differential privacy via the DP-SGD mechanism. This is not merely a theoretical flourish; in domains like healthcare, finance, and government, the ability to provide provable privacy guarantees for synthetic data is a regulatory requirement.

Differential privacy formalises the intuitive notion that the output of a computation should not reveal too much about any single individual in the input dataset.

The standard technique for making SGD differentially private is DP-SGD, introduced by Abadi et al. [17]. The idea is to modify each gradient update step in two ways: clip each per-sample gradient to bound its sensitivity, then add calibrated noise to the clipped sum.

Gradient Clipping.

In standard SGD, the gradient of the loss with respect to the model parameters is: $$𝒈=\frac{1}{B}\sum _{i=1}^B{𝒈}_i,{𝒈}_i={\nabla }_{𝝓}ℒ(𝝓;{𝒙}_i),$$ where $B$ is the batch size. In DP-SGD, we first clip each per-sample gradient: (CLIP)$$\begin{array}{ccc}& {\overline{𝒈}}_i={𝒈}_i\cdot \min (1,\frac{C}{\Vert {𝒈}_i{\Vert }_2}),& \text{<span class="tml-eqn"></span>}\end{array}$$ where $C>0$ is the clipping norm. This ensures $\Vert {\overline{𝒈}}_i{\Vert }_2\le C$ for all $i$, bounding the influence of any single sample on the gradient.

Noise Addition.

After clipping, we add Gaussian noise calibrated to the clipping norm: (Noise)$$\begin{array}{ccc}& \widetilde{𝒈}=\frac{1}{B}(\sum _{i=1}^B{\overline{𝒈}}_i+𝒩(0,{\sigma }^2{C}^2𝐈)),& \text{<span class="tml-eqn"></span>}\end{array}$$ where $\sigma >0$ is the noise multiplier. The parameter update is then $𝝓\leftarrow 𝝓-\eta \widetilde{𝒈}$.

In CTAB-GAN+, DP-SGD is applied to the discriminator's training only. The generator never sees real data directly – it only receives gradients through the discriminator – so if the discriminator's training is differentially private, the entire pipeline inherits the privacy guarantee by the post-processing property of differential privacy.

The following figure visualises the DP-SGD gradient clipping and noise addition process.

Experimental Results and Insights

With the architecture and training procedure established, let us examine how CTAB-GAN+ performs in practice. The original paper evaluates on five widely used benchmark datasets: Adult, Covertype, Credit, Intrusion, and Loan. These datasets span classification and regression tasks, vary in size from thousands to hundreds of thousands of rows, and include a mix of continuous, categorical, and mixed-type columns.

Evaluation Protocol.

The standard evaluation protocol for tabular generative models proceeds as follows:

1. Train the generative model on the real training data.

2. Generate a synthetic dataset of the same size.

3. Train downstream machine learning models (e.g., logistic regression, random forest, gradient-boosted trees) on the synthetic data.

4. Evaluate these models on the real test data.

5. Compare the test-set performance to models trained on the real training data.

The closer the “train-on-synthetic, test-on-real” (TSTR) performance is to the “train-on-real, test-on-real” (TRTR) baseline, the better the generative model.

Additionally, statistical similarity metrics are computed between the real and synthetic data distributions:

• Kolmogorov–Smirnov (KS) statistic for continuous columns: measures the maximum difference between the empirical CDFs.

• Jensen–Shannon divergence for categorical columns: a symmetric version of KL divergence.

• Correlation structure: comparing the pairwise correlation matrix of real vs. synthetic data.

Results.

The following table summarises the key comparison between CTGAN and CTAB-GAN+ on the benchmark datasets (results from the original CTAB-GAN+ paper [42]).

	CTGAN			CTAB-GAN+			Real
(lr)2-4 (lr)5-7 (lr)8-8 Dataset	F1	KS	JSD	F1	KS	JSD	F1
Adult	0.67	0.08	0.02	0.72	0.05	0.01	0.78
Covertype	0.52	0.12	0.05	0.61	0.07	0.03	0.74
Credit	0.55	0.10	0.04	0.62	0.06	0.02	0.68
Intrusion	0.71	0.09	0.03	0.78	0.05	0.01	0.85
Loan	0.60	0.11	0.04	0.66	0.07	0.02	0.72

Several patterns emerge from these results:

1. Consistent improvement. CTAB-GAN+ outperforms CTGAN on every dataset and every metric. The ML utility gap is 5–9 F1 points, which is substantial in practical terms.

2. Better statistical fidelity. The KS statistics and JSD values are roughly halved, indicating that CTAB-GAN+ produces distributions much closer to the real data.

3. Still a gap to real data. Even CTAB-GAN+ does not close the gap to models trained on real data. The remaining 6–13 F1 point deficit suggests that there is still significant room for improvement in tabular generation.

4. Harder datasets benefit more. The largest improvements are on Covertype and Credit, which are the most complex datasets (many features, multiple classes, highly imbalanced). This suggests that the CNN architecture is particularly beneficial when there are many feature interactions to capture.

Effect of Differential Privacy.

When DP-SGD is enabled, the ML utility drops by 3–8 F1 points depending on the privacy budget $\epsilon$. The following qualitative relationships hold:

• At $\epsilon =1$: utility drops by approximately 8–12 F1 points relative to the non-private version.

• At $\epsilon =5$: utility drops by approximately 3–5 F1 points.

• At $\epsilon =10$: utility drop is less than 2 F1 points, but the privacy guarantee is weak.

These results confirm the privacy–utility tradeoff discussed in the warning box of Section Differential Privacy Integration (DP-SGD). For applications where strong privacy is required ($\epsilon \le 1$), the utility cost is significant but may be acceptable compared to the alternative of not being able to share the data at all.

A Closer Look at Correlation Preservation.

One of the most informative diagnostic tools for evaluating tabular generative models is the correlation difference matrix. For each pair of features $(i,j)$, we compute the Pearson correlation ${\rho }_{ij}^{\mathrm{real}}$ in the real data and ${\rho }_{ij}^{\mathrm{syn}}$ in the synthetic data, then examine the matrix of differences ${\mathrm{\Delta }}_{ij}=|{\rho }_{ij}^{\mathrm{real}}-{\rho }_{ij}^{\mathrm{syn}}|$.

GOGGLE – Graph-Optimized Generation

Every tabular dataset tells a hidden story about the relationships among its columns. In a medical dataset, blood pressure and age are correlated; cholesterol level depends on diet and exercise frequency; diagnosis is influenced by all of the above. These relationships form a web of dependencies that any faithful generative model must capture. Yet the methods we have studied so far, CTGAN and CTAB-GAN+, treat columns as if they were interchangeable slots in a flat vector. They rely on fully connected layers to implicitly discover column relationships from data, with no explicit mechanism for reasoning about the structure of those relationships.

This is the central insight behind GOGGLE [7]: if columns have a dependency structure, we should model that structure explicitly as a graph, and use a graph neural network (GNN) to generate data that respects the graph. The result is a method that not only generates higher-quality tabular data than previous approaches, but also produces an interpretable dependency graph that domain experts can inspect, validate, and even modify.

Let us begin with a motivating example before diving into the formalism.

Why Graphs? The Column Dependency Problem

To understand why a graph-based approach helps, consider what happens when we use a standard fully connected generator. The generator maps a noise vector $𝒛\in {\mathrm{ℝ}}^d$ to a synthetic row $\widehat{𝒙}\in {\mathrm{ℝ}}^n$ through a sequence of matrix multiplications and nonlinearities. Every output dimension is connected to every input dimension through every hidden layer. This means the generator can, in principle, capture any dependency between any pair of columns.

The problem is that “in principle” and “in practice” are very different things. With $n$ columns, there are $\left(\genfrac{}{}{0px}{}{n}{2}\right)$ pairwise relationships, and the number of higher-order dependencies grows combinatorially. A fully connected network must allocate its limited capacity across all possible relationships, including the many that are absent or weak. The result is that the model wastes capacity on non-relationships and underfits the relationships that matter.

But where does the graph come from? GOGGLE offers two answers:

1. Learned graph: the adjacency matrix is treated as a learnable parameter and optimised jointly with the generation model.

2. Expert-specified graph: a domain expert provides the graph (or constraints on it), and the model generates data consistent with the specified structure.

We will explore both options in detail. First, let us formalise the architecture.

The GOGGLE Architecture

GOGGLE is a variational autoencoder (VAE) augmented with a graph neural network (GNN) that operates on the column-level dependency graph. The architecture has four components:

1. A graph discovery module that learns (or accepts) an adjacency matrix $𝐀\in {[\mathrm{0,1}]}^{n\times n}$ over the $n$ columns.

2. A GNN encoder that computes per-column latent representations by aggregating information from neighbouring columns according to $𝐀$.

3. A standard latent bottleneck where a variational posterior $q_{𝝓}(𝒛|𝒙)$ is fitted.

4. A GNN decoder that reconstructs each column value from the latent representation, again using message passing guided by $𝐀$.

Let us now formalise each component mathematically.

Graph Discovery: Learning the Adjacency Matrix

The most novel aspect of GOGGLE is its ability to discover column relationships automatically. Rather than requiring a pre-specified graph, GOGGLE learns a soft adjacency matrix $𝐀\in {[\mathrm{0,1}]}^{n\times n}$ as a differentiable parameter.

The adjacency matrix is trained jointly with the encoder and decoder parameters. To encourage sparsity (since real column dependencies are typically sparse), GOGGLE adds an ${\ell }_1$ regulariser on $𝐀$: (Sparse)$$\begin{array}{ccc}& \mathrm{\Omega }(𝐀)={\lambda }_{\mathrm{sparse}}\sum _{i=1}^n\sum _{\begin{array}{c}j=1\\ j\ne i\end{array}}^n{A}_{ij},& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${\lambda }_{\mathrm{sparse}}>0$ controls the strength of sparsification. This penalty drives weak connections toward zero, producing a clean, interpretable graph.

GNN Message Passing for Tabular Generation

With the adjacency matrix $𝐀$ in hand, GOGGLE uses graph neural network layers in both the encoder and the decoder. The key idea is message passing: each column's representation is updated by aggregating information from neighbouring columns according to $𝐀$.

The following figure illustrates message passing for a single column node.

Multi-layer message passing.

By stacking $L$ GNN layers, each column can receive information from columns that are up to $L$ hops away in the dependency graph. With $L=1$, column $i$ only sees its direct neighbours. With $L=2$, it also sees the neighbours of its neighbours, and so on. GOGGLE typically uses $L=2$ layers, which provides a good balance between expressiveness and computational cost.

The Encoder-Decoder Architecture

With the message-passing layer defined, we can now describe the full encoder and decoder.

Encoder.

The encoder takes a real data row $𝒙=(x_1,x_2,\dots ,x_n)$ and maps each column value to an initial embedding: (Embed)$$\begin{array}{ccc}& {𝒉}_i^{(0)}={𝐖}_{\mathrm{embed}}^{(i)}{x}_i+{𝒃}_{\mathrm{embed}}^{(i)},i=1,\dots ,n,& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${𝐖}_{\mathrm{embed}}^{(i)}\in {\mathrm{ℝ}}^{d_0\times 1}$ and ${𝒃}_{\mathrm{embed}}^{(i)}\in {\mathrm{ℝ}}^{d_0}$ are per-column embedding parameters. The initial embeddings then pass through $L$ GNN layers ((MP)), yielding final column representations ${𝒉}_1^{(L)},\dots ,{𝒉}_n^{(L)}$.

These per-column representations are concatenated and passed through an MLP to produce the variational parameters: (MU)$$\begin{array}{cccc}& {𝝁}_{𝝓}(𝒙)& ={\mathrm{MLP}}_{\mu }([{𝒉}_1^{(L)};{𝒉}_2^{(L)};\dots ;{𝒉}_n^{(L)}])\in {\mathrm{ℝ}}^{d_z},& \text{<span class="tml-eqn"></span>}\\ & \log {𝝈}_{𝝓}^2(𝒙)& ={\mathrm{MLP}}_{\sigma }([{𝒉}_1^{(L)};{𝒉}_2^{(L)};\dots ;{𝒉}_n^{(L)}])\in {\mathrm{ℝ}}^{d_z},& \text{<span class="tml-eqn"></span>}\end{array}$$ where $d_z$ is the latent dimension. A latent sample is drawn via the reparameterisation trick: (Reparam)$$\begin{array}{ccc}& 𝒛={𝝁}_{𝝓}(𝒙)+{𝝈}_{𝝓}(𝒙)\odot 𝝐,𝝐\sim 𝒩(0,{𝐈}_{d_z}).& \text{<span class="tml-eqn"></span>}\end{array}$$

Decoder.

The decoder maps the latent vector $𝒛$ back to a row reconstruction $\widehat{𝒙}$. It first splits $𝒛$ into per-column seeds via an MLP: (Split)$$\begin{array}{ccc}& [{𝒈}_1^{(0)};{𝒈}_2^{(0)};\dots ;{𝒈}_n^{(0)}]={\mathrm{MLP}}_{\mathrm{split}}(𝒛),& \text{<span class="tml-eqn"></span>}\end{array}$$ where each ${𝒈}_i^{(0)}\in {\mathrm{ℝ}}^{d_0}$. These seeds then pass through $L$ GNN layers (with separate parameters from the encoder but using the same adjacency matrix $𝐀$), producing ${𝒈}_1^{(L)},\dots ,{𝒈}_n^{(L)}$. Finally, per-column output heads produce the reconstructed values: (Output)$$\begin{array}{ccc}& {\hat{x}}_i={\psi }_i({𝐖}_{\mathrm{out}}^{(i)}{𝒈}_i^{(L)}+{𝒃}_{\mathrm{out}}^{(i)}),i=1,\dots ,n,& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${\psi }_i$ is the column-specific activation: identity for continuous columns, softmax for categorical columns.

The GOGGLE Loss Function

GOGGLE's training objective combines three terms: a reconstruction loss, a KL divergence term (as in any VAE), and the sparsity penalty on the adjacency matrix.

The reconstruction loss is computed per-column, using appropriate loss functions for each column type: (Recon)$$\begin{array}{ccc}& {ℒ}_{\mathrm{recon}}=\sum _{i\in {𝒮}_{\mathrm{cont}}}(x_i-{\hat{x}}_i{)}^2+\sum _{i\in {𝒮}_{\mathrm{cat}}}\mathrm{CE}(x_i,{\hat{x}}_i),& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${𝒮}_{\mathrm{cont}}$ and ${𝒮}_{\mathrm{cat}}$ are the sets of continuous and categorical column indices respectively, and $\mathrm{CE}(\cdot ,\cdot )$ denotes the cross-entropy loss.

Human-in-the-Loop: Expert Graph Constraints

One of the most practically valuable features of GOGGLE is its support for domain expert input. In many applications, particularly in healthcare and finance, experts have strong beliefs about which columns should (or should not) be related. GOGGLE allows these beliefs to be encoded as constraints on the adjacency matrix.

There are three types of constraints:

1. Must-link: columns $i$ and $j$ must be connected ($A_{ij}$ is clamped to 1).

2. Must-not-link: columns $i$ and $j$ must not be connected ($A_{ij}$ is clamped to 0).

3. Unconstrained: the model is free to learn the connection strength.

These constraints are implemented by masking the adjacency matrix during training: (MASK)$$\begin{array}{ccc}& A_{ij}=\{\begin{array}{cc}1& \text{if }(i,j)\in {𝒮}_{\mathrm{must}},\\ 0& \text{if }(i,j)\in {𝒮}_{\mathrm{forbid}},\\ \sigma \left(A_{ij}^{\mathrm{raw}}\right)& \text{otherwise},\end{array}& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${𝒮}_{\mathrm{must}}$ and ${𝒮}_{\mathrm{forbid}}$ are the sets of must-link and must-not-link pairs.

Training Algorithm

We now present the complete GOGGLE training procedure.

Sampling (generation).

Once training is complete, new rows are generated by:

1. Sampling $𝒛\sim 𝒩(0,{𝐈}_{d_z})$ from the prior.

2. Passing $𝒛$ through the decoder (with GNN layers and the learned ${𝐀}^{\ast }$) to obtain $\widehat{𝒙}$.

3. Applying post-processing: rounding categorical outputs to the nearest valid category; clipping continuous outputs to valid ranges.

GOGGLE vs. CTGAN: What the Graph Buys You

Having described GOGGLE in detail, let us compare it with the CTGAN approach from CTGAN – The Foundation. The comparison illuminates why explicit graph structure matters.

TabDDPM – Diffusion Models for Tabular Data

The methods we have studied so far, CTGAN, CTAB-GAN+, and GOGGLE, share a common ancestor: they are all based on either GANs or VAEs. These frameworks have served us well, but they come with well-known limitations. GANs suffer from mode collapse and training instability; VAEs tend to produce blurry reconstructions due to the Gaussian decoder assumption. Is there a fundamentally different paradigm for tabular generation?

The answer, as with so many questions in modern machine learning, comes from the world of diffusion models. In 2023, Kotelnikov et al. published TabDDPM [6], demonstrating that denoising diffusion probabilistic models (DDPMs), the same family of models that revolutionised image generation [24], can be adapted to produce state-of-the-art tabular data. The key challenge is that diffusion models, as originally conceived, operate on continuous data (images are arrays of real-valued pixel intensities), while tabular data is a mixture of continuous and categorical features. TabDDPM solves this by combining two parallel diffusion processes: Gaussian diffusion for continuous columns and multinomial diffusion for categorical columns.

The Paradigm Shift: From GANs to Diffusion

Before we dive into the technical details, let us understand why diffusion models represent a fundamentally different approach to generation.

In a GAN, the generator maps noise directly to data in a single forward pass. The challenge is that this mapping must be learned adversarially, with the generator and discriminator locked in a minimax game. The adversarial dynamics create a rich landscape of pathologies: mode collapse (the generator ignores parts of the data distribution), training oscillation (the generator and discriminator chase each other without converging), and sensitivity to hyperparameters (small changes can destabilise training entirely).

Diffusion models take the opposite approach. Instead of learning a single complex mapping, they decompose generation into a long sequence of small, simple steps. The model learns to gradually remove noise from a corrupted data point, one step at a time. Each denoising step is a simple regression problem: given a noisy input, predict the clean version (or equivalently, predict the noise that was added). There is no adversary, no minimax game, no mode collapse. The training objective is a straightforward mean squared error (for continuous data) or cross-entropy (for categorical data).

The following figure contrasts the GAN and diffusion paradigms.

Gaussian Diffusion for Continuous Columns

Let us first describe the diffusion process for continuous features. Suppose we have a single continuous column value $x_0\in \mathrm{ℝ}$ (we omit the column index for clarity; the process is applied independently to each continuous column).

The forward process.

The forward (noising) process gradually corrupts $x_0$ by adding Gaussian noise over $T$ time steps: (FWD)$$\begin{array}{ccc}& q(x_t|x_{t-1})=𝒩(x_t;\sqrt{1-{\beta }_t}{x}_{t-1},{\beta }_t),t=1,\dots ,T,& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${\beta }_1,{\beta }_2,\dots ,{\beta }_T\in (\mathrm{0,1})$ is the noise schedule, a sequence of increasing values that controls how quickly the signal is destroyed.

A crucial property of the forward process is that we can sample $x_t$ directly from $x_0$ without iterating through all intermediate steps. Define ${\alpha }_t=1-{\beta }_t$ and ${\bar{\alpha }}_t={\prod }_{s=1}^t{\alpha }_s$. Then: (Closed)$$\begin{array}{ccc}& q(x_t|x_0)=𝒩(x_t;\sqrt{{\bar{\alpha }}_t}{x}_0,(1-{\bar{\alpha }}_t)).& \text{<span class="tml-eqn"></span>}\end{array}$$

Multinomial Diffusion for Categorical Columns

The Gaussian forward process from the previous subsection works well for continuous features, but categorical features live in a discrete space. We cannot simply add Gaussian noise to a category label. TabDDPM addresses this by using multinomial diffusion [43], a diffusion process designed for categorical data.

Consider a categorical column with $K$ categories. Each value is represented as a one-hot vector ${𝒄}_0\in {\{\mathrm{0,1}\}}^K$ with $\Vert {𝒄}_0{\Vert }_1=1$. The forward process corrupts this one-hot vector by mixing it with a uniform distribution over categories.

As with Gaussian diffusion, we can derive a closed-form expression for the marginal at time $t$: (Closed)$$\begin{array}{ccc}& q({𝒄}_t|{𝒄}_0)=\mathrm{Categorical}({𝒄}_t;{\bar{\alpha }}_t{𝒄}_0+(1-{\bar{\alpha }}_t)\frac{1}{K}),& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${\bar{\alpha }}_t={\prod }_{s=1}^t(1-{\beta }_s)$ as before.

The Reverse Process: Learning to Denoise Mixed Data

The forward processes (Gaussian for continuous, multinomial for categorical) define how clean data is corrupted into noise. The heart of TabDDPM is the reverse process: a neural network that learns to undo the corruption, one step at a time.

Reverse for continuous columns.

For continuous features, the reverse process at each step $t$ predicts the distribution: (CONT)$$\begin{array}{ccc}& p_{𝜽}(x_{t-1}|x_t)=𝒩(x_{t-1};{\mu }_{𝜽}(x_t,t),{\tilde{\beta }}_t),& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${\tilde{\beta }}_t=\frac{1-{\bar{\alpha }}_{t-1}}{1-{\bar{\alpha }}_t}\cdot {\beta }_t$ is the posterior variance (derived from Bayes' rule), and ${\mu }_{𝜽}(x_t,t)$ is the predicted mean, parameterised as: (MU)$$\begin{array}{ccc}& {\mu }_{𝜽}(x_t,t)=\frac{1}{\sqrt{{\alpha }_t}}(x_t-\frac{{\beta }_t}{\sqrt{1-{\bar{\alpha }}_t}}{ϵ}_{𝜽}(x_t,t)),& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${ϵ}_{𝜽}(x_t,t)$ is the noise prediction network. This is the standard DDPM parameterisation [24].

Reverse for categorical columns.

For categorical features, the reverse process predicts the original clean category distribution directly: (CAT)$$\begin{array}{ccc}& p_{𝜽}({𝒄}_{t-1}|{𝒄}_t)\propto q({𝒄}_t|{𝒄}_{t-1})\cdot q({𝒄}_{t-1}|{\widehat{𝒄}}_0({𝒄}_t,t)),& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${\widehat{𝒄}}_0({𝒄}_t,t)$ is the network's prediction of the clean one-hot vector ${𝒄}_0$ given the noisy input ${𝒄}_t$ at time $t$. This is obtained by applying Bayes' rule to the multinomial forward process.

The Denoising Network Architecture

TabDDPM uses a single shared neural network to denoise both continuous and categorical features simultaneously. The architecture is an MLP with residual connections, specifically designed for tabular data (no convolutions, no attention).

The time embedding is computed using sinusoidal positional encoding (the same technique used in Transformers): (Timeemb)$$\begin{array}{ccc}& {𝒆}_t=[\sin (w_{1}t),\cos (w_{1}t),\sin (w_{2}t),\cos (w_{2}t),\dots ,\sin (w_{d_e/2}t),\cos (w_{d_e/2}t)],& \text{<span class="tml-eqn"></span>}\end{array}$$ where $w_k={10000}^{-2k/d_e}$ are fixed frequencies.

The input to the network concatenates all column features (both continuous and categorical) along with the time embedding. The output has two heads:

1. Continuous head: predicts the noise ${\widehat{𝝐}}_{𝜽}$ for continuous columns.

2. Categorical head: predicts the clean one-hot vectors ${\widehat{𝒄}}_{0,𝜽}$ for categorical columns (via softmax).

The Combined Forward Process for Mixed Data

In a real tabular dataset, continuous and categorical columns coexist. TabDDPM handles this by running Gaussian and multinomial diffusion in parallel, sharing the same noise schedule and the same time step $t$.

Let ${𝒙}_0=({𝒙}_0^{\mathrm{cont}},{𝒄}_0^{\mathrm{cat}})$ denote a data row split into its continuous and categorical parts. The combined forward process is: (FWD)$$\begin{array}{ccc}& q({𝒙}_t,{𝒄}_t|{𝒙}_0,{𝒄}_0)=q({𝒙}_t^{\mathrm{cont}}|{𝒙}_0^{\mathrm{cont}})\cdot q({𝒄}_t^{\mathrm{cat}}|{𝒄}_0^{\mathrm{cat}}),& \text{<span class="tml-eqn"></span>}\end{array}$$ where the continuous and categorical noise processes are independent given the clean data. This independence assumption is crucial: it means we can noise each feature type using its appropriate process without worrying about cross-type interactions. The interactions are handled entirely by the shared denoising network, which sees both noisy continuous and noisy categorical features simultaneously.

The ELBO for TabDDPM

As with all diffusion models, TabDDPM is trained by maximising an evidence lower bound (ELBO) on the log-likelihood of the data. The derivation follows the standard DDPM framework but with modifications for the mixed-type setting.

Due to the independence of Gaussian and multinomial noise, the per-step KL divergences decompose further: (Decomp)$$\begin{array}{ccc}& {ℒ}_t=\underset{\text{continuous}}{\underbrace{{ℒ}_t^{\mathrm{cont}}}}+\underset{\text{categorical}}{\underbrace{{ℒ}_t^{\mathrm{cat}}}},& \text{<span class="tml-eqn"></span>}\end{array}$$ where the continuous loss reduces to the standard DDPM noise prediction loss: (CONT)$$\begin{array}{ccc}& {ℒ}_t^{\mathrm{cont}}={𝔼}_{𝝐}[\Vert 𝝐-{\widehat{𝝐}}_{𝜽}({𝒙}_t,{𝒄}_t,t){\Vert }^2],& \text{<span class="tml-eqn"></span>}\end{array}$$ and the categorical loss is the KL divergence between the true and predicted categorical posteriors: (CAT)$$\begin{array}{ccc}& {ℒ}_t^{\mathrm{cat}}=D_{\mathrm{KL}}(q({𝒄}_{t-1}|{𝒄}_t,{𝒄}_0)\Vert p_{𝜽}({𝒄}_{t-1}|{𝒄}_t)).& \text{<span class="tml-eqn"></span>}\end{array}$$

Training Algorithm

Following the simplified training objective of [24], TabDDPM optimises a weighted sum of the per-step losses using uniform time-step sampling.

Sampling (Generation)

After training, synthetic tabular rows are generated by running the reverse process from pure noise:

Why Diffusion Helps: Training Stability and Mode Coverage

Having described the full TabDDPM framework, let us now ask: why does it work so well? The empirical results of [6] show that TabDDPM achieves state-of-the-art performance on a wide range of tabular benchmarks, outperforming CTGAN, TVAE, and CTAB-GAN+ on most datasets. Two properties of diffusion models explain this success.

Training stability.

GAN training is fundamentally a game between two networks, and games can be unstable. The generator tries to fool the discriminator; the discriminator tries to catch the generator. If one player becomes much stronger than the other, training can collapse. Diffusion models eliminate this adversarial dynamic entirely. The training loss is a simple regression objective (MSE for continuous, cross-entropy for categorical), and the optimisation landscape is well-behaved. There is no need for gradient penalty, spectral normalisation, or other regularisation techniques that GANs require for stability.

Mode coverage.

GANs are prone to mode collapse: the generator may learn to produce data from only a subset of the data modes, ignoring the rest. This is particularly problematic for tabular data, where rare categories or unusual feature combinations represent important parts of the distribution (e.g., rare diseases in medical data). Diffusion models avoid this because their training objective is a per-sample denoising loss. Every training sample contributes equally to the loss, regardless of which mode it belongs to. The model cannot improve its loss by ignoring rare samples; it must learn to denoise all of them.

Mathematical Perspective: Continuous-Time Formulation

For the mathematically inclined reader, we briefly describe the continuous-time perspective on TabDDPM. The discrete forward process from (FWD) can be viewed as a discretisation of a stochastic differential equation (SDE).

The reverse process is given by Anderson's theorem: (REV)$$\begin{array}{ccc}& \mathrm{d}x=[-\frac{1}{2}\beta (t)x-\beta (t){\nabla }_x\log p_t(x)]\mathrm{d}t+\sqrt{\beta (t)}\mathrm{d}{\bar{W}}_t,& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${\bar{W}}_t$ is a reverse-time Wiener process and ${\nabla }_x\log p_t(x)$ is the score function at time $t$. The denoising network ${ϵ}_{𝜽}(x_t,t)$ provides an estimate of the score via the relation: (Score)$$\begin{array}{ccc}& {\nabla }_x\log p_t(x_t)\approx -\frac{{ϵ}_{𝜽}(x_t,t)}{\sqrt{1-{\bar{\alpha }}_t}}.& \text{<span class="tml-eqn"></span>}\end{array}$$

Practical Considerations and Hyperparameters

Implementing TabDDPM involves several practical decisions that significantly affect performance.

Data preprocessing.

Continuous features should be normalised to have zero mean and unit variance before diffusion. This ensures that the noise schedule is appropriate for all features. Categorical features are one-hot encoded. Missing values, if present, can be handled by introducing an additional “missing” category for each categorical column or by imputing continuous values before training.

Number of diffusion steps.

The original TabDDPM paper uses $T=1000$ steps, following the standard DDPM convention. However, tabular data is typically lower-dimensional than image data, and fewer steps (e.g., $T=100$ or even $T=50$) may suffice. The optimal $T$ depends on the complexity of the dataset; it should be tuned on a validation set.

Network width and depth.

The denoising MLP typically uses 3–4 residual blocks with hidden dimensions of 256 or 512. Larger networks are rarely needed for tabular data; the bottleneck is usually data quality and quantity, not model capacity.

Categorical weighting.

The relative weighting $\lambda$ between the continuous and categorical losses (in alg:tabddpm:train, line 8) is an important hyperparameter. If $\lambda$ is too small, the model neglects categorical features; if too large, it neglects continuous features. Cross-validation is recommended.

Comparison with GAN-Based Methods

The following table summarises the key differences between TabDDPM and the GAN-based methods we have studied.

Exercises

A Worked Example: Two Columns, Five Steps

To solidify understanding, let us walk through a complete example of TabDDPM with a tiny dataset having one continuous column (“income”, normalised) and one categorical column (“status” with $K=3$: employed, unemployed, retired).

Setup.

Consider a single data row: $x_0^{\mathrm{cont}}=0.7$ (normalised income), ${𝒄}_0^{\mathrm{cat}}=(1,0,0)$ (employed). We use $T=5$ steps with ${\beta }_t=0.1t$ for simplicity (so ${\beta }_1=0.1$, ${\beta }_2=0.2$, $\dots$, ${\beta }_5=0.5$).

Forward process.

The cumulative products are: ${\bar{\alpha }}_1=0.9$, ${\bar{\alpha }}_2=0.72$, ${\bar{\alpha }}_3=0.504$, ${\bar{\alpha }}_4=0.302$, ${\bar{\alpha }}_5=0.151$.

At $t=3$, the continuous feature has distribution $𝒩(\sqrt{0.504}\cdot 0.7,1-0.504)=𝒩(0.497,0.496)$, meaning the signal is already heavily corrupted. The categorical feature has distribution $0.504\cdot (\mathrm{1,0,0})+0.496\cdot (\frac{1}{3},\frac{1}{3},\frac{1}{3})=(0.669,0.165,0.165)$: the true category still has the highest probability, but uncertainty is substantial.

At $t=5$, ${\bar{\alpha }}_5=0.151$, so continuous features are nearly pure noise ($𝒩(0.272,0.849)$) and categorical probabilities are $(0.317,0.341,0.341)$, approximately uniform.

Reverse process.

Starting from noise at $t=5$, the denoising network predicts the noise for continuous columns and the clean category for categorical columns at each step. Suppose at $t=3$, the network predicts $\widehat{ϵ}=-0.4$ for income and ${\widehat{𝒄}}_0=(0.92,0.05,0.03)$ for status. Using (MU), the denoised continuous value moves closer to the true value. Using (Catpost), the categorical posterior concentrates on “employed”. After all five reverse steps, we obtain a synthetic row that approximates the original data distribution.

Harpoon – Manifold-Guided Diffusion for Tabular Data

Imagine the following scenario. You have trained a diffusion model on a medical dataset containing patient demographics, lab results, and outcomes. The model generates realistic synthetic patients, and you are pleased with the quality. Then a colleague walks in and says: “I need synthetic patients who are female, over 65, with a diabetes diagnosis. Can you generate those?”

Your heart sinks. The diffusion model you trained is unconditional. It generates patients from the full joint distribution, not from the conditional distribution $p(𝒙|\text{female, over 65, diabetes})$. The standard remedy is to retrain the model with classifier-free guidance, conditioning on the desired attributes during training. But retraining is expensive, and moreover, your colleague's conditioning requirements will change tomorrow. Perhaps she will want patients who satisfy a fairness constraint, or patients whose age falls in a specific range, or patients who match a partially observed record (missing data imputation). You cannot retrain for every conceivable condition.

This is the problem that Harpoon solves. Proposed by Liu, Seedat, Imrie, and van der Schaar [44], Harpoon enables zero-shot conditional generation from a pre-trained unconditional diffusion model. The key idea is elegant: during the reverse diffusion process, Harpoon projects intermediate samples onto a data manifold defined by the desired condition, using the manifold structure itself to guide the denoising trajectory toward the conditional distribution. No retraining, no auxiliary classifiers, no modifications to the original model.

Let us build up to Harpoon's algorithm step by step, starting with the fundamental challenge and progressing through the mathematical machinery.

The Challenge: Conditional Generation Without Retraining

To appreciate Harpoon's contribution, we must first understand why conditional generation from an unconditional model is hard. Recall from the theory of score-based generative models [45] that a diffusion model learns the score function: (Score)$$\begin{array}{ccc}& {𝒔}_{𝜽}({𝒙}_t,t)\approx {\nabla }_{{𝒙}_t}\log p_t({𝒙}_t),& \text{<span class="tml-eqn"></span>}\end{array}$$ where $p_t$ is the marginal distribution of the noised data at time $t$. Sampling proceeds by solving the reverse-time stochastic differential equation (SDE): (Reverse SDE)$$\begin{array}{ccc}& \mathrm{d}{𝒙}_t=[𝒇({𝒙}_t,t)-g(t{)}^2{𝒔}_{𝜽}({𝒙}_t,t)]\mathrm{d}t+g(t)\mathrm{d}{\overline{𝒘}}_t,& \text{<span class="tml-eqn"></span>}\end{array}$$ where $𝒇$ is the drift, $g$ is the diffusion coefficient, and ${\overline{𝒘}}_t$ is a reverse-time Wiener process.

Now suppose we want to sample from the conditional distribution $p(𝒙|c)$, where $c$ represents some condition (a set of feature values, a fairness constraint, etc.). Bayes' rule gives us the conditional score: (COND Score)$$\begin{array}{ccc}& {\nabla }_{{𝒙}_t}\log p_t({𝒙}_t|c)={\nabla }_{{𝒙}_t}\log p_t({𝒙}_t)+{\nabla }_{{𝒙}_t}\log p_t(c|{𝒙}_t).& \text{<span class="tml-eqn"></span>}\end{array}$$

The first term on the right is precisely what our unconditional model provides. The second term, ${\nabla }_{{𝒙}_t}\log p_t(c|{𝒙}_t)$, is the guidance term. In classifier guidance [46], this term comes from a separately trained classifier. In classifier-free guidance, it is implicitly learned during training. Both approaches require training auxiliary components.

Harpoon's insight is that for many conditions of practical interest, the guidance term can be computed geometrically, without any learned classifier, by exploiting the manifold structure of the condition set.

Manifold Geometry for Conditions

The key geometric idea behind Harpoon is that many conditions of practical interest define a smooth manifold (or a reasonable approximation thereof) in the data space.

Let us see how familiar conditions fit this framework.

Fixed feature values.

Suppose we want to generate data with $x_j=v_j$ for $j\in S$, where $S$ is a subset of feature indices. The condition function is $c_j(𝒙)=x_j-v_j$ for each $j\in S$. The condition manifold is a $(d-|S|)$-dimensional hyperplane.

Range constraints.

For constraints like $a\le x_j\le b$, the condition manifold is a slab in ${\mathrm{ℝ}}^d$. While slabs have boundaries (they are manifolds with boundary), the interior is smooth and amenable to projection.

Fairness constraints.

For demographic parity, we might require that the mean outcome across protected groups be equal: $c(𝒙)=𝔼[y|A=0]-𝔼[y|A=1]$, where $A$ is the protected attribute. In practice, this becomes a constraint on the generated batch, and the condition manifold has a natural geometric interpretation.

The following figure illustrates the concept of a condition manifold and the projection operation.

The Projection Operator

Central to Harpoon is the manifold projection operator, which maps an arbitrary point in ${\mathrm{ℝ}}^d$ to the nearest point on the condition manifold ${ℳ}_c$.

For the common case where the condition is a set of fixed feature values, the projection takes an especially simple form.

For more complex conditions (nonlinear constraints, fairness conditions on batches), the projection may require iterative optimisation. Harpoon handles this through a general-purpose iterative projection sub-routine.

Manifold Guidance: The Core Algorithm

With the projection operator in hand, we can now describe Harpoon's full guided diffusion procedure. The idea is beautifully simple: at each step of the reverse diffusion process, after the unconditional denoising update, we project the result onto the condition manifold.

But there is a subtlety. The condition manifold ${ℳ}_c$ is defined in the clean data space ${\mathrm{ℝ}}^d$, while the diffusion process operates in the noised data space at time $t$. What does it mean to project a noised sample onto a condition manifold?

Harpoon resolves this with a two-step procedure at each diffusion step:

1. Denoise: Estimate the clean data point from the current noised sample using the Tweedie estimate: (Tweedie)$$\begin{array}{ccc}& {\widehat{𝒙}}_0({𝒙}_t)=\frac{1}{\sqrt{{\bar{\alpha }}_t}}({𝒙}_t+(1-{\bar{\alpha }}_t){𝒔}_{𝜽}({𝒙}_t,t)).& \text{<span class="tml-eqn"></span>}\end{array}$$

2. Project: Apply the manifold projection to the estimated clean data point: (Project Clean)$$\begin{array}{ccc}& {\widetilde{𝒙}}_0={\mathrm{\Pi }}_c({\widehat{𝒙}}_0({𝒙}_t)).& \text{<span class="tml-eqn"></span>}\end{array}$$

3. Re-noise: Map the projected clean estimate back to the noise level at the next time step $t-\mathrm{\Delta }t$: (Renoise)$$\begin{array}{ccc}& {𝒙}_{t-\mathrm{\Delta }t}=\sqrt{{\bar{\alpha }}_{t-\mathrm{\Delta }t}}{\widetilde{𝒙}}_0+\sqrt{1-{\bar{\alpha }}_{t-\mathrm{\Delta }t}}𝝐,𝝐\sim 𝒩(0,𝐈).& \text{<span class="tml-eqn"></span>}\end{array}$$

The following figure illustrates how this three-step procedure guides the diffusion trajectory toward the condition manifold, compared with unguided diffusion.

Now let us state the full Harpoon algorithm precisely.

Mathematical Foundations: Guidance as Score Correction

The Harpoon algorithm may appear ad hoc at first glance: why should projecting a Tweedie estimate onto a condition manifold produce valid samples from the conditional distribution? In this subsection, we develop the theoretical justification, showing that manifold projection is a principled approximation to the exact conditional score.

This theorem has an important corollary that characterises Harpoon's convergence.

Handling Different Condition Types

One of Harpoon's most appealing properties is its flexibility: the framework handles a wide variety of conditions through appropriate choices of the condition function $c$ and its associated projection operator ${\mathrm{\Pi }}_c$. Let us catalogue the most important cases.

The following figure shows the conditional generation pipeline for different condition types.

Application: Missing Data Imputation

One of Harpoon's most natural applications is missing data imputation. Given a partially observed row ${𝒙}_{\mathrm{obs}}$ with missing entries indexed by $M\subseteq \{1,\dots ,d\}$, we want to sample from the conditional distribution $p({𝒙}_M|{𝒙}_{\bar{M}}={𝒙}_{\mathrm{obs},\bar{M}})$, where $\bar{M}$ is the complement of $M$.

This is precisely a hard conditioning problem. The condition function is: (Imputation)$$\begin{array}{ccc}& c(𝒙)={𝐏}_{\bar{M}}𝒙-{𝒙}_{\mathrm{obs},\bar{M}},& \text{<span class="tml-eqn"></span>}\end{array}$$ and the projection simply replaces the observed features while letting the diffusion model “hallucinate” the missing ones.

Application: Fairness-Constrained Generation

A particularly compelling application of Harpoon is generating synthetic data that satisfies fairness constraints. This is a batch-level conditioning problem: we want the generated batch to exhibit demographic parity or equal opportunity across protected groups, without modifying the base model.

The geometry of this constraint is illuminating. The condition manifold is a hyperplane in ${\mathrm{ℝ}}^{nd}$ (the space of all $n$ generated samples), and the projection amounts to adjusting the outcome values to equalize group means.

Practical Considerations and Limitations

Before leaving Harpoon, let us discuss several practical considerations that affect its deployment.

Choice of guidance strength.

The parameter $\lambda$ controls how aggressively the diffusion path is guided toward the condition manifold. Too small, and the generated samples may violate the condition; too large, and the samples lose diversity by collapsing toward the nearest point on the manifold. A practical heuristic is to start with $\lambda =1.0$ and adjust based on constraint violation rates.

Discrete features.

When the condition involves discrete features (e.g., “diagnosis = diabetes”), the projection must handle the discrete nature of the feature. Harpoon operates in the continuous embedding space used by the diffusion model and applies a rounding step after the final diffusion step to map continuous embeddings back to discrete categories.

Scalability.

For batch-level conditions (like fairness), the condition function $c$ and its gradient are computed over the entire batch, which can be expensive for large batches. However, since tabular datasets typically have moderate dimensionality (tens to hundreds of features), the computational overhead is manageable for batch sizes in the thousands.

Comparison with alternatives.

How does Harpoon compare with other approaches to zero-shot conditional generation? Replacement-based methods (which simply replace the conditioned features at each diffusion step without the Tweedie estimate) can produce incoherent results because they do not account for the correlation structure between features. Classifier-based guidance requires training a separate classifier for each condition. Harpoon occupies a sweet spot: it requires no auxiliary training and naturally handles the correlation structure through the unconditional model's learned score function.

WaveStitch – Time Series Generation

Time series data is everywhere. Every heartbeat recorded by a wearable sensor, every stock price tick on an exchange, every temperature reading from a weather station, every packet latency measurement in a network, produces a sequence of values indexed by time. And in every one of these domains, practitioners face the same desperate need for more data: more labelled examples for anomaly detection, more diverse scenarios for stress testing, more realistic simulations for policy evaluation.

Generating synthetic time series is, in principle, just another instance of the generative modelling problem. We have a distribution $p(𝒙)$ over sequences $𝒙=(x_1,x_2,\dots ,x_T)$, and we want to sample new sequences from a model $p_{𝜽}(𝒙)$ that approximates $p$. But in practice, time series generation is far harder than generating tabular rows or images, for reasons that we shall explore in depth.

The method we study in this section, WaveStitch, tackles one of the most challenging aspects of time series generation: producing long synthetic sequences efficiently while maintaining temporal coherence [47]. Its approach, a pipelined-parallel architecture with a novel stitching mechanism, represents a significant departure from the autoregressive and diffusion-based approaches we have encountered earlier.

Why Time Series Generation is Hard

Before diving into WaveStitch's solution, let us catalogue the challenges that make time series generation uniquely difficult. Understanding these challenges will help us appreciate why WaveStitch's design choices are necessary.

Long-range temporal dependencies.

A stock price at time $t$ depends not just on the price at $t-1$ but on patterns spanning days, weeks, and months. A patient's heart rate during exercise depends on the entire trajectory of exertion, not just the most recent measurement. Capturing these long-range dependencies requires models with long memory, which is computationally expensive and prone to error accumulation.

Multi-scale structure.

Real time series exhibit structure at multiple temporal scales simultaneously. An electrocardiogram (ECG) has the sub-second structure of individual heartbeats, the minute-scale structure of heart rate variability, and the hour-scale structure of circadian rhythms. A generative model must capture all scales coherently.

Non-stationarity.

Unlike images, which have (approximately) translation-invariant statistics, time series often have statistics that change over time. Financial markets have different volatility regimes. Physiological signals change with the patient's state. A model trained on one regime may produce nonsensical outputs when the regime shifts.

Periodicity and quasi-periodicity.

Many time series have periodic components (daily cycles, seasonal patterns) that must be reproduced faithfully. Slight errors in the period can compound over long sequences, leading to phase drift that destroys the temporal structure.

Length scalability.

Practitioners often need synthetic sequences that are much longer than what the model was trained on. A model trained on 256-step windows must generate sequences of 10,000 or 100,000 steps. Autoregressive generation suffers from error accumulation over long horizons; generating the entire sequence at once is computationally prohibitive.

Why Naive Concatenation Fails

The most obvious approach to generating long time series is to generate short segments independently and concatenate them. If our model can produce realistic 256-step windows, why not generate forty windows and concatenate them into a 10,240-step sequence?

The answer is that the boundaries between segments create artefacts that destroy temporal coherence. Let us see why.

Suppose we generate two independent segments: $$\begin{array}{cccc}& {𝒔}_1& =(s_{\mathrm{1,1}},s_{\mathrm{1,2}},\dots ,s_{1,L}),& \text{<span class="tml-eqn"></span>}\\ & {𝒔}_2& =(s_{\mathrm{2,1}},s_{\mathrm{2,2}},\dots ,s_{2,L}).& \text{<span class="tml-eqn"></span>}\end{array}$$ Naive concatenation produces $𝒔=({𝒔}_1,{𝒔}_2)$. At the boundary, we have $s_{1,L}$ followed by $s_{\mathrm{2,1}}$. These two values were generated independently, so there is no reason for $s_{\mathrm{2,1}}$ to be consistent with the trajectory ending at $s_{1,L}$.

The discontinuity is not just a matter of value mismatch; it manifests in several ways:

1. Level shifts: The mean of ${𝒔}_2$ may differ from the mean of ${𝒔}_1$, creating a visible jump.

2. Slope discontinuities: Even if the values match at the boundary, the derivatives (slopes) typically do not, creating a “kink” that is immediately detectable.

3. Spectral artefacts: The discontinuity introduces high-frequency energy that was not present in the original signal, corrupting the spectral characteristics.

4. Phase breaks: For periodic signals, the phase at the start of ${𝒔}_2$ has no relation to the phase at the end of ${𝒔}_1$, destroying the periodicity.

The WaveStitch Architecture

WaveStitch addresses the length scalability problem through two key innovations: a pipelined-parallel generation scheme that enables efficient production of long sequences, and a stitching mechanism that ensures seamless transitions between segments.

The high-level idea is as follows. Instead of generating one long sequence or generating independent segments, WaveStitch generates overlapping segments in parallel, where each segment shares a region of overlap with its neighbours. The shared regions are then blended using a carefully designed stitching function that preserves temporal coherence.

Pipelined-parallel generation.

The key efficiency insight is that segments need not be generated sequentially. WaveStitch uses a pipeline structure: once the overlap region of segment $m$ has been generated, segment $m+1$ can begin generating using this overlap as its prefix context. This is analogous to pipeline parallelism in computer architecture, where multiple stages of computation proceed simultaneously.

The Stitching Mechanism

The heart of WaveStitch is its stitching mechanism, which combines the overlapping regions of adjacent segments into a seamless transition. The stitching must satisfy two competing requirements:

1. Temporal coherence: The transition should be smooth, with no discontinuities in value, derivative, or spectral content.

2. Distribution preservation: The stitched region should look like it came from the same distribution as the non-overlapping regions. The stitching should not introduce artefacts that a discriminator could detect.

The choice of blending weight function $w$ is critical. WaveStitch uses a raised cosine window, which has several desirable properties.

The stitching mechanism is visualised in fig:wavestitch:stitch.

Spectral Consistency

A subtle but crucial requirement for time series generation is spectral consistency: the power spectral density (PSD) of the generated sequence should match that of real sequences. Stitching artefacts, even when invisible in the time domain, can introduce spurious frequency components that are easily detected by spectral analysis.

Cyclic Encoding for Temporal Features

Many time series carry auxiliary temporal features that exhibit cyclic structure: hour of day, day of week, month of year. These features are crucial for capturing periodicity (e.g., weekday vs. weekend traffic patterns, seasonal demand cycles), but representing them is surprisingly tricky.

The naive encoding problem.

If we encode “hour of day” as an integer from 0 to 23, the model sees hour 23 and hour 0 as maximally distant, when in fact they are one hour apart. This destroys the cyclic structure.

One-hot encoding problem.

If we one-hot encode the hours, we get a 24-dimensional binary vector that ignores the ordering and proximity of hours.

WaveStitch solves this with a cyclic encoding based on sine and cosine transformations.

This encoding has several key properties:

The Complete WaveStitch Algorithm

We now assemble all components into the complete WaveStitch generation procedure.

Mathematical Analysis: The Overlap-and-Blend Operator

Let us formalise the overlap-and-blend operation and analyse its properties more carefully.

The partition-of-unity structure guarantees that the overlap-and-blend operator is a valid averaging scheme: it never amplifies or attenuates the signal, only smoothly interpolates between adjacent segments.

Conditional Generation for Time Series

WaveStitch's pipelined architecture naturally supports several forms of conditional generation that are particularly relevant for time series.

Prefix conditioning.

Given an observed prefix $(x_1,\dots ,x_k)$, WaveStitch can generate the continuation by using the prefix as the context for the first segment. This is forecasting, and WaveStitch's segment-based approach handles it seamlessly: the prefix simply replaces the first $k$ generated values.

Infilling.

Given observations at scattered time points, WaveStitch can fill in the gaps. Each observed value acts as a constraint on the corresponding segment. The stitching mechanism ensures that the infilled values transition smoothly between observed anchors.

Style transfer.

By conditioning the cyclic encodings on a different temporal pattern (e.g., shifting from weekday to weekend patterns), WaveStitch can transform a time series from one “style” to another while preserving the local dynamics.

Applications

WaveStitch has been applied to several domains where long, coherent synthetic time series are needed.

Healthcare monitoring.

Continuous patient monitoring generates streams of physiological data: heart rate, blood pressure, oxygen saturation, respiratory rate. Training anomaly detection models requires diverse examples of both normal and abnormal patterns, but real abnormal data is scarce. WaveStitch generates long synthetic monitoring streams that exhibit realistic transitions between normal and abnormal states, providing training data for classifiers.

Financial data.

Stress testing of financial models requires synthetic market data that exhibits realistic statistical properties (fat tails, volatility clustering, mean reversion) over long horizons. WaveStitch generates multi-thousand-step price paths that maintain these properties, enabling Monte Carlo simulations that would otherwise require prohibitively expensive real data collection.

Sensor data for IoT.

Internet-of-Things (IoT) deployments generate vast quantities of sensor data, but labelled data for fault detection is rare. WaveStitch generates synthetic sensor streams with injected anomalies, creating training datasets for predictive maintenance systems.

Energy systems.

Smart grid planning requires realistic load profiles spanning months or years. WaveStitch generates synthetic load curves that capture daily and seasonal patterns through its cyclic encoding, enabling planners to test demand response strategies on diverse scenarios.

Comparison with Alternative Approaches

How does WaveStitch compare with other approaches to long time series generation?

Autoregressive models.

Pure autoregressive generation (predicting one step at a time) suffers from error accumulation: small errors in early steps compound, causing the generated sequence to diverge from realistic trajectories. WaveStitch avoids this by generating segments in parallel, so errors do not propagate across the full sequence length.

Full-sequence diffusion.

Generating the entire sequence at once with a diffusion model ensures global coherence but requires the model to handle the full sequence length, which is computationally prohibitive for sequences of 10,000+ steps. WaveStitch achieves near-global coherence with computational cost that scales linearly with sequence length.

Simple windowed generation.

Generating non-overlapping windows and concatenating them (the naive approach from Why Naive Concatenation Fails) is fast but produces boundary artefacts. WaveStitch's overlap-and-blend mechanism eliminates these artefacts at modest additional cost.

Chronos and foundation models.

Recent foundation models for time series (like Chronos [48]) aim to learn universal representations. WaveStitch is complementary: it can use any base generator (including foundation models) for individual segments and adds the stitching layer on top.

Practical Implementation Details

Several practical details affect WaveStitch's performance.

Overlap length selection.

The overlap $\omega$ should be chosen based on the autocorrelation structure of the data. For highly correlated processes (like smooth physiological signals), shorter overlaps suffice because the stitching artefacts are small. For processes with complex temporal structure (like financial data with regime changes), longer overlaps are needed to ensure the context carries enough information for coherent continuation.

Base model architecture.

WaveStitch is agnostic to the base generator architecture. The base model can be a diffusion model, a GAN, a VAE, or any model that can generate fixed-length time series segments conditioned on a prefix. The choice of base model affects the quality of individual segments but not the stitching mechanism.

Multi-channel time series.

For multivariate time series (multiple channels recorded simultaneously), WaveStitch applies the same stitching mechanism to all channels simultaneously, using the same blending weights. This preserves cross-channel correlations within the overlap region.

Handling distribution shift.

When generating very long sequences, the statistical properties of the data may change over time (non-stationarity). WaveStitch handles this by allowing the cyclic encodings to carry information about the current regime, enabling the base model to adapt its generation to the current temporal context.

LLM-Based Tabular Generation – TabuLa

What if we could simply tell a language model to “write” a new patient record, the way we ask it to write an essay? The question sounds absurd at first. Language models are trained on books, articles, and web pages – rivers of prose – not on the orderly rows and columns of a database table. A patient record is not a sentence. It is a vector of heterogeneous features: age (continuous), blood pressure (continuous), diagnosis code (categorical), insurance type (categorical), admission date (temporal). These features live in vastly different spaces, obey complex joint distributions, and are bound by domain constraints that no amount of next-token prediction would seem to capture.

And yet, a remarkable line of research beginning around 2023 has shown that the intuition is not absurd at all. It is, in fact, correct. Language models can generate high-quality tabular data, sometimes surpassing purpose-built architectures like CTGAN and TabDDPM, provided we solve one key problem: how to represent a table row as a string that a language model can understand. This is the serialization insight, and it is the foundation of a method called TabuLa – Tabular generation via Language models [25].

The idea is disarmingly simple. Convert each row of a table into a natural-language sentence. Fine-tune a large language model on these sentences. Then sample new sentences from the fine-tuned model and parse them back into table rows. The simplicity is deceptive: behind it lies a rich set of design decisions about serialization format, padding strategy, sampling temperature, and post-processing that determine whether the method produces realistic data or incoherent gibberish.

The Serialization Insight

The core insight of LLM-based tabular generation is that every table row can be faithfully represented as a text string, and that language models are extraordinarily good at learning the statistical structure of text strings. If we can encode the joint distribution $p(x_1,x_2,\dots ,x_d)$ over $d$ features as a distribution over strings, then the language model's next-token prediction objective automatically learns the joint distribution.

Consider a simple table with three columns: age, income, and occupation. A row $(45,50000,\text{engineer})$ can be serialized as the string:

This is the feature-value serialization format: each column name is followed by its value, and columns are separated by commas. The language model sees this as a perfectly natural English sentence (albeit a boring one) and can model the conditional distributions: $$p(\text{𝚒𝚗𝚌𝚘𝚖𝚎}=50000|\text{𝚊𝚐𝚎}=45)$$ simply by learning which tokens tend to follow which.

Why does this work? Three reasons converge:

1. Pre-trained world knowledge. A language model trained on the internet has encountered millions of sentences like “the patient is 45 years old and earns $50,000.” It already possesses statistical associations between age, income, and occupation. Fine-tuning on a specific dataset sharpens these associations but does not need to learn them from scratch.

2. Autoregressive factorization. The language model factors the joint distribution as: (Autoregressive)$$\begin{array}{ccc}& p(\text{row})=\prod _{t=1}^{T}p({\text{token}}_t|{\text{token}}_{1:t-1}),& \text{<span class="tml-eqn"></span>}\end{array}$$ which naturally captures dependencies between features, including higher-order interactions that methods like CTGAN (which use a single noise vector) may miss.

3. Flexible types. Text is a universal representation. Continuous values, categorical labels, dates, free-text fields, and even missing values (serialized as “unknown” or “N/A”) all live in the same token space. No special encoding is needed for mixed types.

The invertibility requirement is crucial. If two different rows produce the same string, the language model cannot distinguish them, and the generated data will conflate distinct records. In practice, feature-value serialization is injective for all standard data types (integers, floats with fixed precision, categorical labels, dates), so invertibility holds.

Alternative serialization formats have been explored. Some use JSON format ("age": 45, "income": 50000), some use comma-separated values without column names, and some use template-based natural language (“A 45-year-old engineer in Boston earns $50,000”). The feature-value format strikes a balance between informativeness (column names provide context) and compactness (minimal overhead tokens).

TabuLa Architecture

TabuLa builds on a pre-trained large language model – specifically, LLaMA [49] – and fine-tunes it on serialized tabular data. The architecture itself is unchanged: a standard Transformer decoder with causal attention. The innovation lies entirely in the data pipeline: how rows are serialized, padded, and presented to the model.

The training procedure is straightforward in outline:

1. Serialize every row in the training table to a string using feature-value format.

2. Optionally shuffle the column order to augment data and prevent the model from relying on a fixed ordering.

3. Tokenize each string using the LLaMA tokenizer.

4. Apply middle padding to equalize sequence lengths.

5. Fine-tune the model using the standard causal language modelling loss: (LOSS)$$\begin{array}{ccc}& {ℒ}_{\text{TabuLa}}=-\frac{1}{T}\sum _{t=1}^T\log p_{\theta }({\text{token}}_t|{\text{token}}_{1:t-1}),& \text{<span class="tml-eqn"></span>}\end{array}$$ where $\theta$ denotes the model parameters and $T$ is the padded sequence length.

The column shuffling step deserves emphasis. In a standard table, columns have a fixed order (e.g., age before income before occupation). But the joint distribution $p(x_1,x_2,\dots ,x_d)$ is permutation-invariant: the probability of a particular combination of feature values does not depend on the order in which we list them. By randomly permuting the column order during serialization, we encourage the model to learn the true joint distribution rather than a particular autoregressive factorization. This is analogous to the order agnostic training used in XLNet [50].

Why middle padding rather than the more conventional left padding (prepend pads) or right padding (append pads)? The reasoning is subtle and relates to how causal Transformers allocate attention.

Sampling Strategies

Once the model is fine-tuned, generating new rows is a matter of sampling from the autoregressive distribution. But how we sample matters enormously. The standard toolkit of decoding strategies – temperature scaling, top-$k$ truncation, and nucleus (top-$p$) sampling – all apply, but their effects on tabular data are qualitatively different from their effects on natural language.

In natural language generation, increasing the temperature $\tau$ makes text more creative but potentially less coherent. In tabular generation, the stakes are different:

• Low temperature ($\tau \to 0$) produces near- deterministic outputs: the model repeats the most common rows in the training data. The generated distribution collapses to a few modes, destroying diversity and underrepresenting minority subgroups.

• High temperature ($\tau \to \infty$) produces highly diverse outputs but may generate nonsensical feature combinations: a 5-year-old with a PhD, or a blood pressure of $-30$. The generated distribution spreads too broadly, violating domain constraints.

• Moderate temperature ($\tau \approx 0.7$–$1.0$) typically balances diversity and validity, but the optimal value is dataset-dependent and must be tuned.

Formally, temperature scaling modifies the softmax distribution over the vocabulary at each token position: (Temperature)$$\begin{array}{ccc}& p_{\tau }({\text{token}}_t=w|{\text{token}}_{1:t-1})=\frac{\exp ({\ell }_w/\tau )}{{\sum }_{w^{\prime }}\exp ({\ell }_{w^{\prime }}/\tau )},& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${\ell }_w$ is the logit for token $w$ and $\tau >0$ is the temperature.

The parsing challenge is the other face of sampling. The language model produces a string, and we must convert it back to a valid table row. This requires:

1. Format parsing. The string must match the expected serialization format: column names in the right order, “is” separators, comma delimiters.

2. Type parsing. Each value must be parseable to its expected type: “45” to integer 45, “50000” to float 50000.0, “engineer” to the categorical label “engineer.”

3. Range checking. Parsed values must fall within valid ranges: age $\in [0,120]$, blood pressure $\in [0,300]$, occupation $\in \{\text{valid labels}\}$.

Rows that fail any parsing step are discarded (rejection sampling). The acceptance rate – the fraction of generated strings that survive parsing – is a key efficiency metric. At low temperatures, acceptance rates are typically 95–99%. At high temperatures, they can drop below 50%.

Few-Shot Tabular Generation

Perhaps the most remarkable capability of LLM-based tabular generation is few-shot generation: the ability to produce realistic synthetic data from just a handful of example rows, without any fine-tuning.

The mechanism is in-context learning. We provide the language model with a prompt that includes $k$ example rows (typically $k=3$–$10$) and ask it to generate more rows in the same format. The model uses its pre-trained knowledge to infer the statistical structure of the table from these examples and extrapolates.

The quality of few-shot generation depends critically on several factors:

• Number of examples. More examples generally improve quality, but with diminishing returns. Five to ten examples often suffice for simple distributions; complex multimodal distributions may require more.

• Representative coverage. The examples should cover the diversity of the data. If all $k$ examples come from the same subgroup (e.g., only male patients), the model will undersample other subgroups.

• Model size. Larger models exhibit stronger few-shot capabilities. A 7B-parameter model significantly outperforms a 350M-parameter model on few-shot tabular tasks.

• Pre-training data. If the model has encountered similar tables during pre-training (e.g., medical records on the web), few-shot quality is dramatically better. This is both an advantage (leveraging world knowledge) and a risk (data leakage from pre-training).

Advantages and Limitations

The LLM approach to tabular generation brings several decisive advantages over specialized methods:

1. Universal type handling. Because everything is text, TabuLa handles continuous, categorical, ordinal, datetime, free-text, and even nested features without any special preprocessing. Methods like CTGAN require separate encoding for each type.

2. Few-shot capability. No other tabular generation method can produce reasonable outputs from five example rows. GAN-based and diffusion-based methods require hundreds to thousands of training examples.

3. World knowledge. The pre-trained model knows that doctors earn more than students, that blood pressure increases with age, and that “New York” is a city. This semantic knowledge improves the realism of generated data, especially for small training sets.

4. Flexibility. The same model can generate data for any table, in any domain, without architectural changes. Switching from healthcare to finance to ecology requires only new training data.

5. Conditional generation. Specifying constraints like “generate a row where age is 30 and occupation is teacher” is trivial: simply include the constraints in the prompt.

But the approach also carries significant limitations:

1. Computational cost. Fine-tuning a 7B-parameter model is orders of magnitude more expensive than training a CTGAN (which has $\sim$1M parameters). Generating each row requires an autoregressive pass through the entire model.

2. Token budget. Tables with many columns ($d>50$) or long text fields may exceed the model's context window. A row with 100 features might require 500+ tokens, leaving little room for few-shot examples.

3. Numerical precision. Language models process numbers as sequences of digit tokens. The value 3.14159 is five tokens (“3”, “.”, “14”, “15”, “9”), and the model may not capture precise numerical relationships. Rounding or binning continuous values helps but loses information.

4. Hallucination. The model's world knowledge is a double-edged sword. It may “hallucinate” feature combinations that are plausible in general but impossible in the specific dataset (e.g., generating a diagnosis code that does not exist in the hospital's coding system).

Method	ML Utility$\uparrow$	Stat. Sim.$\uparrow$	Privacy$\uparrow$	Speed$\uparrow$	Few-shot$\uparrow$
CTGAN	0.78	0.81	0.65	Fast	55
TVAE	0.80	0.83	0.67	Fast	55
TabDDPM	0.86	0.89	0.72	Moderate	55
GReaT	0.83	0.85	0.70	Slow	51
TabuLa	0.84	0.87	0.71	Slow	51

Tabby and DeLTa – Advanced LLM Approaches

The success of TabuLa demonstrated that language models could generate tabular data. But it also exposed fundamental limitations: a single monolithic language model treats all columns identically, has no mechanism for enforcing logical constraints, and wastes parameters on the serialization overhead (column names, separators) rather than the actual data. Two lines of research address these limitations, each from a different angle.

The first, Tabby, brings the Mixture-of-Experts (MoE) paradigm to tabular language models. The insight is architectural: different columns need different expertise, and a single dense model wastes capacity by routing all information through all parameters. By allowing different subsets of parameters (“experts”) to specialize in different column types, Tabby achieves better generation quality with fewer active parameters.

The second, DeLTa, brings neurosymbolic reasoning to tabular generation. The insight is that neural models are fundamentally probabilistic – they sample from learned distributions – while domain constraints are fundamentally logical – they define hard boundaries on valid outputs. DeLTa marries the two: a neural generator proposes candidate rows, and a symbolic verifier checks and corrects them.

Together, these methods represent the cutting edge of LLM-based tabular generation, and they point toward a broader principle in AI: the best systems combine neural flexibility with symbolic correctness.

Tabby: Mixture-of-Experts for Tables

The Mixture-of-Experts (MoE) architecture, introduced by Shazeer et al. [51], replaces the single feed-forward network (FFN) in each Transformer layer with a collection of $E$ independent FFNs (“experts”) and a lightweight gating network that routes each token to the top-$k$ experts. The result is a model with many more total parameters but the same computational cost per token, because only $k\ll E$ experts are active for each token.

In natural language, MoE has been spectacularly successful: models like Mixtral [52] and Switch Transformers [53] use MoE to scale to trillions of parameters while remaining computationally tractable. But why would MoE help for tabular data?

The answer lies in the heterogeneity of table columns.

The gating function is the key to everything. In a well-trained MoE model, the gate learns to route different types of content to different experts. For tabular data, this means tokens corresponding to numerical values might be routed to experts that have specialized in arithmetic patterns, while tokens corresponding to categorical labels might be routed to experts that have specialized in discrete distributions.

Why MoE Helps for Tables

To understand why MoE is particularly well-suited for tabular data, we must appreciate a fundamental difference between natural language and tables.

In natural language, every word draws on roughly the same type of knowledge: syntax, semantics, pragmatics, world knowledge. A verb needs the same fundamental processing as a noun. The heterogeneity is subtle – differences in part-of-speech, register, topic – and a dense model handles it well.

In serialized tabular data, the heterogeneity is stark. A token representing a numerical value (“45”) requires arithmetic reasoning: understanding that 45 is between 44 and 46, that it is plausible for age but implausible for blood type, that it should correlate with income in a particular way. A token representing a categorical label (“engineer”) requires discrete classification: knowing the set of valid labels, their relative frequencies, and their associations with other features. A date token (“2023”) requires temporal reasoning. These are fundamentally different computational tasks.

The empirical evidence supports this theoretical analysis. Tabby models with 8 experts and top-2 routing consistently outperform dense models of equivalent computational cost (matched FLOPs) on tabular generation benchmarks. The improvement is largest on tables with highly heterogeneous column types – precisely where parameter interference would be most severe.

DeLTa: Neurosymbolic Tabular Generation

We now turn to a fundamentally different approach to improving LLM-based tabular generation. Where Tabby asks “how can we make the neural network itself better?”, DeLTa asks “what if the neural network does not need to be perfect – what if we add a checker?”

The motivation is a simple observation. Many real-world datasets come with domain constraints: logical rules that every valid record must satisfy. In a healthcare dataset, a patient listed as “pregnant” must have sex = “F.” In a financial dataset, the total of line items must equal the reported sum. In a census dataset, a person's age minus their birth year must equal the current year (approximately). These constraints are hard – they admit no exceptions – and they are symbolic – they can be expressed as logical predicates.

A purely neural generator, no matter how powerful, treats these constraints as soft statistical regularities. It learns that pregnant patients are usually female (probability $>0.99$) but may occasionally generate a male pregnant patient. For many applications – healthcare, finance, government – even a 0.1% violation rate is unacceptable.

The name DeLTa stands for Deep Learning with Tabular constraints, emphasizing the marriage of deep learning (the neural generator) with tabular constraints (the symbolic verifier).

The DeLTa Pipeline

The DeLTa generation pipeline operates in three stages, forming a loop that iterates until a valid row is produced or a maximum number of attempts is reached.

Stage 1: Neural Generation.

The LLM generates a candidate row $\widehat{𝐱}$ from a prompt. The prompt may include few-shot examples, column schemas, and (crucially) a natural-language description of the constraints:

Generate a patient record satisfying: age > 0, if sex is M then pregnant is No, bp_systolic > bp_diastolic.

Including constraints in the prompt provides soft guidance: the model attempts to satisfy them but is not guaranteed to succeed.

Stage 2: Symbolic Verification.

The generated row is parsed and checked against a formal constraint set $𝒞=\{C_1,C_2,\dots ,C_m\}$. Each constraint $C_j$ is a logical predicate over the feature values: (Constraint Check)$$\begin{array}{ccc}& C_j(\widehat{𝐱})\in \{\text{𝚝𝚛𝚞𝚎},\text{𝚏𝚊𝚕𝚜𝚎}\}.& \text{<span class="tml-eqn"></span>}\end{array}$$ The row passes if and only if all constraints are satisfied: ${\bigwedge }_{j=1}^m{C}_j(\widehat{𝐱})=\text{𝚝𝚛𝚞𝚎}$.

Stage 3: Rejection or Correction.

If the row fails verification, the system has two options:

1. Rejection. Discard the row and generate a new candidate. Simple but potentially wasteful if the constraint satisfaction rate is low.

2. Correction. Use the violated constraints to modify the row. For example, if the constraint “bp_systolic $>$ bp_diastolic” is violated with bp_systolic $=80$ and bp_diastolic $=90$, the repair module can swap the values or resample bp_systolic from the conditional distribution given bp_diastolic $=90$.

The correction approach is more sample-efficient but requires a repair module that can modify individual features without destroying the overall statistical coherence of the row.

Learning and Encoding Domain Constraints

The power of the neurosymbolic approach hinges on the quality of the constraint set $𝒞$. Too few constraints, and invalid rows slip through. Too many, and the acceptance rate drops to the point where generation becomes impractically slow. Where do the constraints come from?

There are two complementary sources:

Expert-Specified Constraints.

Domain experts formalize their knowledge as logical rules. In healthcare, a physician might specify “if diagnosis is Type 2 Diabetes, then HbA1c $\ge$ 6.5” (the diagnostic criterion). In finance, an auditor might specify “total_revenue = domestic_revenue + international_revenue.” These constraints are typically few in number but high in importance – they represent hard invariants that must never be violated.

Automatically Discovered Constraints.

Given a training dataset, we can mine constraints directly from the data using techniques from database theory and data profiling:

1. Functional dependencies. If column $A$ determines column $B$ in the data (i.e., $A=a⟹B=b$ for all rows), this becomes a constraint: $C_{A\to B}(𝐱)=(x_A=a⟹x_B=b)$.

2. Range constraints. The minimum and maximum values of each numerical column define a range constraint: $C_j(𝐱)=(x_j^{\min }\le x_j\le x_j^{\max })$.

3. Conditional range constraints. Ranges that depend on another column: “if age $<18$, then income $<$ 100000” (a soft constraint mined from data).

4. Cross-column arithmetic. Relations like $x_3=x_1+x_2$ or $x_5\le x_4$ can be discovered by checking arithmetic identities across all rows.

5. Uniqueness constraints. If a column (or combination of columns) contains only unique values in the training data, generated rows should also be unique.

This grammar is expressive enough to capture the vast majority of real-world tabular constraints (range checks, conditional rules, cross-column comparisons) while remaining computationally tractable: checking a single constraint takes $O(1)$ time, and checking $m$ constraints takes $O(m)$ time.

The LLM Revolution in Tabular Data

We conclude this section by stepping back to survey the landscape. The evolution of tabular generation methods from 2019 to 2025 represents one of the most rapid transformations in applied machine learning: from specialized architectures designed specifically for tabular data to general-purpose language models repurposed for table generation.

The trade-offs between these methods can be organized along several axes. Speed favours GANs and VAEs. Fidelity on large datasets favours diffusion models. Flexibility and few-shot performance favour LLMs. Constraint satisfaction favours neurosymbolic hybrids. No single method dominates on all axes – a recurring theme in machine learning that reflects the no-free-lunch theorem at a practical level.

Method	Base LLM	Few-shot	MoE	Constraints	Params	Year
GReaT	GPT-2	Partial	55	55	124M	2023
TabuLa	LLaMA	51	55	55	7B	2023
REaLTabFormer	GPT-2	55	55	55	124M	2023
Tabby	Custom	51	51	55	7B${}^{\ast }$	2024
DeLTa	Any	51	Optional	51	Varies	2024
${}^{\ast }$ Total parameters; active parameters per token are $\sim$2B with top-2 routing.

Let us formalize the comparison along the most important dimension: sample quality versus computational cost.

The question of what comes next is fascinating. Several trends are visible:

1. Larger models. As language models scale from 7B to 70B to 400B+ parameters, their tabular generation quality will likely improve, particularly in few-shot and zero-shot settings.

2. Multi-table generation. Current methods generate single tables. Real databases contain multiple related tables with foreign key relationships. Generating coherent multi-table datasets is an open problem.

3. Privacy-preserving generation. Combining LLM-based generation with differential privacy guarantees is challenging because the pre-training data may itself contain sensitive information that “leaks” into the generated tables [28].

4. Interactive generation. Using conversational AI interfaces to let users specify, refine, and validate generated data in natural language: “Make the income distribution more skewed” or “Add 10% more female patients over 60.”

Before stating the formal guarantees, let us examine the computational geometry of constraint satisfaction in the tabular setting.

This geometric perspective reveals why some constraint sets are harder than others. If the constraints carve out a thin sliver of the feature space (e.g., requiring that ten features simultaneously satisfy narrow bounds), the volume $p_s$ is tiny, and rejection sampling becomes impractical. The repair module is most valuable precisely in these high-constraint settings.

We now state the mathematical formalization of the core guarantee provided by the neurosymbolic approach.

The following corollary ties together the efficiency analysis with a practical stopping criterion.

The interplay between neural generation and symbolic verification that DeLTa embodies is not unique to tabular data. It reflects a broader principle in AI that has been recognized since the earliest days of the field: the tension between connectionism (learning patterns from data) and symbolicism (reasoning from rules). The genius of the neurosymbolic approach is that it does not take sides. It lets each paradigm do what it does best – the neural network generates, the symbolic system verifies – and combines them in a way that is more than the sum of its parts.

We have now traversed the full landscape of LLM-based tabular generation, from the simple serialization insight of TabuLa through the architectural innovations of Tabby's Mixture-of-Experts to the neurosymbolic guarantees of DeLTa. The journey illustrates a broader trend in AI: the convergence of large language models with domain-specific structure. Language models provide the statistical backbone – the ability to learn and sample from complex distributions over sequences. Domain structure provides the guardrails – the constraints, types, and relationships that distinguish valid outputs from hallucinated noise. The most powerful systems combine both.

TabStruct – Evaluating Structural Fidelity

We have now studied an impressive arsenal of generative methods for tabular data: CTGAN, CTAB-GAN+, TabDDPM, GReaT, TabuLa, and several more. Each method claims to produce “realistic” synthetic tables. But what does “realistic” actually mean? How do we know whether a synthetic table is good enough to use in practice? And, perhaps more importantly, how do we compare two methods to decide which one is better?

These questions sound simple. They are not. In fact, evaluation is arguably the hardest unsolved problem in tabular data generation. The difficulty stems from a fundamental asymmetry: generating synthetic data requires learning a single distribution, but evaluating synthetic data requires comparing two distributions, the real and the synthetic, across an enormous number of dimensions simultaneously. A synthetic table might perfectly reproduce marginal distributions but completely destroy pairwise correlations. It might preserve correlations but introduce subtle privacy leaks. It might pass every statistical test yet prove useless for training a downstream classifier.

The field has struggled with this problem for years, and the result has been a chaotic landscape of ad hoc metrics, inconsistent benchmarks, and papers that cherry-pick evaluation criteria to favour their own methods. Into this landscape stepped TabStruct, a framework introduced by Liu, Kolossov, Yoon, and van der Schaar in 2025 [54], which proposed a principled, hierarchical approach to evaluating the structural fidelity of synthetic tabular data.

In this section, we dissect the evaluation crisis, explain TabStruct's solution, formalise each metric mathematically, and build intuition for why structural fidelity matters more than any single summary statistic.

The Evaluation Crisis

To appreciate why evaluation is so difficult, consider a concrete example. Suppose we have a medical dataset with three columns: age (continuous), diagnosis (categorical), and treatment_cost (continuous). In the real data, older patients are more likely to have certain diagnoses, and those diagnoses are associated with higher treatment costs. The joint distribution $p(\text{age},\text{diagnosis},\text{cost})$ has a complex, multi-modal structure.

Now suppose we evaluate a synthetic dataset using only marginal distributions. The synthetic data might perfectly reproduce the distribution of ages, the distribution of diagnoses, and the distribution of costs, each considered in isolation. Every column looks perfect. But if the synthetic data assigns treatment costs uniformly at random, independent of diagnosis, then the crucial three-way structure is destroyed. A model trained on this synthetic data to predict treatment costs from age and diagnosis would perform terribly on real data.

This example illustrates the core problem: marginal fidelity does not imply structural fidelity. Yet many early papers in tabular data generation evaluated their methods using only column-wise metrics, creating a misleading picture of progress.

The evaluation crisis manifests in several concrete ways:

1. Metric shopping: researchers report only the metrics on which their method performs best, omitting others.

2. Inconsistent baselines: different papers compare against different methods, often using different hyperparameters, making cross-paper comparison impossible.

3. Tiny benchmarks: many papers evaluate on only 2–3 datasets, often the same ones (Adult, Covertype, Intrusion), providing limited evidence for generalisation.

4. Missing privacy analysis: papers that claim to produce “privacy-preserving” synthetic data often provide no formal privacy guarantees and no empirical privacy evaluation.

TabStruct addresses these problems by defining a comprehensive, standardised evaluation protocol that spans five levels. We now examine each level in detail.

Column-Wise Metrics: Marginal Fidelity

The simplest question we can ask about synthetic data is: does each column, considered individually, look like the corresponding column in the real data? This is the question of marginal fidelity.

For a continuous column $X$ with real distribution $p_{\text{real}}$ and synthetic distribution $p_{\text{syn}}$, we measure marginal fidelity using the Kolmogorov–Smirnov (KS) statistic:

For categorical columns, the natural analogue is the Total Variation Distance (TVD), also known as the $L^1$ distance between probability vectors:

TabStruct computes these metrics for every column and reports both the per-column values and an aggregate (typically the mean). The per-column breakdown is crucial: a method might achieve excellent average KS scores while completely failing on one or two important columns.

Pairwise Metrics: Joint and Conditional Fidelity

Column-wise metrics, as we have argued, are necessary but not sufficient. The next level asks: for every pair of columns $(X_i,X_j)$, does the joint distribution $p(X_i,X_j)$ in the synthetic data match that of the real data?

For two continuous columns, we can measure the discrepancy between their joint distributions using the pairwise correlation difference:

(CORR DIFF)$$\begin{array}{ccc}& \mathrm{\Delta }{\rho }_{ij}=|{\rho }_{\text{real}}(X_i,X_j)-{\rho }_{\text{syn}}(X_i,X_j)|,& \text{<span class="tml-eqn"></span>}\end{array}$$

where $\rho$ denotes the Pearson correlation coefficient. While simple, this metric captures only linear relationships. TabStruct also employs the contingency similarity for pairs involving categorical columns and the mutual information difference for a more general measure:

(MI DIFF)$$\begin{array}{ccc}& \mathrm{\Delta }{I}_{ij}=|I_{\text{real}}(X_i;X_j)-I_{\text{syn}}(X_i;X_j)|,& \text{<span class="tml-eqn"></span>}\end{array}$$

where $I(X_i;X_j)=H(X_i)+H(X_j)-H(X_i,X_j)$ is the mutual information between columns $X_i$ and $X_j$.

For mixed pairs (one continuous, one categorical), TabStruct computes the conditional distribution distance. Given a categorical column $C$ and continuous column $X$, we partition the samples by category and compare the conditional distributions:

(COND DIST)$$\begin{array}{ccc}& D_{\text{cond}}(X|C)=\sum _{c\in 𝒞}p(C=c)\cdot D_{\text{KS}}(X_{\text{real}}|C=c,X_{\text{syn}}|C=c).& \text{<span class="tml-eqn"></span>}\end{array}$$

This is a weighted average of KS distances, where each weight is the real frequency of category $c$. A synthetic dataset could have perfect marginal distributions for both $X$ and $C$ while having large $D_{\text{cond}}$: the per-category distributions of $X$ might be scrambled.

The pairwise correlation matrix provides a particularly revealing visualisation. For a table with $d$ columns, we compute the $d\times d$ correlation matrix for both the real and synthetic data, then examine the element-wise difference. A good synthetic generator produces a difference matrix close to zero everywhere.

Global Utility Metrics

Global utility metrics ask a broader question: does the synthetic dataset, taken as a whole, preserve the statistical properties of the real dataset? While column-wise and pairwise metrics decompose quality into individual components, global utility takes a holistic view.

The most fundamental global utility metric is based on the Maximum Mean Discrepancy (MMD), a kernel-based distance between distributions:

In practice, we estimate the MMD using an unbiased estimator over finite samples ${\{x_i\}}_{i=1}^n\sim P$ and ${\{y_j\}}_{j=1}^m\sim Q$:

(MMD EST)$$\begin{array}{ccc}& {\widehat{\text{MMD}}}^2=\frac{1}{n(n-1)}\sum _{i\ne i^{\prime }}k(x_i,x_{i^{\prime }})-\frac{2}{nm}\sum _{i,j}k(x_i,y_j)+\frac{1}{m(m-1)}\sum _{j\ne j^{\prime }}k(y_j,y_{j^{\prime }}).& \text{<span class="tml-eqn"></span>}\end{array}$$

The MMD is attractive because it captures differences in all moments of the distributions simultaneously, not just means or variances. However, it requires careful kernel selection. For tabular data, TabStruct recommends using a mixture of RBF kernels with multiple bandwidth parameters to capture structure at different scales.

Another important global utility metric is the log-cluster metric, which measures whether the real and synthetic data form distinct clusters or overlap naturally:

Machine Learning Utility: TSTR and Beyond

For many practitioners, the ultimate question is not whether the synthetic data looks statistically similar to the real data, but whether it is useful. Specifically: can we train a machine learning model on synthetic data and have it perform well on real data?

This question is formalised through the TSTR (Train on Synthetic, Test on Real) protocol:

The TSTR protocol has several important variants:

• TSTR with multiple models: to avoid bias from a single model choice, evaluate TSTR with several classifiers (logistic regression, random forest, XGBoost, neural network) and report the average or worst-case ratio.

• TSTR with augmentation : train on a mixture of real and synthetic data, which often outperforms training on either alone. This measures whether synthetic data is useful as a complement to real data.

• TRTS (Train on Real, Test on Synthetic) : the reverse protocol, which measures whether the synthetic data covers the same support as the real data. If TSTR is high but TRTS is low, the synthetic data may be “too easy” and missing important edge cases.

Detection Metrics

Detection metrics are the adversarial cousin of utility metrics. Instead of asking “is the synthetic data useful?” they ask “is the synthetic data detectable?” The underlying idea is simple: if a classifier can reliably distinguish real samples from synthetic samples, then the synthetic data has detectable artefacts.

The log-cluster metric from Definition 42 is one approach. TabStruct extends this idea with more sophisticated detection methods:

An important subtlety is the choice of detection classifier. A simple logistic regression might miss complex artefacts that a gradient-boosted tree would catch. Conversely, an overly powerful classifier (e.g., a deep neural network) might overfit on irrelevant differences. TabStruct recommends using XGBoost with default hyperparameters as a reasonable middle ground.

Privacy Metrics

The promise of synthetic data is not merely utility; it is privacy. The whole point of generating synthetic tables is often to share data without exposing individual records. But does generating synthetic data actually protect privacy? The answer is nuanced, and it depends critically on how we define and measure privacy.

TabStruct includes two families of privacy metrics: membership inference and attribute inference.

Membership Inference.

A membership inference attack asks: given a synthetic dataset and a target record $x$, can we determine whether $x$ was in the training set used to generate the synthetic data? If the answer is yes, then the synthetic data is “memorising” aspects of the training data, which constitutes a privacy breach.

Formally, the attacker has access to the synthetic dataset ${𝒟}_{\text{syn}}$ and a target record $x$. The attacker must output a binary decision: $\hat{m}(x)\in \{0,1\}$, where $\hat{m}(x)=1$ means the attacker believes $x$ was in the training set. The membership advantage is:

(MI Advantage)$$\begin{array}{ccc}& {\text{Adv}}_{\text{MI}}=\mathrm{Pr}[\hat{m}(x)=1|x\in {𝒟}_{\text{train}}]-\mathrm{Pr}[\hat{m}(x)=1|x\notin {𝒟}_{\text{train}}].& \text{<span class="tml-eqn"></span>}\end{array}$$

An advantage of $0$ means the attacker is no better than random guessing; an advantage of $1$ means perfect inference. The most common attack strategy computes the distance from $x$ to its nearest neighbour in ${𝒟}_{\text{syn}}$: records that were in the training set tend to have closer synthetic neighbours due to memorisation.

Attribute Inference.

Attribute inference is a stronger attack: given a synthetic dataset and a target record $x$ with one attribute $x_j$ hidden, can we infer the hidden attribute? This directly measures whether the synthetic data leaks individual-level information.

(AI Advantage)$$\begin{array}{ccc}& {\text{Adv}}_{\text{AI}}=\mathrm{Pr}[{\hat{x}}_j=x_j|{𝒟}_{\text{syn}},x_{-j}]-\mathrm{Pr}[{\hat{x}}_j=x_j|x_{-j}],& \text{<span class="tml-eqn"></span>}\end{array}$$

where $x_{-j}$ denotes all attributes except $x_j$. The second term is the base rate: how well could we guess $x_j$ without any synthetic data, using only the other attributes? A positive advantage means the synthetic data provides additional information about individual records, which is a privacy concern.

The mathematical relationship between privacy and utility can be made precise through differential privacy:

Differential privacy provides a worst-case guarantee. TabStruct's empirical privacy metrics (membership inference, attribute inference) provide average-case assessments that complement the formal guarantee. A method might satisfy $(\epsilon ,\delta )$-DP with a specific $\epsilon$ yet still show high membership inference advantage on certain datasets, particularly when the dataset has outliers that are easy to detect.

TabStruct's Structural Fidelity Score

TabStruct's key contribution is the structural fidelity score, which aggregates the hierarchical metrics into a single, interpretable number. The score is computed as a weighted combination:

(SFID)$$\begin{array}{ccc}& S_{\text{struct}}=w_1\cdot S_{\text{col}}+w_2\cdot S_{\text{pair}}+w_3\cdot S_{\text{global}}+w_4\cdot S_{\text{ml}}+w_5\cdot (1-S_{\text{detect}}),& \text{<span class="tml-eqn"></span>}\end{array}$$

where each component score $S_i\in [0,1]$ with $1$ being best, $S_{\text{detect}}$ is inverted because lower detection is better, and the weights satisfy ${\sum }_i{w}_i=1$. The default weights in TabStruct assign higher importance to pairwise and ML utility metrics, reflecting the insight that structural preservation matters more than marginal preservation.

Each component score is computed from the raw metrics via normalisation. For the column-wise score:

(SCOL)$$\begin{array}{ccc}& S_{\text{col}}=1-\frac{1}{d}\sum _{j=1}^d\{\begin{array}{cc}{D}_{\text{KS}}(X_j)& \text{if }{X}_j\text{ is continuous},\\ D_{\text{TV}}(X_j)& \text{if }{X}_j\text{ is categorical}.\end{array}& \text{<span class="tml-eqn"></span>}\end{array}$$

For the pairwise score:

(Spair)$$\begin{array}{ccc}& S_{\text{pair}}=1-\frac{2}{d(d-1)}\sum _{i<j}\mathrm{\Delta }{I}_{ij},& \text{<span class="tml-eqn"></span>}\end{array}$$

where $\mathrm{\Delta }{I}_{ij}$ is the normalised mutual information difference from (MI DIFF).

Worked Example: Applying TabStruct

Let us walk through a complete TabStruct evaluation on a toy example. Suppose we have a real dataset with $n=1000$ samples and three columns:

• $X_1$: age (continuous, range $[18,90]$)

• $X_2$: income bracket (categorical, $\{$low, medium, high$\}$)

• $X_3$: spending (continuous, range $[0,\mathrm{50,000}]$)

The real data has the following structure: older individuals tend to have higher income brackets, and higher income brackets are associated with higher spending. There is also a direct age–spending correlation.

We generate $m=1000$ synthetic samples using two methods: Method A (a well-tuned CTGAN) and Method B (a simple method that independently samples each column from its marginal distribution).

Step 1: Column-wise Metrics.

Both methods achieve excellent column-wise scores. Method B is slightly better because it samples directly from the true marginals.

Step 2: Pairwise Metrics.

The difference is dramatic. Method B, which samples independently, destroys all pairwise structure. Method A preserves it reasonably well.

Step 3: Aggregate Scores.

Evaluation Frameworks and Metrics for Synthetic Data

TabStruct gave us a hierarchical way to think about evaluation. But evaluation in practice requires more than a set of metrics: it requires frameworks, libraries, benchmarks, and protocols that make evaluation reproducible, standardised, and fair. In this section, we broaden our scope from TabStruct's specific contributions to the entire landscape of synthetic data evaluation.

We will cover standardised libraries, statistical tests, information-theoretic metrics, downstream evaluation, fairness, and comprehensive benchmarking protocols. By the end, you will have a complete toolkit for evaluating any synthetic data generator, along with a clear understanding of which metrics to use in which situations.

SDMetrics and Standardised Evaluation

The Synthetic Data Vault (SDV) project, led by researchers at MIT, introduced SDMetrics as an open-source library for evaluating synthetic data. SDMetrics provides a standardised API for computing a wide range of quality metrics, organised into three categories:

1. Column Shapes: measures how well the synthetic data preserves the shape (distribution) of each individual column.

2. Column Pair Trends: measures how well the synthetic data preserves the relationships between pairs of columns.

3. New Row Synthesis: measures whether the synthetic rows are genuinely new (not copies of real rows), addressing both novelty and privacy.

Each category produces a score between $0$ and $1$, and the overall quality score is the average of these three categories.

Statistical Tests for Distribution Comparison

Beyond point metrics, we often want to perform formal hypothesis tests: is there statistically significant evidence that the real and synthetic distributions differ? This question calls for classical statistical machinery.

The Kolmogorov–Smirnov Test.

We have already seen the KS statistic (Definition 38). The associated KS test provides a $p$-value under the null hypothesis $H_0:F_{\text{real}}=F_{\text{syn}}$. Under $H_0$, the scaled statistic $\sqrt{n}\cdot D_{\text{KS}}$ converges to the Kolmogorov distribution.

The KS test is distribution-free (it requires no parametric assumptions) and is applicable to any continuous column. For categorical columns, we use the chi-squared test instead.

The Chi-Squared Test.

For a categorical column with $k$ categories, the chi-squared statistic compares observed and expected frequencies:

(Chisq)$$\begin{array}{ccc}& {\chi }^2=\sum _{c=1}^k\frac{{(O_c-E_c)}^2}{E_c},& \text{<span class="tml-eqn"></span>}\end{array}$$

where $O_c$ is the observed frequency of category $c$ in the synthetic data and $E_c$ is the expected frequency based on the real data. Under $H_0$, this statistic follows a chi-squared distribution with $k-1$ degrees of freedom.

For comparing joint distributions of two categorical columns, we use the chi-squared test of independence on the contingency table.

The Anderson–Darling Test.

The Anderson–Darling (AD) test is a refinement of the KS test that gives more weight to the tails of the distribution:

(AD)$$\begin{array}{ccc}& A^2=-n-\frac{1}{n}\sum _{i=1}^n(2i-1)[\ln F(x_{(i)})+\ln (1-F(x_{(n+1-i)}))],& \text{<span class="tml-eqn"></span>}\end{array}$$

where $x_{(i)}$ are the order statistics and $F$ is the reference CDF. The AD test is particularly useful for tabular data because tail behaviour often matters: rare events (outliers, edge cases) are precisely the cases where synthetic data tends to fail.

Information-Theoretic Metrics

Information theory provides a principled framework for measuring the distance between distributions. The two most important metrics for synthetic data evaluation are the Kullback–Leibler (KL) divergence and mutual information preservation.

KL Divergence.

The KL divergence from the real distribution $p$ to the synthetic distribution $q$ is:

(KL)$$\begin{array}{ccc}& D_{\text{KL}}(p\Vert q)=\int p(x)\ln \frac{p(x)}{q(x)}dx.& \text{<span class="tml-eqn"></span>}\end{array}$$

For discrete distributions:

(KL Discrete)$$\begin{array}{ccc}& D_{\text{KL}}(p\Vert q)=\sum _{x}p(x)\ln \frac{p(x)}{q(x)}.& \text{<span class="tml-eqn"></span>}\end{array}$$

The KL divergence is non-negative and equals zero if and only if $p=q$. However, it is asymmetric: $D_{\text{KL}}(p\Vert q)\ne D_{\text{KL}}(q\Vert p)$ in general. This asymmetry has practical implications.

The Jensen–Shannon (JS) divergence symmetrises the KL divergence:

(JS)$$\begin{array}{ccc}& D_{\text{JS}}(p\Vert q)=\frac{1}{2}{D}_{\text{KL}}(p\Vert \frac{p+q}{2})+\frac{1}{2}{D}_{\text{KL}}(q\Vert \frac{p+q}{2}).& \text{<span class="tml-eqn"></span>}\end{array}$$

The JS divergence is symmetric, bounded ($0\le D_{\text{JS}}\le \ln 2$ when using natural logarithm), and always well-defined even when $p$ and $q$ have different support.

Mutual Information Preservation.

Beyond comparing marginal distributions, we want to know whether the synthetic data preserves the dependency structure of the real data. This is captured by the mutual information matrix:

(MI Matrix)$$\begin{array}{ccc}& {𝐌}_{\text{real}}=[I_{\text{real}}(X_i;X_j){]}_{i,j},{𝐌}_{\text{syn}}=[I_{\text{syn}}(X_i;X_j){]}_{i,j}.& \text{<span class="tml-eqn"></span>}\end{array}$$

The mutual information preservation score is:

(MI PRES)$$\begin{array}{ccc}& S_{\text{MI}}=1-\frac{2}{d(d-1)}\sum _{i<j}|\frac{I_{\text{real}}(X_i;X_j)}{H_{\text{real}}(X_i,X_j)}-\frac{I_{\text{syn}}(X_i;X_j)}{H_{\text{syn}}(X_i,X_j)}|,& \text{<span class="tml-eqn"></span>}\end{array}$$

where we normalise by joint entropy to obtain a value in $[0,1]$.

The Wasserstein Distance.

The Wasserstein distance (also called the earth mover's distance) provides yet another way to compare distributions, with the advantage of being sensitive to the geometry of the sample space:

For one-dimensional distributions, the Wasserstein-1 distance has a closed-form expression in terms of the CDFs:

(Wasserstein 1D)$$\begin{array}{ccc}& W_1(P,Q)={\int }_{-\infty }^{\infty }|F_P(x)-F_Q(x)|dx.& \text{<span class="tml-eqn"></span>}\end{array}$$

Benchmarking Protocols

Individual metrics, however well-chosen, are only as useful as the benchmarking protocol in which they are embedded. A rigorous benchmark must specify:

1. Datasets: which real datasets are used, how they are pre-processed, and how train/test splits are created.

2. Baselines: which methods are compared, with what hyperparameters, and whether the hyperparameters were tuned on a validation set.

3. Metrics: which metrics are reported, at which levels of the evaluation hierarchy, and how they are aggregated.

4. Repetitions: how many random seeds are used, and whether confidence intervals are reported.

5. Compute: what hardware was used and how long each method was trained.

Standard Benchmark Datasets.

The tabular data generation community has converged on a small set of benchmark datasets, each chosen to test different aspects of generation quality:

The Benchmarking Pipeline.

A well-designed benchmark follows a standardised pipeline, which we illustrate in fig:struct:eval:benchmark_pipeline.

Downstream Task Evaluation

Statistical similarity is a necessary condition for good synthetic data, but it is not sufficient. The ultimate test is performance on downstream tasks: the actual analyses and models that practitioners intend to build with the data.

Classification and Regression.

The TSTR protocol from Machine Learning Utility: TSTR and Beyond is the standard approach for classification and regression tasks. To make the evaluation robust, we recommend:

1. Use at least three diverse classifiers/regressors: logistic regression (linear baseline), random forest (tree-based), and a neural network (deep learning).

2. Report both the absolute performance on the synthetic-trained model and the TSTR ratio relative to the real-trained model.

3. Use stratified cross-validation rather than a single train/test split.

4. For imbalanced datasets, use metrics that are robust to class imbalance (AUC-ROC, F1, balanced accuracy) rather than simple accuracy.

Clustering.

For unsupervised tasks, we can evaluate whether the synthetic data preserves cluster structure. The protocol is:

1. Apply a clustering algorithm (e.g., $k$-means) to both the real and synthetic data separately.

2. Compare the cluster centres, sizes, and assignments using the Adjusted Rand Index (ARI) or Normalised Mutual Information (NMI).

3. A high ARI between real and synthetic clusterings indicates that the global structure is preserved.

Feature Importance Preservation.

Another important downstream metric is whether the synthetic data preserves the relative importance of features. If a random forest trained on real data identifies features $A$, $B$, $C$ (in that order) as most important for predicting the target, and a random forest trained on synthetic data identifies the same ranking, then the synthetic data preserves the signal structure.

(FI PRES)$$\begin{array}{ccc}& S_{\text{FI}}=1-\frac{d_{\text{Kendall}}({\pi }_{\text{real}},{\pi }_{\text{syn}})}{d_{\text{Kendall}}^{\max }},& \text{<span class="tml-eqn"></span>}\end{array}$$

where ${\pi }_{\text{real}}$ and ${\pi }_{\text{syn}}$ are the feature importance rankings and $d_{\text{Kendall}}$ is Kendall's tau distance.

Fairness Evaluation of Synthetic Data

Synthetic data is sometimes proposed as a tool for debiasing: generating data that is more balanced or fair than the original. Conversely, a poorly designed generator might amplify biases present in the training data. Fairness evaluation is therefore essential.

The Fairness Challenge.

Consider a dataset used to train a loan approval model. The real data contains historical biases: applicants from certain demographic groups are approved at lower rates, even after controlling for creditworthiness. If we generate synthetic data from this biased dataset, the synthetic data will inherit (and potentially amplify) these biases.

The key fairness metrics for synthetic data evaluation are:

For synthetic data evaluation, we compute these gaps on both the real and synthetic data and compare:

(Fairness Change)$$\begin{array}{ccc}& {\mathrm{\Delta }}_{\text{fairness}}={\mathrm{\Delta }}_{\text{DP}}^{\text{syn}}-{\mathrm{\Delta }}_{\text{DP}}^{\text{real}}.& \text{<span class="tml-eqn"></span>}\end{array}$$

A negative ${\mathrm{\Delta }}_{\text{fairness}}$ indicates the synthetic data has reduced bias (good for debiasing applications); a positive value indicates amplification (a red flag).

Comprehensive Method Comparison

With our evaluation toolkit in hand, we can now present a comprehensive comparison of the methods covered in this chapter. The following tables summarise results across multiple benchmark datasets, using the TabStruct evaluation hierarchy.

Overall Quality Scores.

TSTR Results by Downstream Model.

The columns are: LR = Logistic Regression, RF = Random Forest, XGB = XGBoost, MLP = Multi-Layer Perceptron. Each value is the TSTR ratio (higher is better, $1.0$ is perfect).

Privacy–Utility Trade-off.

When to Use Which Method.

The comparison results suggest the following practical guidelines:

• Default choice: TabDDPM offers the best overall quality and is a strong default for most applications.

• Privacy-sensitive applications: Use CTGAN or CTAB-GAN+ with differential privacy, or TabDDPM with DP-SGD. LLM-based methods should be used cautiously when privacy matters.

• Small datasets: LLM-based methods (GReaT, TabuLa) excel on small datasets because they leverage pre-trained knowledge. When the real dataset has fewer than 1,000 rows, LLM-based methods often outperform purpose-built architectures.

• High-cardinality categorical columns: CTGAN and CTAB-GAN+ handle high-cardinality categories more gracefully than diffusion-based methods.

• Speed constraints: CTGAN and TVAE are the fastest methods. TabDDPM and LLM-based methods are significantly slower.

Building a Complete Evaluation Report

We conclude this section by outlining how to build a complete evaluation report for a synthetic data generator. A thorough report should include all five levels of the TabStruct hierarchy, plus additional context.

Causal Structure Preservation

A frontier topic in synthetic data evaluation is whether the generated data preserves the causal structure of the real data, not merely its statistical associations. Two datasets can have identical joint distributions yet encode entirely different causal mechanisms. If the real data satisfies $X\to Y$ (income causes spending) but the synthetic data is generated in a way that reverses the causal direction, then interventional queries on the synthetic data will yield incorrect answers.

In practice, we estimate causal graphs from both datasets using a structure learning algorithm (e.g., the PC algorithm or GES) and compare them using the SHD. A low SHD indicates that the synthetic data preserves causal relationships.

(Causal Score)$$\begin{array}{ccc}& S_{\text{causal}}=1-\frac{\text{SHD}({𝒢}_{\text{real}},{𝒢}_{\text{syn}})}{{\text{SHD}}_{\max }},& \text{<span class="tml-eqn"></span>}\end{array}$$

where ${\text{SHD}}_{\max }=d(d-1)$ for $d$ variables (the maximum possible SHD in a DAG with $d$ nodes).

Scalability and Computational Considerations

Evaluation itself can be computationally expensive. For a dataset with $d$ columns, computing all pairwise metrics requires $O(d^2)$ comparisons. Each comparison may involve sorting ($O(n\log n)$ for the KS statistic) or kernel evaluations ($O(n^2)$ for MMD). For large datasets with many columns, the evaluation cost can rival or exceed the generation cost.

For practitioners working with large-scale datasets (hundreds of thousands of rows, dozens of columns), a staged evaluation strategy is recommended:

1. Quick check: compute column-wise metrics (fast, $O(d\cdot n\log n)$) to catch gross failures.

2. Medium check: add pairwise correlation comparison ($O(d^2\cdot n)$) and the log-cluster detection metric.

3. Full evaluation: compute MMD, TSTR across multiple models, privacy metrics, and fairness analysis. This is expensive but necessary for publication or deployment.

Open Challenges in Evaluation

Despite the progress represented by TabStruct and SDMetrics, several fundamental challenges remain.

Higher-Order Dependencies.

All the metrics we have discussed capture at most pairwise relationships. Real data often contains three-way, four-way, or even higher-order dependencies. For example, the interaction between age, diagnosis, and treatment cost in a medical dataset cannot be fully captured by any combination of pairwise metrics. Developing scalable metrics for higher-order fidelity is an open problem.

Temporal and Sequential Structure.

When tabular data has a temporal component (e.g., patient records over time), evaluation must also assess whether the synthetic data preserves temporal patterns: trends, seasonality, and auto- correlations. Standard TabStruct metrics do not address this.

Domain-Specific Validity.

A synthetic medical record might pass every statistical test yet contain clinically impossible combinations (e.g., a 5-year-old patient with a diagnosis of age-related macular degeneration). Domain-specific validity checks require expert knowledge and are difficult to automate.

The Curse of Dimensionality.

As the number of columns increases, the number of pairwise comparisons grows quadratically, and the difficulty of estimating joint distributions grows exponentially. For high-dimensional tables (50+ columns), current metrics may fail to detect subtle but important structural failures.

GFlowNets for Structured Generation

Every generative model we have studied so far shares a common strategy: learn a single distribution over the data space, then sample from it. GANs learn a generator that maps noise to data. Diffusion models learn a reverse process that denoises. Autoregressive models learn to predict the next token. In each case, the goal is to produce samples that look like the training data, and the training signal comes from matching the model's output distribution to the empirical distribution of observed examples.

But what if the goal is not to imitate a dataset, but to discover new objects that score highly under a given reward function? What if we want not the single best object, but a diverse set of high-quality objects? What if the objects are combinatorial structures, molecules, graphs, or discrete sequences, and the reward landscape is rugged, multimodal, and expensive to evaluate?

This is the setting that Generative Flow Networks, or GFlowNets, were designed to address. Introduced by Yoshua Bengio and collaborators in 2021 [55], and refined in a series of subsequent papers [56][57], GFlowNets represent a fundamentally different approach to generative modelling. They do not learn to imitate data. They learn to sample in proportion to a reward.

The distinction is profound. A GAN trained on molecules will generate molecules that resemble the training set. A GFlowNet trained with a reward function that measures drug-likeness will generate a diverse collection of molecules, each with high drug-likeness, including molecules that look nothing like any training example. This is not reinforcement learning (which seeks the single highest-reward object) and it is not density estimation (which seeks to match an observed distribution). It is something new: proportional sampling from a reward function over a combinatorial space.

The DAG Formulation: States, Actions, and Flows

The central mathematical object in GFlowNet theory is a directed acyclic graph (DAG) that describes how composite objects are constructed step by step. Let us build this formalism carefully.

The intuition is simple. Think of building a molecule atom by atom. The initial state $s_0$ is the empty molecule. Each action adds an atom or a bond. Non-terminal states are partially assembled molecules. Terminal states are complete molecules ready for evaluation. The DAG structure ensures that construction is irreversible: once you add an atom, you cannot remove it (though the same complete molecule may be reachable via different construction orders).

Now we introduce the key concept: flow.

The analogy to fluid dynamics is intentional. Imagine water flowing through a network of pipes. Water enters at the source $s_0$, flows through the DAG along edges, and exits at terminal states. The amount of water exiting at each terminal state $x\in S_f$ is $F(x)$. If we normalize by the total flow $Z=F(s_0)={\sum }_{x\in S_f}F(x)$, then $F(x)/Z$ gives a probability distribution over terminal states.

The fundamental insight of GFlowNets is this: if we choose the flow function so that $F(x)=R(x)$ for all terminal states $x$, then the resulting distribution samples proportionally to the reward: (Gflownet Sampling)$$\begin{array}{ccc}& \phantom{GFlowNet!samplingdistribution}p(x)=\frac{F(x)}{Z}=\frac{R(x)}{{\sum }_{x^{\prime }\in S_f}R(x^{\prime })}.& \text{<span class="tml-eqn"></span>}\end{array}$$

Flow Matching and the Detailed Balance Condition

The elegant picture of fig:struct:gflownet-dag raises an immediate practical question: how do we learn the flow function? We cannot simply set $F(x)=R(x)$ at terminal states and work backwards, because the number of paths through the DAG grows combinatorially with the object size. We need a trainable parametric model and a loss function.

The original GFlowNet paper [55] proposed the flow matching condition as the basis for training.

The flow matching loss penalizes violations of this condition: (FLOW Matching LOSS)$$\begin{array}{ccc}& \phantom{GFlowNet!flowmatchingloss}{ℒ}_{\mathrm{FM}}(\theta )={𝔼}_{s\sim \pi }\left[\right(\log \frac{{\sum }_{s^{\prime }\to s}{F}_{\theta }(s^{\prime }\to s)}{{\sum }_{s\to s^{\prime \prime }}{F}_{\theta }(s\to s^{\prime \prime })}{)}^2],& \text{<span class="tml-eqn"></span>}\end{array}$$ where $\pi$ is a distribution over non-terminal states used for training (typically a mixture of on-policy and off-policy samples). The use of log-space is crucial: it prevents numerical issues when flows span many orders of magnitude, which they typically do in combinatorial spaces.

However, flow matching has a significant limitation. It requires summing over all parents and children of every state visited during training. For states with many parents (which arise when the same partial object can be reached by many construction orders), this sum is expensive. This motivated the detailed balance condition.

The detailed balance condition is a local condition on individual edges, not a global condition on states. This makes it cheaper to evaluate but also more restrictive: detailed balance implies flow matching, but not vice versa. Nevertheless, detailed balance is often preferred in practice because it avoids the expensive summation over parents and children.

Training: Trajectory Balance and Sub-Trajectory Balance

The most important training objective for GFlowNets, and the one used in most practical implementations, is the trajectory balance (TB) loss introduced by Malkin et al. [57]. The key insight is beautiful in its simplicity: instead of enforcing balance at individual edges or states, enforce it over entire trajectories from $s_0$ to a terminal state.

The trajectory balance loss takes the log ratio and penalizes its squared deviation from zero: (TB LOSS)$$\begin{array}{ccc}& \phantom{GFlowNet!TBloss}{ℒ}_{\mathrm{TB}}(\theta )={𝔼}_{\tau \sim \pi }\left[{\left(\log \frac{Z_{\theta }{\prod }_{t=0}^{n-1}{P}_F(s_{t+1}|s_t;\theta )}{R(x){\prod }_{t=1}^n{P}_B(s_{t-1}|s_t;\theta )}\right)}^2\right].& \text{<span class="tml-eqn"></span>}\end{array}$$

A further refinement is the sub-trajectory balance (SubTB) condition [57], which enforces balance over sub-paths rather than complete trajectories. This provides denser training signal and faster credit assignment.

The SubTB loss sums the squared log-ratio over all sub-trajectories (or a random subset) within each sampled trajectory: (Subtb LOSS)$$\begin{array}{ccc}& {ℒ}_{\mathrm{SubTB}}(\theta )={𝔼}_{\tau }\left[\sum _{0\le i<j\le n}{\lambda }^{j-i}{\left(\log \frac{F_{\theta }(s_i){\prod }_{t=i}^{j-1}{P}_F(s_{t+1}|s_t)}{F_{\theta }(s_j){\prod }_{t=i+1}^j{P}_B(s_{t-1}|s_t)}\right)}^2\right],& \text{<span class="tml-eqn"></span>}\end{array}$$ where $\lambda \in (0,1]$ is a weighting parameter that downweights longer sub-trajectories.

Connection to MCMC and Variational Inference

GFlowNets did not emerge in a vacuum. They connect deeply to two pillars of probabilistic computation: Markov chain Monte Carlo (MCMC) and variational inference (VI). Understanding these connections illuminates why GFlowNets work and where they fit in the broader landscape.

Connection to MCMC.

Both MCMC and GFlowNets aim to sample from a distribution proportional to a reward (or unnormalized density). MCMC does this by constructing a Markov chain whose stationary distribution is the target. The chain starts from an arbitrary state and, after a burn-in period, produces correlated samples from the target.

GFlowNets take a fundamentally different approach: they learn a constructive policy that builds objects from scratch. Each sample is independent (no Markov chain, no burn-in, no mixing time). The price is that the policy must be trained, which requires many reward evaluations during training. The benefit is that, once trained, sampling is fast and embarrassingly parallel.

Connection to variational inference.

Variational inference posits a family of distributions $q_{\theta }$ and minimizes $\mathrm{KL}(q_{\theta }\Vert p)$, where $p(x)\propto R(x)$. A GFlowNet's forward policy $P_F$ defines a distribution over terminal states $p_{\theta }(x)={\sum }_{\tau \to x}{\prod }_t{P}_F(s_{t+1}|s_t)$. When the trajectory balance loss is minimized, $p_{\theta }(x)=R(x)/Z$, which is exactly the target. Thus, GFlowNet training can be interpreted as a form of variational inference where the variational family is the set of all constructive policies on the DAG.

Why GFlowNets Matter for Structured Data

The connection between GFlowNets and structured data generation is both natural and powerful. Structured objects, whether molecules, graphs, programs, or combinatorial configurations, are typically built step by step. A molecule is assembled atom by atom. A graph grows edge by edge. A program is written token by token. This sequential construction is exactly what the GFlowNet DAG captures.

Moreover, structured data problems often come with a reward function rather than a dataset. In drug discovery, the reward might be a combination of binding affinity, synthetic accessibility, and drug-likeness. In combinatorial optimization, the reward is the objective function. In Bayesian structure learning, the reward is the posterior probability of a graph given data. GFlowNets are tailor-made for these settings.

Combinatorial optimization.

GFlowNets also connect to combinatorial optimization through the temperature parameter. If the reward is $R(x)=\exp (\beta \cdot f(x))$, then as $\beta \to \infty$, the sampling distribution concentrates on the global optimum. At moderate $\beta$, the GFlowNet explores a neighborhood of high-quality solutions. This allows practitioners to trade off between exploitation (high $\beta$, finding the best solution) and exploration (low $\beta$, finding diverse good solutions).

Applications of GFlowNets

Since their introduction, GFlowNets have been applied to a growing list of structured generation problems. We highlight three representative applications.

Molecule generation.

The original GFlowNet paper demonstrated the approach on molecule generation, building molecules as graphs where nodes are atoms and edges are bonds. The action space includes adding atoms of various types and forming bonds between existing atoms. Using a reward based on a proxy model of binding affinity, the GFlowNet generates diverse molecules that are both novel and high-scoring. Compared to reinforcement learning baselines (which collapse to a single high-reward molecule and minor variants), GFlowNets produce orders of magnitude more unique molecules while maintaining comparable reward quality.

Bayesian structure learning.

Deleu et al. applied GFlowNets to the problem of sampling directed acyclic graphs from the posterior distribution given data [56]. The state space consists of partially constructed DAGs, and actions add directed edges. The acyclicity constraint is enforced by only allowing edges that do not create cycles. The reward is the marginal likelihood, which can be computed in closed form for linear Gaussian models. The GFlowNet posterior approximation matches or exceeds variational inference baselines while producing more diverse graph samples.

Biological sequence design.

GFlowNets have been applied to the design of DNA aptamers, RNA sequences, and antimicrobial peptides. In each case, the state space is the set of all prefixes of a biological sequence, actions append nucleotides or amino acids, and the reward reflects biological activity predicted by a learned oracle. The diversity of GFlowNet samples is particularly valuable in biological design, where synthesizing and testing multiple diverse candidates increases the probability of finding a successful drug or diagnostic tool.

Advanced Topics: Consistency Models, Uncertainty, and Beyond

The structured data generation landscape continues to evolve rapidly. In this section, we survey several advanced topics that push the boundaries of what is possible: consistency models adapted for tabular data, uncertainty quantification for synthetic data, conformal prediction guarantees, active learning strategies, and federated approaches to synthetic data generation. Each topic addresses a fundamental limitation of the methods we have studied so far, and each opens new avenues for research and application.

Consistency Models for Tabular Data

Diffusion models have proven remarkably effective for tabular data, as we saw with TabDDPM and its successors. However, they share a fundamental limitation with all iterative denoising approaches: sampling requires many sequential denoising steps, making generation slow. For applications that demand real-time synthetic data, such as online data augmentation during training or interactive data exploration tools, this latency is unacceptable.

Consistency models, introduced by Song et al. [58], offer an elegant solution. The key idea is to learn a function that maps any point on the diffusion trajectory directly to the clean data point, enabling one-step generation without iterative denoising.

Adapting to tabular data.

The adaptation of consistency models to tabular data requires handling the same challenges that arose in TabDDPM: mixed data types, multimodal continuous distributions, and categorical variables. The approach combines the hybrid forward process (Gaussian noise for continuous columns, multinomial diffusion for categorical columns) with a consistency distillation procedure.

The practical benefits are substantial. Empirical evaluations on standard tabular benchmarks (Adult, Default of Credit, Covertype) show that consistency-distilled tabular models achieve 95% or more of the quality of the original diffusion model while reducing generation time by 10–50$\times$. For a table with one million rows, this can reduce generation time from hours to minutes.

Uncertainty Quantification in Synthetic Data

When we replace real data with synthetic data, a natural question arises: how much can we trust the synthetic data? If a classifier trained on synthetic data makes a prediction, how confident should we be in that prediction? If a statistical analysis reveals a correlation in synthetic data, does that correlation exist in the real population, or is it an artefact of the generative model?

These questions fall under the umbrella of uncertainty quantification (UQ) for synthetic data, a topic that is critical for high-stakes applications but has received surprisingly little attention until recently.

Sources of uncertainty.

There are three distinct sources of uncertainty when working with synthetic data:

1. Aleatoric uncertainty: inherent randomness in the data-generating process. Even with a perfect generative model, synthetic samples will vary from one generation to another. This is irreducible.

2. Epistemic uncertainty: uncertainty about the generative model itself. The model was trained on a finite dataset and may not have learned the true data distribution. This is reducible with more training data or a better model.

3. Generative uncertainty: uncertainty introduced by the sampling process. A finite synthetic dataset may not represent the full learned distribution, especially in the tails. This is reducible by generating more synthetic samples.

Ensemble methods for epistemic uncertainty.

A principled approach to epistemic uncertainty is to train multiple generative models (an ensemble) and examine the variability across models. If all models agree on a particular statistical property (e.g., the correlation between two columns), we can be confident that the property reflects the training data. If models disagree, the property is uncertain.

Concretely, given $M$ independently trained generative models $p_{{\theta }_1},\dots ,p_{{\theta }_M}$, each generates a synthetic dataset ${𝒟}_m^{\mathrm{synth}}$. For a statistic of interest $T$ (e.g., a regression coefficient), the ensemble estimate and uncertainty are: (Ensemble MEAN)$$\begin{array}{cccc}& \hat{T}& =\frac{1}{M}\sum _{m=1}^{M}T({𝒟}_m^{\mathrm{synth}}),& \text{<span class="tml-eqn"></span>}\\ & {\hat{\sigma }}_T^2& =\frac{1}{M-1}\sum _{m=1}^M(T({𝒟}_m^{\mathrm{synth}})-\hat{T}{)}^2.& \text{<span class="tml-eqn"></span>}\end{array}$$

Conformal Prediction for Synthetic Data Quality

Can we provide rigorous statistical guarantees about the quality of synthetic data? Conformal prediction, a framework for distribution-free uncertainty quantification, offers a surprisingly practical answer.

The application to synthetic data quality proceeds in two steps. First, train a generative model on real data and generate a synthetic dataset. Second, use a held-out real dataset (not used for training the generative model) as the calibration set. The conformal framework then provides prediction sets that are valid for real data, even though the downstream model was trained on synthetic data.

This is a remarkable result. It means that conformal prediction provides a safety net: even if the generative model is poor, the prediction sets remain valid (they simply become larger, reflecting the reduced confidence). This makes conformal prediction an ideal companion to synthetic data generation in safety-critical applications.

Active Learning with Synthetic Data

A powerful application of synthetic data lies in active learning: the strategy of iteratively selecting the most informative data points to label, thereby reducing the total labelling cost. Synthetic data adds a new dimension to active learning: the ability to generate targeted training examples for underrepresented or uncertain regions of the input space.

The active synthesis loop.

The basic idea is a cycle with four steps:

1. Train a downstream model on the current (real + synthetic) dataset.

2. Identify regions of high uncertainty or poor performance.

3. Generate synthetic examples targeted at those regions using a conditional generative model.

4. Validate the synthetic examples using a small amount of real data or expert feedback, then add them to the training set.

The conditional generation in step 3 is critical. Rather than generating synthetic data uniformly at random, we steer the generator toward the regions where the downstream model is most uncertain. For a tabular CTGAN or diffusion model, this can be achieved by conditioning on specific feature values or by rejection sampling with an uncertainty-based acceptance criterion.

Federated Synthetic Data Generation

In many real-world settings, the data needed to train a generative model is distributed across multiple institutions, and privacy regulations prohibit sharing it. Hospitals each have patient records. Banks each have transaction logs. Government agencies each have census responses. Can we train a high-quality generative model without ever centralizing the data?

Federated synthetic data generation combines federated learning with generative modelling to address this challenge.

Challenges in federated generative modelling.

Federated training of generative models faces several challenges beyond those of standard federated learning:

• Non-IID data: different institutions may have very different data distributions (e.g., a pediatric hospital versus a geriatric hospital). Standard federated averaging can struggle to learn a good global model in this setting.

• Mode collapse: GANs are already prone to mode collapse in the centralized setting; federated training exacerbates this because the discriminator sees only local data at each institution.

• Privacy leakage: model updates can leak information about the training data. Combining federated learning with differential privacy (adding noise to the updates) is essential but degrades model quality.

• Communication cost: generative models (especially diffusion models with large U-Nets) have many parameters, and transmitting full parameter updates is expensive over limited network connections.

Applications and Case Studies

The methods we have developed throughout this chapter, from CTGAN to TabDDPM, from GFlowNets to consistency models, are not academic curiosities. They solve real problems in real organizations, often problems that have resisted solution for years due to data scarcity, privacy constraints, or the sheer cost of data collection.

In this section, we present four detailed case studies spanning healthcare, finance, government statistics, and scientific discovery. Each case study follows a common structure: we describe the problem, explain why synthetic data is needed, present the technical approach, and discuss results and lessons learned. Our goal is not simply to list applications but to develop the reader's intuition for when and how to deploy generative models for structured data in practice.

Healthcare: Electronic Health Record Generation

The problem.

Healthcare is simultaneously one of the most data-hungry and most data-restricted domains. Machine learning models for clinical decision support, disease prediction, and treatment optimization require large, representative patient datasets. Yet healthcare data is governed by stringent privacy regulations, including HIPAA in the United States, GDPR in Europe, and similar frameworks worldwide. Sharing patient records across institutions for research is a process fraught with legal, ethical, and logistical barriers.

The result is a paradox: the domain that most needs data-driven approaches is the domain where data is hardest to obtain. Synthetic electronic health records (EHRs) offer a way out.

The technical approach.

A typical EHR contains dozens to hundreds of features per patient: demographic information (age, sex, ethnicity), vital signs (blood pressure, heart rate), lab results (glucose, cholesterol, haemoglobin), diagnosis codes (ICD-10), medication lists, and temporal sequences of visits. This is a formidable modelling challenge that combines continuous features, categorical features, ordinal features, hierarchical codes, and temporal dependencies.

The state of the art for EHR generation uses a multi-stage pipeline:

1. Static feature generation: A tabular generative model (CTGAN, TabDDPM, or a Transformer-based model) generates the static features of each patient: demographics, baseline lab values, and initial diagnoses.

2. Temporal sequence generation: Conditioned on the static features, a sequential model (recurrent network, temporal diffusion model, or autoregressive Transformer) generates the sequence of clinical events over time: visits, lab results at each visit, medication changes, and new diagnoses.

3. Post-processing and validation: Generated records are checked for clinical plausibility (e.g., rejecting records where a patient diagnosed with Type 1 diabetes has no insulin prescriptions) and adjusted to match known population statistics.

Results and lessons learned.

Several large-scale evaluations have demonstrated the viability of synthetic EHRs. Key findings include:

• Predictive performance: Models trained on synthetic EHRs achieve 85–95% of the performance of models trained on real data for common prediction tasks (mortality, readmission, length of stay). The gap narrows as the synthetic dataset grows larger, suggesting that quantity can partially compensate for the distributional shift from real to synthetic.

• Privacy: Properly generated synthetic EHRs resist membership inference attacks and attribute inference attacks, especially when combined with differential privacy during training. However, extremely rare patients (those with unique combinations of conditions) require special care, as the generative model may memorize them.

• Clinical plausibility: The post-processing step is essential. Without it, generative models produce records that are statistically plausible but clinically absurd (e.g., a 25-year-old male with a diagnosis of ovarian cancer). Domain-specific constraints, implemented either as hard filters or as soft penalties during training, are necessary for clinical validity.

Finance: Fraud Detection and Risk Modelling

The problem.

Financial institutions face two perennial data challenges. First, fraud is rare: in a typical credit card dataset, fewer than 0.2% of transactions are fraudulent. Training a fraud detector on such imbalanced data is notoriously difficult. Second, financial data is sensitive: sharing transaction records across institutions or with external researchers risks exposing customer information and violating regulations such as PCI DSS and GDPR.

Synthetic data addresses both challenges simultaneously. It can augment the minority class (fraudulent transactions) to train better classifiers, and it can replace real data for model development, testing, and regulatory reporting.

The technical approach.

Financial transaction data is tabular with strong temporal dependencies and complex constraints. A single credit card transaction includes the timestamp, merchant category, transaction amount, location, currency, card type, and numerous derived features. The key technical challenges are:

• Extreme class imbalance: The generative model must learn the distribution of both legitimate and fraudulent transactions, even though the latter constitute a tiny fraction of the training data.

• Sequential patterns: Fraud often manifests as anomalous sequences (e.g., ten transactions in different countries within an hour). The generative model must capture these temporal patterns.

• Adversarial realism: If synthetic fraud data is “too easy,” classifiers trained on it will fail on real fraud. The synthetic fraudulent transactions must be as challenging to detect as real ones.

The most successful approaches use class-conditional generation: train separate models (or a single conditional model) for legitimate and fraudulent transactions, then combine the synthetic outputs at the desired class ratio.

Results and lessons learned.

• Fraud detection improvement: Augmenting the minority class with synthetic fraudulent transactions typically improves F1 scores by 30–70% compared to training on the imbalanced real data alone. The improvement is most pronounced for novel fraud patterns that are underrepresented in the training data.

• Stress testing: Synthetic data enables “what-if” analyses that are impossible with historical data. What if interest rates spike by 500 basis points? What if a major counterparty defaults? By training the generative model to condition on macroeconomic variables, financial institutions can generate synthetic datasets reflecting hypothetical scenarios and stress-test their risk models.

• Regulatory compliance: Several regulatory frameworks now explicitly recognize synthetic data as a tool for model validation and reporting, provided the synthetic data is generated with appropriate privacy guarantees.

Census and Government Data

The problem.

Government statistical agencies collect detailed information about populations, including income, employment, health status, education, household composition, and more. This data is invaluable for policy research, economic planning, and public health. However, releasing microdata (individual-level records) poses serious reidentification risks. Even after removing names and addresses, combinations of quasi-identifiers (age, zip code, occupation) can uniquely identify individuals.

Traditional privacy protection methods, such as suppression, generalization, and perturbation, degrade the analytical utility of the data. Synthetic data offers an alternative: release a fully synthetic microdata file that preserves the statistical properties of the real data while protecting individual privacy.

The technical approach.

The U.S. Census Bureau has been a pioneer in synthetic data for government statistics, producing synthetic versions of the American Community Survey (ACS) and other surveys. The technical approach typically involves:

1. Sequential synthesis: Generate features one at a time, conditioning each on the previously generated features. This preserves the conditional dependencies and is particularly effective for hierarchical data (household $\to$ person $\to$ income).

2. Multiple synthesis: Generate $M>1$ synthetic datasets from the same generative model. Analysts run their analysis on each synthetic dataset and combine results using Rubin's combining rules, which provide valid confidence intervals that account for the additional uncertainty introduced by the synthesis.

3. Utility verification: Compare key statistics (means, medians, regression coefficients, cross-tabulations) between the real and synthetic data. The goal is not perfect reproduction (which would compromise privacy) but sufficient fidelity for the intended analytical use cases.

Results and lessons learned.

• Preserving marginals: Modern generative models (especially diffusion-based approaches) can preserve one-way and two-way marginal distributions with high fidelity. Higher-order interactions (three-way and above) are harder to preserve and require either more training data or explicit constraints.

• Rare subpopulations: Small geographic areas or rare demographic groups are particularly challenging. If only five people in a census tract are Native American women aged 65+, the generative model may either fail to represent this group or effectively memorize those five individuals. Hierarchical models that share information across geographic areas can help.

• Public trust: The success of synthetic census data depends not only on technical quality but also on public trust. Transparency about the synthesis methodology, the privacy guarantees provided, and the limitations of synthetic data is essential for public acceptance.

Scientific Discovery: Drug Design and Materials Science

The problem.

Scientific discovery is perhaps the most exciting application domain for structured data generation, because the goal is not merely to reproduce existing data but to discover entirely new objects: molecules, materials, catalysts, or proteins with desired properties. This is the regime where GFlowNets, conditional generation, and reward-guided sampling truly shine.

Drug design.

The traditional drug discovery pipeline takes 10–15 years and costs over $2 billion per approved drug. A major bottleneck is the identification of “lead compounds”: small molecules that bind to a disease target with high affinity and acceptable pharmacokinetic properties. The chemical space of drug-like molecules is estimated at ${10}^{60}$ or more, making exhaustive search impossible.

Generative models for molecular design aim to explore this vast space efficiently. The technical approaches include:

• SMILES-based generation: Represent molecules as SMILES strings and use language models (autoregressive Transformers, RNNs) to generate new strings. This approach is simple but suffers from the fragility of SMILES syntax: a single invalid character produces an unparseable string.

• Graph-based generation: Represent molecules as graphs and use graph neural networks to generate nodes and edges. GFlowNets operate naturally in this setting, building molecules atom by atom.

• 3D generation: Generate the three-dimensional coordinates of atoms directly, using equivariant diffusion models that respect the rotational and translational symmetries of physical space.

Materials science.

The search for new materials (superconductors, battery cathodes, photovoltaic absorbers) faces analogous challenges to drug design. The space of possible crystal structures is vast, property evaluation is expensive (requiring quantum mechanical calculations such as density functional theory, or DFT), and diversity is valuable (materials scientists want multiple candidates with different compositions and structures, not just the single best).

Generative models for crystal structures represent each material as a combination of:

• A lattice (three basis vectors defining the unit cell);

• Atom types and positions within the unit cell;

• Space group symmetry (one of 230 possible symmetry groups).

Diffusion models and GFlowNets have been adapted to this representation, with the lattice vectors generated as continuous quantities, atom types as categorical variables, and fractional coordinates as continuous values constrained to $[0,1{)}^3$.

Protein engineering.

Beyond small molecules and crystals, generative models have made striking advances in protein engineering. Proteins are linear chains of amino acids (chosen from an alphabet of 20) that fold into complex three-dimensional structures determining their function. The sequence space for a protein of length 100 is ${20}^{100}\approx {10}^{130}$, vastly exceeding the number of atoms in the observable universe.

GFlowNets and diffusion models have been adapted to protein design by treating the amino acid sequence as the structured object. The state space consists of partially specified sequences (with some positions fixed and others left undetermined), and the reward function combines predicted stability, predicted function, and novelty relative to known proteins. Recent results have demonstrated that GFlowNet-designed enzyme variants can exhibit catalytic activity exceeding their wild-type counterparts by 2–5$\times$, while diffusion-based approaches have generated entirely novel protein folds not observed in nature.

The broader landscape.

The applications described above, drug design, materials science, and protein engineering, are representative but far from exhaustive. Generative models for structured data have also been applied to catalyst design for green chemistry, battery electrolyte discovery, polymer design for recyclable plastics, and the design of novel organic semiconductors. In each case, the common pattern holds: a combinatorial space of structured objects, an expensive evaluation function, and a need for diverse high-quality candidates. The toolbox we have built in this chapter (GFlowNets for proportional sampling, diffusion models for continuous structure generation, conditional models for targeted design) provides a unified framework that transfers across all of these domains.

Summary of application domains.

Table 4 provides a comparative overview of the four application domains we have discussed, highlighting the key requirements, methods, and evaluation criteria for each.

Domain	Primary Need	Preferred Methods	Key Metric	Main Risk
Healthcare	Privacy-preserving data sharing	TabDDPM, conditional Transformers	Predictive utility on downstream tasks	Privacy leakage, clinical implausibility
[6pt] Finance	Class balancing, stress testing	Class-conditional CTGAN, diffusion	F1 on fraud detection, scenario realism	Adversarial insufficiency
[6pt] Government	Public data release	Sequential synthesis, multiple imputation	Marginal and joint fidelity	Rare subpopulation disclosure
[6pt] Science	Novel object design	GFlowNets, conditional diffusion	Hit rate, diversity, novelty	Unrealistic candidates

Connections and Synthesis

We have now traversed a long and winding path. From the earliest attempts to model tabular data with simple GANs, through the subtleties of mode-specific normalization, conditional training, diffusion on discrete spaces, graph-based representations, and large language model serialization, we have assembled a rich toolkit for generating structured data. But these ideas did not develop in isolation. Each thread connects to broader themes in generative modelling – themes that span the entire arc of this book.

In this section, we step back and trace those connections explicitly. We show how structured data generation intersects with diffusion models, privacy, continual learning, and retrieval. We then synthesize everything into a unified perspective. The message is simple but powerful: all generative models, regardless of their surface differences, share the same fundamental goal – learning a distribution and sampling from it – and the differences lie in the inductive biases each approach brings to the table.

Structured Data Generation Meets Diffusion

When we first encountered diffusion models in 19, they operated on continuous signals – images, audio waveforms, latent vectors. The forward process added Gaussian noise, and the reverse process learned to denoise. How does this elegant framework extend to tabular data, where a column might contain integers, categories, or free-text annotations?

The answer, as we saw with TabDDPM and related methods, requires rethinking what “noise” means for each data type. Let us now make this connection precise.

The crucial observation is that both processes share a common structure: they gradually destroy information, converging to a known prior (Gaussian for continuous, uniform categorical for discrete). The reverse process then reconstructs.

Structured Data Meets Privacy

The intersection of structured data generation and privacy, explored in 29, is perhaps the most practically consequential connection in this entire chapter. Tabular data – medical records, financial transactions, census responses – often contains sensitive information about individuals. The promise of synthetic data is that we can share “fake” records that preserve statistical properties without exposing real individuals. But how secure is this promise?

Structured Data Meets Continual Learning

In 26, we studied the challenge of learning sequentially without forgetting. One of the most elegant solutions is generative replay: instead of storing raw data from previous tasks (which may be prohibited by storage or privacy constraints), we train a generative model on each task's data and use synthetic samples for rehearsal when learning subsequent tasks.

Structured data provides a natural testbed for generative replay because many real-world continual learning scenarios involve evolving tabular datasets – a hospital that receives patients with new conditions over time, a bank that encounters new transaction types, a sensor network that monitors changing environments.

Structured Data Meets Retrieval

The connection to retrieval (32) is both practical and deep. In many retrieval applications, the bottleneck is not the model architecture but the training data. Consider a retrieval system for rare medical conditions: by definition, few examples exist in any real dataset. Synthetic tabular data can augment these sparse datasets, providing the retrieval model with more diverse training signal.

A Unified View

We now step back to the highest level of abstraction. Every generative model we have studied in this chapter – and indeed in this entire book – shares the same fundamental goal:

Given samples from an unknown distribution $p_{\text{data}}$, learn a model $p_{\theta }$ such that samples from $p_{\theta }$ are indistinguishable from samples from $p_{\text{data}}$.

The differences among methods lie entirely in their inductive biases – the structural assumptions each approach makes about the data and the learning process. Let us catalogue these biases explicitly.

Paradigm	Key Inductive Bias	Strength	Weakness
GAN	Adversarial loss, implicit density	Sharp samples, fast generation	Mode collapse, training instability
VAE	Latent Gaussian prior, ELBO	Smooth latent space, principled	Blurry samples, posterior collapse
Diffusion	Iterative denoising	High fidelity, stable training	Slow sampling, hybrid needed
GFlowNet	Flow conservation on DAG	Diverse modes, compositional	Limited to structured spaces
Autoregressive	Column ordering, chain rule	Exact likelihood	Order sensitivity, slow
LLM-based	Text serialization	Zero-shot transfer	Tokenization artifacts
GNN-based	Graph structure	Captures dependencies	Scalability limits
NF	Invertible transform	Exact likelihood	Architecture constraints

Open Problems and Future Directions

Every mature field reaches a point where its practitioners can clearly articulate what they do not know. Tabular data generation has reached this stage. The basic tools work – we can generate synthetic tables that fool simple statistical tests. But deeper challenges remain, challenges that will occupy the next generation of researchers. In this section, we lay out the open landscape: theoretical questions without answers, practical challenges without solutions, and emerging directions that may transform the field.

Theoretical Foundations

Practical Challenges

The reproducibility crisis.

Tabular generation suffers from a severe reproducibility problem. Different papers use different datasets, different preprocessing pipelines, different evaluation metrics, and different train/test splits. A model that appears state-of-the-art on one benchmark may perform poorly on another – not because the model is bad, but because the evaluation protocol is different.

Standardized evaluation protocols.

What the field needs is a standardized evaluation framework – the equivalent of ImageNet for tabular generation. Such a framework would specify:

1. A canonical set of datasets spanning different sizes, dimensionalities, and column types.

2. Fixed preprocessing (normalization, encoding, missing value handling).

3. A fixed set of metrics with precise definitions.

4. Fixed downstream models and hyperparameters for machine learning efficiency evaluation.

5. A standard computational budget for each generator.

High-dimensional tables.

Most existing methods have been tested on tables with tens to at most a few hundred columns. Real-world datasets frequently have thousands of columns – genomic data with tens of thousands of features, electronic health records with thousands of coded diagnoses, financial data with hundreds of time-lagged variables. Scaling tabular generators to these regimes requires new ideas.

Towards Foundation Models for Tables

The most ambitious vision for the future of tabular generation is the construction of a single foundation model for tables – a model trained on a massive corpus of diverse tabular datasets that can generate realistic data for any new table schema with minimal or no fine-tuning.

The vision.

Imagine uploading a table schema – column names, types, a brief description of each variable, and perhaps a handful of example rows – and receiving in return a large, realistic synthetic dataset that respects the schema's constraints, captures plausible correlations, and matches the expected marginal distributions. This is the “GPT moment” for tabular data.

Transfer learning across domains.

The key enabler would be a universal representation of tabular data. The LLM-based approaches (TabuLa, GReaT, REaLTabFormer) point in this direction: by serializing tables as text, they inherit the transfer learning capabilities of the underlying language model. But text serialization loses much of the numerical precision and structural information that makes tabular data unique.

The architecture question.

What should the backbone of a tabular foundation model look like? Several candidates have emerged:

• Transformer-based. Following the success of LLMs, one approach is to treat tables as sequences (via serialization) and use a standard transformer architecture. This inherits the powerful transfer learning properties of transformers but requires solving the tokenization problem for numerical data.

• Diffusion-based. A latent diffusion model trained on a diverse corpus of tables, with a schema-conditioned denoiser. This approach naturally handles mixed types (via hybrid diffusion) but requires a universal encoding of table schemas into the conditioning mechanism.

• Graph-based. A GNN architecture that takes the column dependency graph as input and generates column values via message passing. This approach explicitly models relationships but requires learning the dependency structure for new schemas.

• Hybrid. Combining elements of all three: a transformer backbone with graph-structured attention patterns and diffusion-based generation. This is the most promising but also the most complex approach.

Few-shot and zero-shot generation.

The most exciting capability of a tabular foundation model would be zero-shot generation: given only a schema and perhaps a textual description, generate realistic synthetic data without any training examples. This requires the model to have internalized a rich prior over plausible real-world distributions – knowing, for example, that “age” is typically between 0 and 120, that “blood pressure” correlates with “BMI”, and that “zip code” and “city” are functionally dependent.

Ethical Considerations

The power of synthetic data generation brings responsibilities that the field is only beginning to reckon with. We highlight three critical ethical dimensions.

Bias amplification.

If the training data contains biases – racial disparities in medical treatment, gender gaps in loan approvals, socioeconomic segregation in housing data – the generative model will learn these biases. Worse, the generation process can amplify them.

Fair synthetic data generation.

The goal of fair synthetic data generation is to produce data that satisfies specified fairness constraints while maintaining utility. This can be formulated as a constrained optimization: $$\begin{array}{cccc}& \underset{\theta }{\min }& D_{\mathrm{KL}}(p_{\text{data}}\Vert p_{\theta })& \text{<span class="tml-eqn"></span>}\\ & \text{s.t.}& |\underset{\theta }{\mathrm{Pr}}(Y=1|A=a)-\underset{\theta }{\mathrm{Pr}}(Y=1|A=a^{\prime })|\le \tau \forall a,a^{\prime }\in 𝒜,& \text{<span class="tml-eqn"></span>}\end{array}$$ where $A$ is the sensitive attribute, $Y$ is the outcome, and $\tau$ is the fairness tolerance. This formulation trades off fidelity (matching $p_{\text{data}}$) against fairness (equalizing outcomes across groups).

Regulatory landscape.

The legal status of synthetic data is evolving rapidly. Under the EU's General Data Protection Regulation (GDPR), synthetic data that cannot be linked back to any individual is generally not considered personal data and is therefore exempt from many GDPR requirements. However, this exemption depends critically on the quality of the anonymization – if synthetic records can be re-identified (as our membership inference analysis showed is possible), the exemption may not apply.

Under the US Health Insurance Portability and Accountability Act (HIPAA), synthetic data generated with sufficient privacy guarantees may qualify as de-identified data under the Expert Determination method. However, HIPAA does not explicitly address synthetic data, creating legal uncertainty.

Historical Timeline

Every intellectual discipline benefits from knowing its own history. The field of generative models for structured data is young – barely a decade old – but its trajectory reveals patterns of progress, convergence, and occasional revolution. We present a comprehensive timeline, annotating each milestone with its conceptual contribution and its connection to the broader landscape.

Year	Method/Event	Key Contribution
2014	GAN [33]	Adversarial training framework; foundational for all GAN-based tabular methods
2017	MedGAN [34]	First application of GANs to generating discrete medical records (ICD codes, lab results)
2017	TableGAN	Convolutional architecture for tabular data with label consistency loss
2018	TGAN	LSTM-based generator with reversible data transformations for mixed-type columns
2019	CTGAN [4]	Mode-specific normalization (VGM) and training-by-sampling; establishes the modern tabular GAN paradigm
2020	TVAE	VAE counterpart to CTGAN using the same preprocessing pipeline; competitive but distinct trade-offs
2020	DP-CTGAN	Integration of differential privacy (DP-SGD) with CTGAN; first formal privacy guarantees for tabular GANs
2021	GFlowNet [35]	Flow networks over DAGs for sampling proportional to reward; new paradigm for discrete structured generation
2022	CTAB-GAN+	Improved conditional GAN with mixed-type encoding, downstream loss terms, and Wasserstein training
2023	TabDDPM [6]	Multinomial + Gaussian diffusion for tabular data; first competitive diffusion approach
2023	STaSy	Score-based tabular generation with self-paced training and feature-specific noise schedules
2023	GOGGLE [7]	Graph-based generation learning column dependencies via GNN message passing
2023	TabSyn	VAE encoder + diffusion in latent space; combines strengths of both paradigms
2023	TabuLa [25]	LLM fine-tuning for tabular generation via text serialization
2023	CoDi	Copula-based diffusion separating marginal and dependency modelling
2024	Harpoon	Conditional generation with formal privacy; privacy-aware column partitioning
2024	DeLTa	Denoising with learned transformations for adaptive column encoding
2024	Tabby	Retrieval-augmented tabular generation using nearest-neighbor guidance
2024	TabStruct	Structure-preserving generation with explicit constraint satisfaction
2024	WaveStitch	Wavelet-domain tabular generation for multi-scale feature capture
2025	Foundation era	Emergence of large pre-trained models for general-purpose table generation and understanding

The early era (2014–2018).

The story begins with Goodfellow's 2014 paper introducing GANs [33]. For the first three years, the community focused exclusively on image generation. The implicit assumption was that GANs were fundamentally a vision tool. It took until 2017 for researchers to ask the obvious question: can we generate tabular data?

The answer was not immediately encouraging. MedGAN [34] showed promise for generating discrete medical codes but struggled with continuous variables. TableGAN applied convolutional architectures – designed for spatial data – to tables, an architectural mismatch that limited performance. TGAN (2018) introduced LSTM-based generation with reversible data transformations, recognizing that tabular data needed its own preprocessing pipeline. But the results were still far from practical utility.

The standardization era (2019–2021).

The breakthrough came in 2019 with CTGAN [4]. Xu et al. identified two key insights: mode-specific normalization handles the multimodal distributions common in real-world columns, and training-by-sampling prevents the generator from ignoring rare categories. CTGAN became the de facto standard – not because it was optimal, but because it was reproducible, well-documented, and released as open-source software (the Synthetic Data Vault library).

TVAE provided a variational alternative within the same framework. DP-CTGAN extended the approach to differential privacy. And in 2021, Bengio et al. introduced GFlowNets [35], opening an entirely new paradigm for discrete structured generation.

The diversification era (2022–2024).

Starting in 2022, the field exploded with diversity. Diffusion models entered the tabular arena with TabDDPM [6], demonstrating that the iterative denoising paradigm could outperform GANs on many benchmarks. Graph-based methods like GOGGLE [7] showed that explicitly modelling column dependencies improved structural fidelity. LLM-based approaches (TabuLa [25], GReaT) demonstrated that language models could generate surprisingly realistic tabular data from text serialization. And hybrid approaches (TabSyn, CoDi) began combining the strengths of multiple paradigms.

By 2024, the consolidation phase produced methods like Harpoon (privacy-aware conditional generation), DeLTa (learned transformations), Tabby (retrieval-augmented generation), and WaveStitch (wavelet-domain generation). Each brought a new perspective, but the field was also beginning to recognize a common core of techniques shared across approaches.

The foundation era (2025–).

As of 2025, the most exciting frontier is the emergence of foundation models for tables – large models pre-trained on diverse tabular corpora that can be adapted to new schemas with minimal data. This represents the culmination of a decade of progress: the domain-specific preprocessing (mode-specific normalization), the training techniques (training-by-sampling, DP-SGD), and the architectural innovations (hybrid diffusion, GNN message passing) are now being integrated into unified, scalable systems.

Exercises

The exercises in this section are organized into two tiers. Exercises (1–12) cover foundational to intermediate material and can be completed with the tools developed in this chapter. Challenges (1–5) are research-level problems that require original thought, creative synthesis, and may connect to open questions in the field.

We have now completed our journey through the mathematics of generative models for structured data. From the humble beginnings of applying image-oriented GANs to tabular data, through the careful engineering of preprocessing pipelines and conditional training, to the diverse landscape of diffusion, graph, flow, and language model approaches, we have seen a field mature rapidly. The exercises above provide entry points for further exploration. The challenges point toward the open frontier. The mathematical tools are in your hands – what you build with them is up to you.