How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Models
α-Precision, β-Recall, and Authenticity provide sample-level, domain-agnostic evaluation of fidelity, diversity, and memorization.
Key Findings
Methodology
The paper defines E=(α-Precision, β-Recall, Authenticity). α-Precision is the probability that a generated point lies in the α-support of the real distribution; β-Recall is the fraction of real points covered by the β-support of the generator; Authenticity tests whether a synthetic point is copied from training data. A one-class neural network creates hyperspheric evaluation embeddings, while sample-level classifiers estimate all quantities.
Key Results
- On SIVEP-Gripe, a Brazilian COVID-19 database with 99,557 patients, the study compares GAN, VAE, WGAN-GP, and ADS-GAN. Using mortality-prediction AUC-ROC on real data as the external ground truth, the ranking is ADS-GAN, WGAN-GP, VAE, GAN; IPα recovers this ranking exactly.
- Standard P1/R1, Density/Coverage, and likelihood-style measures overestimate VAE or WGAN-GP in parts of the experiment. The α-Precision and β-Recall curves expose mode collapse, mode invention, and density shifts that single-point support metrics can miss.
- Theorem 1 states that IPα=IRβ=1 only when Pg=Pr. Authenticity adds a separate memorization test, and the sample-level scores support post-hoc auditing: low-fidelity or unauthentic outputs can be rejected without changing the black-box generator.
Significance
The work changes generative-model evaluation from one opaque score to interpretable diagnostics for fidelity, diversity, and generalization. This matters in healthcare, finance, and privacy-sensitive settings, where a highly realistic record may also be a copied individual record. Because scores are assigned per sample, the framework connects benchmarking with data curation, rejection sampling, and operational auditing.
Technical Contribution
The main technical components are minimum-volume α/β-supports, a one-class neural-network hyperspheric embedding, and a nearest-neighbor hypothesis test for Authenticity. The definitions are Pα=P(Xg∈Sαr) and Rβ=P(Xr∈Sβg), with summaries IPα=1−2∫|Pα−α|dα and IRβ=1−2∫|Rβ−β|dβ. Theorem 1 gives an exact distribution-matching guarantee: both summaries reach one iff Pg=Pr.
Novelty
Unlike image-centric Inception Score, FID, and conventional Precision/Recall, this framework is domain-agnostic, sample-level, and explicitly three-dimensional. Its fundamental novelty is treating training-data copying as an independent axis rather than confusing memorization with fidelity. The same estimators can rank models, diagnose failure modes, and filter individual outputs.
Limitations
- Estimating α/β-supports depends on the embedding Φ, the hypersphere approximation, quantile radii, and the k-nearest-neighbor choice. If high-dimensional or multimodal data cannot be represented adequately by this geometry, local density and mode structure may be distorted.
- Authenticity depends on the distance metric and the noisy-copy model. Distance concentration, duplicates, or legitimate near-duplicates can cause false positives or negatives; the supplied paper excerpt also does not report every numerical result from the full experiment.
Future Work
Future research should develop manifold-aware or adaptive density embeddings, principled calibration for k, α, β, and distances, and evaluations for text, time series, and multimodal data. A broader goal is to connect these metrics formally with privacy leakage, differential-privacy budgets, downstream utility, and uncertainty estimates.
AI Executive Summary
Generative models can now produce convincing images, medical records, and other synthetic data, but visual or statistical plausibility is not enough. Likelihood is often unavailable or misleading in high dimensions, while FID, Inception Score, and standard Precision/Recall compress distinct failures—mode collapse, outliers, density mismatch, and memorization—into scores that are difficult to interpret.
Alaa and colleagues propose a three-dimensional framework: α-Precision, β-Recall, and Authenticity. α-Precision asks whether generated samples are both realistic and typical of the real distribution. β-Recall asks whether the generator covers the major variation present in real data. Authenticity asks whether a sample is genuinely novel rather than a copied training record. Minimum-volume sets define typical regions; a one-class neural network maps data into hyperspheric features; binary sample-level classifiers and nearest-neighbor tests estimate the scores. Integrated measures IPα and IRβ summarize the curves, and both equal one only when Pg=Pr.
The authors test GAN, VAE, WGAN-GP, and ADS-GAN on SIVEP-Gripe, a Brazilian COVID-19 dataset containing 99,557 patients. Logistic regression trained on each synthetic set and evaluated on real data provides an external ranking: ADS-GAN, WGAN-GP, VAE, GAN. IPα reproduces this ranking, whereas P1/R1 and Density/Coverage overestimate some alternatives. The framework also enables model auditing: reject low-fidelity or unauthentic records without retraining the generator. Its geometry and distance assumptions remain important limitations, but the study offers a practical foundation for evaluating synthetic data where realism, coverage, and privacy must be judged together.
Deep Analysis
Background
VAE and GAN research made high-quality generation practical, but many modern models lack tractable likelihoods. Inception Score, FID, conventional Precision/Recall, MMD, and Wasserstein distance provide useful summaries yet are often image-oriented or distribution-level. They may miss mode collapse, invented modes, density calibration, and memorization. Sensitive-data applications require a separate test of whether realistic records are actually copied from training data.
Core Problem
The objective is an evaluator that both diagnoses a generator’s global distribution and assigns a quality decision to each sample. The challenge is that high-dimensional real and synthetic distributions may have different supports, α- and β-supports are difficult to compute directly, and a copied training record can look indistinguishable from an excellent novel sample under ordinary fidelity metrics.
Innovation
- �� α-Precision evaluates membership in the densest α fraction of the real distribution. • β-Recall evaluates coverage by the densest β fraction of the generated distribution. • Authenticity makes non-memorization a third, independent axis. • IPα and IRβ integrate the full curves rather than relying on one operating point. • A theorem proves simultaneous optimality exactly characterizes Pg=Pr. • The same scores enable black-box auditing and rejection sampling.
Methodology
- �� Embed real Xr and generated Xg with Φ. The one-class neural-network loss L=Σ[r²+(1/ν)max{0,||Φ(Xr,i)−cr||²−r²}] compresses real data into a minimum-volume hypersphere. • Estimate Ŝrα=B(cr,r̂α), where r̂α is the α-quantile of real radii; classify generated points inside it to estimate α-Precision. • Center generated features at cg and use a β-quantile radius; for each real point, test whether its nearest typical synthetic point lies within its k-th real-neighbor distance to estimate β-Recall. • For Authenticity, compute each synthetic point’s distance dg,j to training data and compare it with the leave-one-out distance of the nearest training record. If dg,j is unusually smaller, mark it as copied. • Average binary decisions to obtain P̂α, R̂β, and Â.
Experiments
The principal dataset is SIVEP-Gripe, containing 99,557 Brazilian COVID-19 patients and sensitive attributes such as ethnicity. The study compares GAN, VAE, WGAN-GP, and ADS-GAN, which explicitly targets patient identifiability. Baselines include FID, P1/R1, Density/Coverage, Parzen-window likelihood, and Wasserstein distance. A logistic-regression mortality predictor is trained on each synthetic dataset and evaluated on real data; its AUC-ROC supplies an external utility-based ranking.
Results
ADS-GAN gives the best real-data mortality-prediction performance, followed by WGAN-GP, VAE, and GAN. IPα recovers this ordering exactly, while P1, Coverage, and Density overestimate some models. The paper’s schematic examples show that a model can obtain P1=R1=1 while inventing or overproducing an abnormal mode; the Pα and Rβ curves reveal the mismatch. Theorem 1 further guarantees that IPα=IRβ=1 only for identical real and generated densities.
Applications
Healthcare organizations can screen synthetic patient data before release: Authenticity flags possible record copying, α-Precision removes atypical or noisy outputs, and β-Recall checks whether clinically important subgroups are covered. Data vendors can audit black-box generation APIs using sampled outputs. Practical deployment requires a representative reference dataset, a defensible distance metric, and a validated evaluation embedding.
Limitations & Outlook
The framework relies on hyperspheric representations, quantile thresholds, k-nearest-neighbor geometry, and input-space distances; these assumptions may be unstable for discrete, sequential, multimodal, or very high-dimensional data. Authenticity is not a differential-privacy guarantee, and legitimate near-duplicates may be rejected. Adversarial generators could also evade simple nearest-neighbor tests. Future work should add manifold-aware embeddings, confidence intervals, privacy-attack benchmarks, and cross-modal validation.
Plain Language Accessible to non-experts
Imagine a factory making dishes that are supposed to imitate a famous restaurant. A single quality score is like asking only whether each dish looks edible. The factory might repeatedly make one hamburger, or specialize in bizarre edge-case dishes, and still appear successful.
α-Precision is an inspector checking whether factory dishes belong to the restaurant’s most common and reliable menu items. β-Recall reverses the question: of all the restaurant’s dishes, how many kinds can the factory actually make? By changing α and β, the inspector can distinguish normal dishes from rare accidents instead of treating every item equally.
Authenticity checks whether the factory copied an old customer order instead of inventing a new dish. The paper first turns complicated data into a useful map, then uses distances and nearby examples to perform these checks. Because every dish receives its own verdict, the restaurant can discard bad or copied dishes without rebuilding the whole factory. The method therefore evaluates quality and provides a practical cleanup system.
ELI14 Explained like you're 14
Picture a video game that automatically creates new characters. The generator can fail in three different ways: characters may not fit the game, it may keep producing only red warriors, or it may secretly copy characters from your saved game.
α-Precision asks, “How many new characters look like the normal characters in this world?” β-Recall asks, “Of all the character types in the original game, how many are actually represented?” If the generator only makes red warriors, the first answer might look okay, but the second exposes that it cannot make wizards, thieves, healers, or anything else.
Authenticity catches copying. A character can look perfect and still be almost identical to an old saved character, so it should not count as a genuinely new creation. The researchers give every character an individual score, which means they can delete the bad ones without rewriting the generator.
They tested GAN, VAE, WGAN-GP, and ADS-GAN using records from 99,557 Brazilian COVID-19 patients. They trained a death-prediction model on each synthetic dataset and tested it on real patients. ADS-GAN performed best, and IPα correctly recovered the same ranking. Pretty useful! But the referee still depends on how “distance” is measured and how the data are mapped, so it is not a magical perfect judge.
Glossary
α-Precision
The probability that a generated sample lies inside the densest α-support of the real distribution. Unlike ordinary precision, it rewards both realism and typicality.
It is the paper’s sample-level fidelity measure and is evaluated across α values.
β-Recall
The fraction of real samples that lie inside the densest β-support of the generated distribution. It measures how much real variation the generator covers.
It diagnoses insufficient diversity and mode collapse.
Authenticity
The estimated probability that a synthetic sample is novel rather than copied from training data. The paper uses nearest-neighbor distances and a copying hypothesis test.
It measures generalization and potential memorization risk.
Minimum-volume set
The smallest-volume region containing a specified probability mass. It preferentially captures dense, typical parts of a distribution.
It defines the α-support and β-support used by the proposed metrics.
Evaluation embedding
A representation that maps raw data into a space where fidelity and coverage can be measured. The paper learns a hyperspheric representation with a one-class neural network.
It makes support estimation and sample-level classification tractable.
IPα / IRβ
Integrated summaries of the α-Precision and β-Recall curves, each ranging from zero to one. They equal one only when the generated and real distributions match.
They provide compact model-comparison scores.
Open Questions Unanswered questions from this research
- 1 How to construct evaluation embeddings for text, time series, and multimodal data without relying on hyperspheric geometry remains unresolved.
- 2 The quantitative relationship among Authenticity, differential privacy, membership-inference risk, and downstream utility is not yet established.
- 3 Distance metrics, k, α, and β can change conclusions; principled calibration and uncertainty intervals are needed.
Applications
Immediate Applications
Synthetic healthcare-data release
Hospitals and data platforms can compute all three metrics before release, remove low-fidelity or copied patient records, and then verify mortality prediction or other downstream utility on held-out real data.
Black-box generator auditing
Regulators or customers can sample outputs from an API, compare them with an authorized reference dataset, and identify unrealistic samples or possible training-record leakage without inspecting model parameters.
Long-term Vision
Joint privacy-utility certification
The framework could combine fidelity, coverage, Authenticity, differential-privacy budgets, membership-inference tests, and downstream AUC into certification standards for medical, financial, and public-sector synthetic data.
Abstract
Devising domain- and model-agnostic evaluation metrics for generative models is an important and as yet unresolved problem. Most existing metrics, which were tailored solely to the image synthesis setup, exhibit a limited capacity for diagnosing the different modes of failure of generative models across broader application domains. In this paper, we introduce a 3-dimensional evaluation metric, ($α$-Precision, $β$-Recall, Authenticity), that characterizes the fidelity, diversity and generalization performance of any generative model in a domain-agnostic fashion. Our metric unifies statistical divergence measures with precision-recall analysis, enabling sample- and distribution-level diagnoses of model fidelity and diversity. We introduce generalization as an additional, independent dimension (to the fidelity-diversity trade-off) that quantifies the extent to which a model copies training data -- a crucial performance indicator when modeling sensitive data with requirements on privacy. The three metric components correspond to (interpretable) probabilistic quantities, and are estimated via sample-level binary classification. The sample-level nature of our metric inspires a novel use case which we call model auditing, wherein we judge the quality of individual samples generated by a (black-box) model, discarding low-quality samples and hence improving the overall model performance in a post-hoc manner.