The Fairness Collapse Phenomenon: Bias Amplification in Language Models Trained on Synthetic Data
Using Qwen2.5-0.5B, recursive training raised EO GAP 13.18→19.38 while PPL improved 16.07→10.55.
Key Findings
Methodology
The study uses 27,752 balanced Bias in Bios biographies to train Qwen2.5-0.5B under human-only, iterative-regeneration, and recursive-contamination regimes. Synthetic biographies come from seeded generation or K=3 few-shot prompting. The authors measure MTLD and Sinkhorn-2 Wasserstein distance, then evaluate perplexity, MMLU, Equality of Opportunity GAP, signed NLL-GAP, CrowS-Pairs, and SoFA.
Key Results
- Under recursive seeded training, EO GAP rose from 13.18 at iteration 0 to 19.38 at iteration 5, while perplexity improved from 16.07 to 10.55. Thus, in-distribution next-token prediction improved while gender fairness deteriorated.
- MMLU declined more gradually, from 42.14 to 32.08. In few-shot training, SoFA bias increased from 0.509 to about 0.714, while CrowS-Pairs stereotype preference also increased with generation, though not monotonically.
- Low-temperature seeded decoding remained anchored to prompts; T=0.6 and 0.9 best matched human continuations, whereas T=1.2 caused semantic drift. Profession-level EO trajectories amplified associations for nurse, professor, and software engineer.
Significance
The paper identifies an early risk that standard collapse indicators can miss. Lower perplexity and relatively stable lexical diversity may coexist with stronger demographic associations. This matters for continued pretraining, web-scale data collection, synthetic annotation, hiring, and ranking systems. Fairness should therefore be monitored as a longitudinal property of the training data lineage, not treated only as a final deployment test.
Technical Contribution
The work operationalizes fairness collapse and separates regenerated-data effects from cumulative parameter effects. In the iterative regime, every training run restarts from Mθ0; in recursive contamination, training continues from Mθt. EO GAP measures group disparities in profession prediction, while signed NLL-GAP measures probability asymmetry. CrowS-Pairs and SoFA provide external likelihood-based validation, making the framework broader than a single fairness score.
Novelty
Most model-collapse studies track distributional tails, diversity, perplexity, or downstream accuracy, whereas bias studies usually analyze static checkpoints. This paper places both processes in one controlled feedback experiment and argues that demographic bias can amplify before conventional language-model collapse becomes severe. Its central innovation is an early-warning perspective on synthetic-data contamination.
Limitations
- The evidence comes from one small model, one English dataset, and binary gender–occupation tasks. Generalization to larger models, multilingual data, and race, age, disability, or intersectional identities remains untested.
- The corpus is professionally balanced and the main experiment fixes 30-token seeds and T=0.9. Real web mixtures, source proportions, prompting, decoding, and filtering may produce different trajectories.
- The results establish a strong empirical pattern but not a complete causal theory; human quality judgments and systematic debiasing controls are limited.
Future Work
Future studies should vary model scale, languages, protected attributes, human/synthetic ratios, decoding temperatures, and filtering policies. The field also needs predictive theory for EO-GAP drift and experiments with provenance tracking, watermarking, counterfactual training, debiased sampling, representation editing, and data-mixture controls.
AI Executive Summary
As machine-generated text enters the open web, future language models may repeatedly learn from their own outputs. Classical model-collapse research focuses on lost diversity, distributional-tail erosion, perplexity, and task accuracy. But a model can become more predictable while becoming less fair. Proskurina, Gourru, and Velcin name this hypothesized early failure mode “fairness collapse.”
Using 27,752 balanced biographies from Bias in Bios, the authors train and regenerate data with Qwen2.5-0.5B. They compare seeded generation, K=3 few-shot generation, iterative regeneration from Mθ0, and recursive continuation from Mθt. MTLD and Sinkhorn-2 Wasserstein distance characterize synthetic text; EO GAP, NLL-GAP, MMLU, CrowS-Pairs, and SoFA assess fairness and general capability.
The key warning is metric divergence. After five recursive seeded iterations, EO GAP increased from 13.18 to 19.38 while perplexity improved from 16.07 to 10.55; MMLU later fell from 42.14 to 32.08. Bias can therefore grow silently before obvious collapse. Synthetic-data pipelines need provenance, human-data retention, and fairness gates, although the findings remain limited to one small model, one English corpus, and gender–occupation bias.
Deep Analysis
Background
Shumailov et al. describe model collapse as recursive self-training that removes low-probability events; Dohmatob et al. connect it to distribution tails and scaling. Separately, Bolukbasi, Zhao, and Nadeem showed that language models encode gendered occupational stereotypes. These research streams have rarely been combined: performance collapse is documented, but whether synthetic exposure accelerates social bias lacks controlled evidence.
Core Problem
The central question is whether repeated reuse of generated biographies weakens, preserves, or amplifies gender–occupation associations. The challenge is temporal and diagnostic: fairness may drift while perplexity, lexical diversity, and task accuracy remain stable. Monitoring only classical collapse metrics can therefore miss an important pre-collapse harm.
Innovation
- ��Defines fairness collapse as an early failure mode.
- ��Uses iterative regeneration, restarting from Mθ0, to isolate data-distribution effects.
- ��Uses recursive contamination, continuing from Mθt, to model parameter-mediated feedback.
- ��Combines EO GAP, NLL-GAP, CrowS-Pairs, and SoFA across prediction and likelihood.
- ��Maps generation behavior using seed lengths 5–50, temperatures 0.3–1.2, MTLD, and Wasserstein distance.
Methodology
- ��Data: Bias in Bios contains roughly 300,000 biographies; the study constructs 27,752 examples uniformly distributed across 28 professions.
- ��Generation: seeded generation conditions on the first k tokens; few-shot prompting supplies K=3 human examples per profession.
- ��Training: compare human-only, iterative, and recursive regimes over multiple iterations.
- ��Representation analysis: Qwen3-Embedding-0.6B embeddings are compared using entropically regularized Sinkhorn-2 Wasserstein distance; ΔW=W2(model, seed)−W2(human, seed).
- ��Fairness: EO GAP averages squared profession-level TPR differences; NLL-GAP, MMLU, CrowS-Pairs, and SoFA add complementary evidence.
Experiments
All controlled experiments use Qwen2.5-0.5B with shared architecture and hyperparameters. Generation analysis varies T=0.3, 0.6, 0.9, 1.2 and seed lengths from 5 to 50 tokens. Main feedback experiments use 30-token seeds and T=0.9, selected for a diversity–fidelity trade-off. Models are evaluated on held-out human biographies, MMLU, CrowS-Pairs, and the gender subset of SoFA.
Results
Recursive seeded training increased EO GAP from 13.18 to 19.38 while PPL fell from 16.07 to 10.55. Signed NLL-GAP moved in magnitude from about 0.093 to 0.106; MMLU decreased from 42.14 to 32.08. Few-shot MMLU fell to 24.63 and SoFA rose from 0.509 to about 0.714. Nurse, professor, and software-engineer trajectories showed directional EO drift despite modest MTLD changes.
Applications
Continued-pretraining teams should gate each data refresh on EO GAP, NLL-GAP, CrowS-Pairs, and SoFA rather than relying on PPL. Web crawlers, synthetic annotation systems, hiring models, resume rankers, and job recommenders should preserve source provenance, separate human from generated text, and test occupation-specific demographic disparities before release.
Limitations & Outlook
The study covers one 0.5B English model, one balanced corpus, and binary gender labels; real web mixtures are more heterogeneous. Prompting, temperature, seed length, and filtering affect generated distributions. Future work should replicate across scales, languages, and identities, add causal decomposition, and test counterfactual training, debiased sampling, representation editing, provenance, and human-data quotas.
Plain Language Accessible to non-experts
Imagine a restaurant whose first menu is based on thousands of real customer recipes. Later, the chef starts copying dishes made by the restaurant itself, then uses those copies to design the next menu. The chef will repeat familiar flavors because they are easiest to reproduce. Rare tastes gradually disappear, while an old assumption—such as “this kind of customer prefers that kind of dish”—can become stronger simply through repetition.
Sales reports may even improve: the chef predicts the next popular order more confidently. Yet the menu is narrower and less fair to different customers. That is the paper’s warning. A language model may sound fluent and score better on its training distribution while quietly assigning different opportunities to biographies associated with different genders.
The lesson is simple: checking whether an AI sounds polished is not enough. Developers must also check whether it treats groups equally, track which text came from people or machines, and keep enough human-written material in the loop. Otherwise the system becomes an echo chamber: increasingly confident, increasingly repetitive, and increasingly biased.
ELI14 Explained like you're 14
Think of an AI game that keeps generating its own maps. The first map was drawn by real players, so it has highways, weird shortcuts, and unpopular paths. Then the AI draws a new map, trains on it, draws another map, and repeats. Sounds efficient, right?
Here is the catch: the AI loves routes it already knows. Small paths disappear, while stereotypes such as “boys are engineers and girls are nurses” may get copied again and again. The map can still look complete, and the AI may become better at predicting its usual routes—but players with different characters no longer get equal treatment!
The paper tested this with Bias in Bios biographies and Qwen2.5-0.5B. After five recursive rounds, the fairness gap rose from 13.18 to 19.38, even though perplexity improved from 16.07 to 10.55. The general MMLU score fell later, from 42.14 to 32.08.
So an AI can pass ordinary tests while secretly becoming more biased. It is like getting higher quiz scores while grading one group of students more harshly. Future AI training needs fairness checks every round, not just a final fluency test—and it must record which examples were written by humans and which by machines!
Glossary
Fairness collapse
A hypothesized process in which recursive synthetic-data training progressively amplifies social bias. It may appear before severe language-model degradation.
The paper’s central phenomenon, measured mainly through gender–occupation disparities.
Model collapse
Progressive deviation from the original data distribution after models repeatedly train on earlier model outputs. It is commonly tracked through diversity, perplexity, and downstream performance.
The established failure mode contrasted with fairness collapse.
EO GAP
A disparity in true-positive rates between demographic groups conditioned on the correct class. Here it is the average squared profession-level difference.
Primary Bias in Bios fairness metric.
NLL-GAP
A signed difference between average negative log-likelihoods for gender-associated biographies. It captures demographic asymmetry directly in the model’s probability distribution.
Used alongside EO GAP to detect likelihood bias.
Seeded generation
Generation conditioned on the first k tokens of a human biography, followed by a model-written continuation. The study varies k from 5 to 50.
Main controlled synthetic-data construction.
MTLD
Measure of Textual Lexical Diversity, an estimate of vocabulary richness less sensitive to document length than raw type-token ratio.
Used to compare surface diversity of human and synthetic biographies.
Open Questions Unanswered questions from this research
- 1 It remains unknown whether the phenomenon scales to larger models, multilingual corpora, and attributes such as race, age, disability, or intersectional identity; broader replication is required.
- 2 The causal roles of prompting, decoding, data selection, and parameter inheritance are not separated fully. Controlled mixture experiments and intervention studies are needed to identify durable safeguards.
Applications
Immediate Applications
Continued-pretraining audit
After every synthetic-data refresh, model teams can measure EO GAP, NLL-GAP, CrowS-Pairs, and SoFA alongside PPL. A fairness regression should pause training, trigger data review, and preserve generation provenance.
Hiring-system release gate
Recruiting, resume-ranking, and job-recommendation systems can use Bias in Bios profession pairs to compare gender-conditioned likelihoods and true-positive rates. This helps detect allocation harm even when fluency and accuracy improve.
Long-term Vision
Traceable synthetic-data ecosystem
Future data platforms could combine provenance labels, watermarking, human-data quotas, and continuous fairness monitoring. Such infrastructure would allow generation-by-generation tracing and automatically interrupt bias-amplifying feedback loops.
Abstract
Generative models trained on artificially generated data have been shown to exhibit model collapse, resulting in significant performance degradation. As synthetic content increasingly contaminates the training corpora of language models, this raises critical concerns about the use of open data in continued pretraining. Although previous work has demonstrated model collapse in language models, it remains unclear whether exposure to synthetic data amplifies or attenuates the social biases already present in pretrained models. Because language models are known to reproduce and amplify demographic stereotypes, recursive training on self-generated data may create a self-reinforcing feedback loop in which biased associations become progressively stronger across generations. We call this hypothesized phenomenon fairness collapse. In this work, we construct controlled training regimes in which models are repeatedly trained on synthetic data using the Bias in Bios dataset. Across experiments, we observe a consistent and concerning pattern: fairness degradation emerges before substantial degradation is reflected by standard language-modeling metrics. This result highlights a critical risk associated with synthetic data contamination in language model training: bias can increase silently before strong indicators of model collapse become apparent.