Language models suffer from a curse of ambiguity

Over the past decade, the development of Transformer-based models such as GPT, BERT, and their variants has dramatically advanced NLP capabilities. These models leverage massive datasets and deep architectures to learn rich contextual representations, enabling tasks like translation, summarization, and question answering. However, as models grow larger and more complex, understanding their fundamental limitations becomes increasingly critical. Prior work has identified issues like the softmax bottleneck, where limited embedding dimensions restrict the diversity of output distributions. Recent studies also highlight the challenge of modeling multi-modal and multi-meaning phenomena, but a systematic theoretical framework quantifying the impact of distribution ambiguity remains lacking. This paper addresses this gap by formalizing the notion of distribution ambiguity through support set size and analyzing its implications using tools from information theory and compressed sensing, thus providing a rigorous foundation for understanding the statistical limits of neural language models.

Despite the impressive empirical performance of large language models, their ability to accurately learn and represent highly ambiguous distributions remains limited. When multiple plausible next tokens exist—such as in contexts with high polysemy or multi-modal continuations—the models tend to leak probability mass outside the true support, leading to inaccurate predictions and reduced calibration. This problem is rooted in the finite capacity of neural networks, especially in their embedding layers, which constrains their ability to distinguish multiple equally likely outcomes. The core challenge is to quantify how the ambiguity level, measured by support set size k, affects the model’s capacity, learning speed, and distribution fidelity. Existing approaches often overlook this fundamental bottleneck, focusing instead on architectural improvements or training heuristics, thus leaving a theoretical understanding of the limits imposed by ambiguity underdeveloped.

This paper introduces several key innovations: 1) Quantification of distribution ambiguity via support set size k, providing a precise measure of multi-modality in next-token distributions. 2) Application of compressed sensing principles to neural network theory, deriving that model capacity (parameter count) must scale linearly with k to store and distinguish multiple outcomes. 3) Demonstration that embedding dimension h must also grow proportionally with k, establishing a geometric constraint on representational capacity. 4) Empirical validation across synthetic and real datasets, confirming the theoretical scaling laws and revealing the impact of ambiguity on training dynamics, distribution leakage, and learning speed. 5) Analysis of the effects of sampling noise, showing that high ambiguity amplifies stochastic gradient variance, further impairing training efficiency. These contributions collectively deepen our understanding of the fundamental statistical limits faced by neural language models under ambiguous conditions.

• �� Define the ambiguity of a distribution via support set size k, modeling the next-token distribution as uniform over a randomly chosen subset of size k.
• �� Use a linear model (Hebbian-like construction) to analyze the capacity needed to store multiple support sets, deriving that capacity scales as hd/k.
• �� Apply compressed sensing theory to establish that the embedding dimension h must satisfy h ≥ 2k for exact representation of size-k distributions.
• �� Conduct synthetic experiments with varying support sizes to empirically verify the linear scaling of capacity and embedding dimension.
• �� Extend the analysis to real transformer models trained on natural language datasets, measuring distribution leakage, KL divergence, and training dynamics across contexts with different support sizes.
• �� Analyze training dynamics using gradient flow equations, revealing how ambiguity slows convergence and amplifies sampling noise, supported by theoretical derivations and simulations.

• �� Synthetic experiments involve constructing datasets with support sizes from 10 to 1000, training linear models to validate the capacity scaling law.
• �� Real data experiments use a 4-layer transformer trained on TinyStories 3-gram dataset, analyzing the relationship between context support size and model performance.
• �� Ablation studies vary model parameters such as total parameters (from 10K to 1M), embedding dimensions (32 to 512), and training steps, to observe their effects on distribution fidelity.
• �� Performance metrics include KL divergence, probability leakage outside the support, and entropy gap between true and predicted distributions.
• �� Training dynamics are analyzed through differential equations modeling gradient flow, confirming that higher support sizes slow convergence and increase noise, consistent with theoretical predictions.

• �� Theoretical derivations confirmed that model capacity scales linearly with support set size k, with empirical results showing that increasing k from 10 to 1000 requires proportionally more parameters to maintain performance.
• �� In synthetic tasks, models with limited capacity leak probability mass outside the support as k increases, validating the predicted distribution leakage phenomenon.
• �� Embedding dimension experiments demonstrate that to represent distributions with support size k, h must be at least 2k, aligning with compressed sensing bounds.
• �� In real datasets, models trained on contexts with larger support sizes exhibit higher KL divergence and slower convergence, confirming the impact of ambiguity on learning efficiency.

• �� The findings inform the design of calibration strategies for language generation, enabling models to better handle multi-modal and ambiguous contexts.
• �� Improve dialogue systems by dynamically adjusting model capacity or embedding size based on context ambiguity, enhancing response diversity and accuracy.
• �� Develop ambiguity-aware training protocols that allocate resources adaptively, reducing distribution leakage and improving robustness in real-world applications.
• �� Extend to multi-modal and multi-task learning environments, where ambiguity arises from complex data interactions, to foster more flexible and scalable AI systems.

• �� The primary analysis relies on simplified linear models and synthetic datasets; the behavior of deep nonlinear transformers may introduce additional complexities not captured here.
• �� Support set size k is a coarse, uniform measure of ambiguity; real language distributions often exhibit non-uniform, context-dependent structures that require more nuanced modeling.
• �� In extremely high ambiguity regimes, the linear scaling laws may break down, and computational costs for training and inference could become prohibitive, limiting practical deployment.