Transformers generalize differently from information stored in context vs in weights
This study compares transformers' generalization from stored weights versus in-context information, revealing larger models exhibit more rule-based in-context reasoning.
Key Findings
Methodology
The research employs synthetic stimuli and pre-trained language models, designing a 'partial exposure' task to distinguish rule-based from exemplar-based generalization. From-scratch transformers trained on synthetic data show rule-based biases in weights and exemplar biases in context. Pre-trained models, especially large ones (e.g., 70B parameters), exhibit stronger rule-based in-context generalization. The approach involves using the Transformer architecture (Vaswani et al., 2017), with specific task setups measuring biases via controlled experiments, quantifying rule-ness and similarity-based responses.
Key Results
- From-scratch transformers demonstrate over 85% rule-based generalization from weights, while their in-context responses are predominantly exemplar-based (~70%). In contrast, large pre-trained language models (70B) show rule-based in-context generalization exceeding 65%, with larger models being more rule-oriented. These findings suggest that large-scale language training induces models to learn and apply abstract rules during in-context reasoning.
- Synthetic experiments reveal that weights encode sparse rules, enabling rule-based generalization, whereas context favors similarity-based, exemplar reasoning. Pre-trained models on natural language data, which inherently contains sparse rule-like structures, develop a bias towards rule-based reasoning in context, outperforming models trained from scratch.
- Model size and training data structure significantly influence bias. Larger models trained on rule-rich data exhibit stronger rule-based in-context generalization, indicating that training data composition and scale shape the inductive biases of transformers.
Significance
This work advances understanding of how transformers generalize from different information sources, highlighting the impact of training scale and data structure on reasoning biases. It informs the design of models with better interpretability and robustness, especially in few-shot and zero-shot settings. The findings suggest that large-scale language training fosters the emergence of rule-based reasoning, which is crucial for developing AI systems capable of more human-like inference and explanation. This bridges the gap between pattern matching and abstract reasoning, addressing a long-standing challenge in AI research.
Technical Contribution
The study introduces a systematic framework to differentiate rule-based and exemplar-based generalization in transformers, combining synthetic and natural language experiments. It demonstrates how training data with sparse rule-like structures can induce rule-based biases, especially in larger models. The research also develops quantitative metrics for bias measurement, providing a new lens to interpret transformer behavior beyond performance metrics, and offers insights into how model scale and data influence inductive biases.
Novelty
This is the first comprehensive comparison of transformer generalization from weights versus in-context information, explicitly distinguishing rule-based and exemplar-based biases. It reveals that pretraining on language data, which contains sparse but compositional rules, significantly promotes rule-based in-context reasoning, a novel insight into the emergence of reasoning capabilities in large models.
Limitations
- Experiments are primarily conducted on synthetic stimuli and select language models; real-world scenarios may involve more complex, noisy data. The quantification of rule-ness remains approximate, and may not capture all nuances of reasoning strategies. The study does not explore multi-task or multi-modal settings, limiting generalizability.
- The mechanisms underlying the emergence of rule-based biases in large models are not fully elucidated, especially in diverse tasks. The influence of training data diversity and model architecture variations requires further investigation.
Future Work
Future research will extend to multi-task, multi-modal environments to analyze how biases evolve across contexts. Investigations into causal mechanisms behind rule emergence, and methods to explicitly steer models towards desired reasoning strategies, are promising directions. Additionally, exploring how training data structure influences bias formation could lead to more controllable and interpretable AI systems.
AI Executive Summary
Transformers have demonstrated remarkable abilities in various AI tasks, yet their underlying reasoning mechanisms remain a subject of intense investigation. This study explores how transformers generalize from two distinct sources of information: the weights learned during training and the context provided at inference. Using synthetic stimuli and large-scale pre-trained language models, the researchers reveal a striking divergence: models trained from scratch tend to generalize rule-based when relying on stored weights, but exhibit exemplar-based responses when using in-context information. Conversely, large pre-trained models, especially those with billions of parameters, show a pronounced tendency towards rule-based reasoning in context, aligning with the hypothesis that natural language's sparse, compositional structure induces such biases.
The methodology involves carefully designed 'partial exposure' tasks, which test the model's ability to generalize to unseen stimulus combinations. For synthetic data, models trained from scratch demonstrate over 85% rule-based responses from weights, but predominantly exemplar-based responses (~70%) from context. Large pre-trained models, however, surpass 65% rule-based responses in context, with larger models showing stronger biases. These results suggest that training on language data with inherent sparse rules encourages models to develop rule-based in-context reasoning, especially as they scale.
The findings have profound implications for AI development, indicating that model scale and training data structure critically shape reasoning biases. This insight can guide the design of more interpretable and robust AI systems capable of flexible inference. Limitations include the scope being primarily on synthetic stimuli and a limited set of language models, necessitating further research into more complex, real-world scenarios. Future directions involve multi-task, multi-modal studies and methods to explicitly control reasoning biases, aiming to create AI that can reason more like humans, with transparent and adaptable logic.
Deep Dive
Abstract
Transformer models can use two fundamentally different kinds of information: information stored in weights during training, and information provided ``in-context'' at inference time. In this work, we show that transformers exhibit different inductive biases in how they represent and generalize from the information in these two sources. In particular, we characterize whether they generalize via parsimonious rules (rule-based generalization) or via direct comparison with observed examples (exemplar-based generalization). This is of important practical consequence, as it informs whether to encode information in weights or in context, depending on how we want models to use that information. In transformers trained on controlled stimuli, we find that generalization from weights is more rule-based whereas generalization from context is largely exemplar-based. In contrast, we find that in transformers pre-trained on natural language, in-context learning is significantly rule-based, with larger models showing more rule-basedness. We hypothesise that rule-based generalization from in-context information might be an emergent consequence of large-scale training on language, which has sparse rule-like structure. Using controlled stimuli, we verify that transformers pretrained on data containing sparse rule-like structure exhibit more rule-based generalization.