The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets
This study demonstrates that large language models (LLMs) linearly encode truth/falsehood in their internal representations, validated through transfer and causal intervention experiments, especially at scale.
Key Findings
Methodology
This work employs curated high-quality datasets of simple true/false statements, combining principal component analysis (PCA) visualizations, transfer learning experiments, and causal interventions. Probes trained on one dataset are tested for generalization across different datasets, assessing the universality of the truth representation. Residual stream activations are patched to localize causally relevant hidden states, and linear directions are manipulated to observe causal effects on model outputs. The core focus is on whether models develop an abstract ‘truth’ space that generalizes across diverse inputs and scales.
Key Results
- In LLaMA-2-70B, the true/false representations form a clear linear separation in PCA space, with larger models showing more abstract and generalizable truth encodings. Transfer experiments reveal that probes trained on one dataset achieve over 95% accuracy when applied to structurally different datasets, indicating a shared linear truth space.
- Causal intervention experiments demonstrate that shifting the hidden states along identified linear directions can flip the model’s judgment of statements, e.g., turning false statements into true ones and vice versa. These directions are localized to a small set of hidden states, especially in the final layers, which encode the model’s explicit truth/falsehood prediction.
- Simple difference-in-means probes outperform more complex methods in causal relevance, providing a robust, interpretable way to identify the directions most causally implicated in the model’s outputs. These findings suggest that the internal ‘truth’ space is not only linearly represented but also causally operative.
Significance
This research addresses a fundamental question in AI interpretability: do large language models internally represent factual truth in a structured, manipulable way? The evidence that truth is linearly encoded and causally accessible at scale advances our understanding of how models store and utilize knowledge. It provides a pathway for developing more transparent, trustworthy AI systems by enabling targeted interventions and explanations. Moreover, the methodology bridges the gap between statistical correlation and causal inference, offering a new toolkit for model interpretability. These insights are crucial for deploying AI in high-stakes environments where understanding the basis of decisions is essential.
Technical Contribution
The paper introduces a comprehensive framework combining high-quality datasets, PCA visualization, transferability tests, and causal interventions to analyze internal representations. The key technical innovation is the use of simple difference-in-means probes, which are computationally efficient and causally more relevant than traditional logistic regression probes. The study also systematically examines the evolution of truth representations across model scales, revealing a hierarchical development from surface features to abstract concepts. These contributions provide new theoretical and practical tools for interpretability research, emphasizing causal operability of internal spaces.
Novelty
This work is the first to rigorously demonstrate that large-scale LLMs develop a linear, causally relevant ‘truth’ space that generalizes across diverse datasets and is manipulable via targeted interventions. Unlike prior studies that relied solely on correlation-based probes, this research employs causal experiments to validate the functional importance of identified directions. The integration of transfer learning, visualization, and causal manipulation offers a holistic view of internal knowledge representation, setting a new standard in interpretability and causal analysis of neural models.
Limitations
- The datasets used are limited to simple, unambiguous factual statements; the approach’s effectiveness on complex, multi-hop reasoning or ambiguous statements remains untested. This limits the scope of the current findings to straightforward factual knowledge.
- Causal interventions focus on linear directions in residual stream activations, which may oversimplify the potentially distributed or nonlinear nature of internal representations, leaving open the question of how these findings extend to more complex or entangled representations.
- While large models show strong linear truth encoding, smaller models or models trained on different architectures may not exhibit the same properties, indicating a potential scale-dependence and architecture-specific effect that warrants further investigation.
Future Work
Future research will explore the generalization of linear truth spaces to more complex reasoning tasks, multi-modal data, and real-world scenarios. Developing methods to identify and manipulate nonlinear or distributed representations of truth will be crucial. Additionally, integrating these insights into model training and fine-tuning processes could improve factual accuracy and robustness. Extending causal analysis to other internal components and understanding the developmental trajectory of truth representations across training stages are promising directions. The ultimate goal is to build models with transparent, causally grounded internal knowledge structures that can be reliably controlled and interpreted.
AI Executive Summary
In the rapidly advancing field of artificial intelligence, understanding how large language models (LLMs) internally represent knowledge remains a central challenge. Despite their impressive performance across various NLP tasks, these models often produce false or misleading outputs, raising concerns about their reliability and interpretability. Existing interpretability methods, such as probing internal activations with logistic regression, have yielded mixed results, with questions about their generalization and causal relevance.
This study by Samuel Marks and Max Tegmark addresses this gap by systematically investigating whether LLMs develop an internal, linear representation of factual truth. Using high-quality datasets of simple true/false statements, the authors employ a multi-pronged approach: visualizing internal representations with PCA, testing transferability of probes trained on one dataset to others, and performing causal interventions by manipulating hidden states. Their findings reveal that, at sufficient scale, LLMs encode truth and falsehood in a linearly separable space, which generalizes across diverse datasets and is causally implicated in the model’s outputs.
The core technical innovation is the use of a simple, yet powerful, difference-in-means probe that identifies directions in the internal space most causally related to the model’s judgments. These directions, localized to a small set of hidden states, can be manipulated to flip the model’s output, providing strong causal evidence for the internal encoding of truth. The experiments demonstrate that larger models, such as LLaMA-2-70B, exhibit more abstract and generalizable truth representations, highlighting the role of scale in emergent capabilities.
Beyond technical insights, this work has profound implications for AI safety, transparency, and trustworthiness. It suggests that models do not merely memorize facts but develop structured, manipulable internal spaces for factual knowledge. This opens avenues for targeted interpretability, bias mitigation, and robust knowledge editing. However, the study also acknowledges limitations: the focus on simple factual statements, the linearity assumption, and the need to extend findings to more complex reasoning.
Looking ahead, future work will explore how these linear truth spaces evolve during training, how they can be extended to multi-hop reasoning, and how to leverage them for building more transparent AI systems. Overall, this research marks a significant step toward demystifying the internal mechanics of large language models, paving the way for more trustworthy and controllable AI technologies.
Deep Dive
Abstract
Large Language Models (LLMs) have impressive capabilities, but are prone to outputting falsehoods. Recent work has developed techniques for inferring whether a LLM is telling the truth by training probes on the LLM's internal activations. However, this line of work is controversial, with some authors pointing out failures of these probes to generalize in basic ways, among other conceptual issues. In this work, we use high-quality datasets of simple true/false statements to study in detail the structure of LLM representations of truth, drawing on three lines of evidence: 1. Visualizations of LLM true/false statement representations, which reveal clear linear structure. 2. Transfer experiments in which probes trained on one dataset generalize to different datasets. 3. Causal evidence obtained by surgically intervening in a LLM's forward pass, causing it to treat false statements as true and vice versa. Overall, we present evidence that at sufficient scale, LLMs linearly represent the truth or falsehood of factual statements. We also show that simple difference-in-mean probes generalize as well as other probing techniques while identifying directions which are more causally implicated in model outputs.
References (20)
LLaMA: Open and Efficient Foundation Language Models
Hugo Touvron, Thibaut Lavril, Gautier Izacard et al.
Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Models
Matthew Finlayson, Aaron Mueller, Stuart M. Shieber et al.
Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task
Kenneth Li, Aspen K. Hopkins, David Bau et al.
Discovering Latent Knowledge in Language Models Without Supervision
Collin Burns, Haotian Ye, D. Klein et al.
The Internal State of an LLM Knows When its Lying
A. Azaria, Tom M. Mitchell
Inference-Time Intervention: Eliciting Truthful Answers from a Language Model
Kenneth Li, Oam Patel, Fernanda B. Viégas et al.
Still no lie detector for language models: probing empirical and conceptual roadblocks
B. A. Levinstein, Daniel A. Herrmann
Explore, Establish, Exploit: Red Teaming Language Models from Scratch
Stephen Casper, Jason Lin, Joe Kwon et al.
What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Models
Fahim Dalvi, Nadir Durrani, Hassan Sajjad et al.
What Is Data Science
Longbing Cao
Implicit Representations of Meaning in Neural Language Models
Belinda Z. Li, Maxwell Nye, Jacob Andreas
Multimodal Neurons in Artificial Neural Networks
Gabriel Goh, Nick Cammarata, Chelsea Voss et al.
Probing Classifiers: Promises, Shortcomings, and Advances
Yonatan Belinkov
Understanding the role of individual units in a deep neural network
David Bau, Jun-Yan Zhu, Hendrik Strobelt et al.
Neural Natural Language Inference Models Partially Embed Theories of Lexical Entailment and Negation
Atticus Geiger, Kyle Richardson, Christopher Potts
Designing and Interpreting Probes with Control Tasks
John Hewitt, Percy Liang
Investigating Gender Bias in Language Models Using Causal Mediation Analysis
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov et al.
Mapping Language Models to Grounded Conceptual Spaces
Roma Patel, Ellie Pavlick
The Implicit Bias of Gradient Descent on Separable Data
Daniel Soudry, Elad Hoffer, Suriya Gunasekar et al.
TruthfulQA: Measuring How Models Mimic Human Falsehoods
Stephanie C. Lin, Jacob Hilton, Owain Evans
Cited By (20)
Linear representations of grammaticality in neural language models
Language Models Encode the Contextual Truth of Propositions
On the Robustness of LLMs'Internal Representation of Code Correctness
STEREODISCO: Discovering Stereotypicality in LLMs
Efficient LLM Adversarial Training via Low-Rank Defense and Circuit-Guided Surrogates
The Refusal Residue: When Probes Catch Alignment Faking and When They Don't
Steering Robustness into World Action Models via Mechanistic Interpretability and Optimal Control
How Does Alignment Tuning Shape Representations of Sycophancy and Related Cue-Induced Biases in LLMs?
The Anatomy of a Truth Direction: Knowledge-Dependent Dimensionality, a Relational Law, and a Convergent Category Geometry in Small Language Models
Operational Proto-Introspection in Looped Language Models: Process-Quality Taps, Executable Branching, and the Readout-Control Boundary
Diagnosing Correctness Probes under Self-Judgement Confounding
Peirce's revenge on the Chinese Room
Misalignment Has a Personality: A Big Five Account of Emergent Misalignment
Isotropy Cliffs: The Geometric Signature of Decision-Making in Large Language Models
Cloud-ScPO: Hidden-State Geometry for Semi-Supervised Preference Optimization in LLM Reasoning
Probing the 3D Object-Level Understanding of Pre-Trained Detection Transformers
Semantic Alignment of AI Models: Concept Collapse, Checkpoint Dynamics, and Cross-Lingual Transfer
Cultural Awareness is Represented but Not Decoded: Tracing Mythological Knowledge across 18 Open-Source LLMs
When Outputs Disperse, Does Epistemic Revision Follow? A Black-Box Diagnostic for Machine Collectives
Test, then Route: How Language Models Execute In-Context Conditional Rules Across Models and Languages