Linear representations of grammaticality in neural language models

The development of neural language models (NLMs) such as GPT, Llama, and BERT has revolutionized NLP, enabling unprecedented performance in text generation and understanding. Early research focused on probabilistic assessments, where higher likelihoods of sentences were taken as indicators of grammaticality. However, these methods face criticism because sentence probability is influenced by lexical frequency, sentence length, and world knowledge, confounding true syntactic understanding. Recent studies have explored internal representations, activation patterns, and circuit analysis, revealing that models encode some syntactic information. Yet, a definitive, scalable method to quantify and interpret this internal knowledge remains elusive. Understanding whether models develop an abstract, linearly accessible notion of grammaticality is crucial for interpretability, robustness, and cross-lingual transfer, especially as models grow larger and more complex.

Despite evidence of syntactic information in neural models, it remains unclear whether grammaticality is internally represented as a distinct, linearly separable dimension. Existing approaches relying on sentence probabilities are limited by their entanglement with lexical and contextual factors. The core challenge is to develop a method that isolates grammaticality as an abstract property within the internal embedding space, independent of surface features. This is essential for validating models' syntactic competence and for building interpretable systems capable of explicit syntactic reasoning. Moreover, establishing cross-phenomena and cross-lingual generalization is vital for demonstrating the universality of these internal representations, which has been lacking in prior work.

The key innovation is the application of mass-mean probing—a simple, interpretable linear classifier—to sentence embeddings, enabling direct detection of a grammaticality dimension. Unlike complex neural circuit analysis, this method requires minimal data and assumptions, making it scalable and transparent. The approach involves computing mean vectors for grammatical and ungrammatical sentences, then deriving a difference vector that defines a linear boundary. This boundary is tested across multiple models, phenomena, and languages, demonstrating that grammaticality is encoded as a stable, continuous dimension. The method's simplicity and interpretability distinguish it from prior probabilistic and activation-based techniques, offering a new paradigm for internal representation analysis.

• �� Collect sentence embeddings from 25 pretrained models across multiple layers, focusing on the final hidden state. • Use datasets like BLiMP and SCaMP to obtain balanced grammatical and ungrammatical sentences, controlling for confounding factors. • Compute mean vectors for each class, then derive the difference vector θ. • Fit a linear classifier g(x) = σ(θ·x) to predict grammaticality, ensuring control over sentence properties. • Validate the classifier on held-out datasets, assessing success via accuracy and significance tests. • Analyze the relationship between model size and encoding strength, and test cross-phenomena and cross-lingual generalization.

• �� Use datasets spanning multiple syntactic phenomena, including agreement, island effects, and filler-gap dependencies, in English, Russian, and Chinese. • Fit probes on a subset of sentences, then evaluate on the remaining data, ensuring control for confounding factors. • Measure success by classification accuracy, statistical significance, and comparison with probability-based metrics. • Conduct ablation studies to assess the impact of model size, layer choice, and sentence properties. • Test across diverse phenomena and languages to evaluate the universality and robustness of the grammaticality dimension.

• �� All models exhibit significant linear separability of grammatical vs. ungrammatical sentences, with success rates from 13.4% (small models) to 94% (large models). • Larger models show stronger encoding, with correlation R=0.79 between size and success rate. • The encoding generalizes across multiple phenomena and languages, indicating a universal internal grammaticality dimension. • Controlling for lexical frequency, sentence length, and world knowledge, the separation remains significant, confirming the abstraction of the syntactic dimension. • Linear probes outperform probability-based measures, especially in larger models, highlighting the internal structure's importance.

• �� Enhances interpretability of language models by revealing internal syntactic representations, aiding explainability and debugging. • Supports multilingual NLP by providing a universal measure of syntactic understanding applicable across languages. • Can inform training strategies to reinforce syntactic encoding, improving model robustness and linguistic fidelity. • Facilitates development of models with explicit syntactic reasoning capabilities, advancing natural language understanding.

• �� Focused on sentence-level grammaticality, leaving hierarchical and long-distance dependencies less explored. • The causal relationship between model size and encoding strength is not fully established. • Cross-lingual results, while promising, vary with language typology, requiring broader multilingual validation. • The method does not specify the neural circuits responsible for encoding, necessitating further circuit-level analysis.