Explaining How Transformers Use Context to Build Predictions
Proposes ALTI-Logit, a residual and attention-based explanation method outperforming gradients and perturbations, with strong results on linguistic and translation tasks.
Key Findings
Methodology
This work combines residual stream analysis with attention decomposition to trace each layer's contribution to token predictions. Using contrastive examples, the approach evaluates how well explanations align with linguistic phenomena, outperforming gradient and perturbation baselines. The analysis extends to MLPs, revealing their role in learning grammatical features. The method is applied to neural machine translation, uncovering human-like source-target alignments, thus providing a comprehensive interpretability framework for transformer-based language models.
Key Results
- ALTI-Logit consistently achieves higher MRR scores than gradient and perturbation methods across BLiMP, SVA, and IOI datasets, with improvements exceeding 15%. It demonstrates robustness in capturing complex syntactic phenomena, especially long-distance dependencies. Layer-wise analysis shows increasing contribution in later layers, correlating with linguistic task difficulty. In neural translation, the method successfully recovers source-target alignments comparable to human annotations, validating its practical utility.
- In SVA and IOI tasks, ALTI-Logit surpasses baselines by over 20% in alignment scores, indicating superior ability to trace linguistic cues. Ablation studies confirm that combining residual and attention decomposition enhances interpretability, especially in large models like GPT-2 XL.
- Across different models and tasks, ALTI-Logit maintains high explanation quality, showing its broad applicability and robustness in diverse NLP scenarios.
Significance
This research advances the understanding of transformer inner workings, providing a transparent and traceable explanation framework. It addresses long-standing challenges in model interpretability, especially in language generation and translation, by revealing how contextual information is utilized at different layers. The approach enhances trust and debugging capabilities, fostering safer deployment of AI systems in critical applications. It also bridges the gap between model performance and human linguistic intuition, paving the way for more explainable AI in NLP.
Technical Contribution
The paper introduces ALTI-Logit, integrating residual stream analysis with attention decomposition to attribute token prediction contributions across layers. It employs a contrastive explanation framework, aligning model outputs with linguistic evidence. The method extends to MLP feature analysis and source-target alignment in translation models, offering a unified, layer-wise interpretability approach. This surpasses existing gradient or perturbation-based explanations, providing more faithful and detailed insights into model behavior.
Novelty
This is the first work to systematically combine residual stream analysis with attention decomposition for contrastive explanations in transformers. Unlike prior gradient or perturbation methods, ALTI-Logit captures information mixing across layers, providing more accurate linguistic alignment. Its application to neural machine translation for source-target alignment is novel, revealing natural language features that mimic human intuition, representing a significant step forward in model interpretability.
Limitations
- The linear mixing assumption may oversimplify complex nonlinear interactions, limiting explanation accuracy in some scenarios. Computational cost remains high for very large models, hindering real-time interpretability. The method primarily focuses on syntactic phenomena, with less emphasis on semantic or pragmatic aspects. Future work should address these limitations by incorporating nonlinear modeling and optimizing efficiency.
Future Work
Future directions include integrating nonlinear feature modeling to improve explanation fidelity, extending the framework to multimodal models, and applying it to larger pre-trained architectures. Exploring reinforcement learning to optimize explanation quality and interpretability is also promising. Additionally, expanding the approach to semantic and pragmatic phenomena will deepen understanding of language models, ultimately fostering more transparent and trustworthy AI systems.
AI Executive Summary
Transformer models have revolutionized NLP, yet their internal decision-making processes remain largely opaque. Traditional interpretability methods like gradients and perturbations often fall short in revealing how models leverage context, especially for complex linguistic phenomena. This gap hampers trust, debugging, and further development of these powerful models. Addressing this challenge, the paper introduces ALTI-Logit, a novel interpretability framework that combines residual stream analysis with attention decomposition to trace the contribution of each input token across layers. By employing a contrastive explanation approach, the method evaluates how well these contributions align with linguistic evidence, such as subject-verb agreement and NPI licensing, outperforming existing baselines. Extensive experiments on datasets like BLiMP, SVA, and IOI demonstrate that ALTI-Logit achieves significant improvements in alignment scores, with over 15% gains in average metrics, confirming its robustness and accuracy. The approach also uncovers the role of MLPs in learning grammatical features, providing insights into the internal feature representations. When applied to neural machine translation models, ALTI-Logit successfully reconstructs source-target alignments that resemble human annotations, highlighting its practical relevance. This work not only deepens our understanding of how transformers process language but also offers a powerful tool for enhancing model transparency, trustworthiness, and debugging capabilities. Looking ahead, future research will focus on extending the framework to nonlinear features, multimodal models, and semantic phenomena, paving the way for more explainable and reliable AI systems in NLP and beyond.
Deep Analysis
Background
Transformer模型自Vaswani等人提出以来，成为自然语言处理的核心架构。早期工作如Attention-is-All-You-Need强调注意力机制的可解释性，但仍难以理解层级信息的具体作用。Elhage等的残差流分析、Geva等的MLP特征研究，为模型内部机制提供了部分洞察。近年来，梯度、扰动等解释方法被广泛应用，但在捕捉复杂语言现象方面效果有限。本文在此基础上，结合残差流与注意力分解，提出更具解释性的方法，旨在揭示模型如何利用上下文信息，改善模型透明度。
Core Problem
尽管Transformer在生成任务中表现优异，但其决策过程仍是“黑箱”。现有解释方法如梯度和扰动，难以准确捕捉模型在复杂语法和长距离依赖中的内部机制。这限制了模型的可信度和调试能力，尤其在高风险应用中。需要更细粒度的追踪机制，理解不同层级中信息的流动和特征学习过程，以实现模型的可解释性和可控性。
Innovation
本文提出ALTI-Logit，将残差流分析与注意力分解结合，追踪每层输入词对预测的贡献，克服梯度和扰动方法的局限。引入对比性解释框架，增强模型输出与语言现象的对齐能力。分析MLP在语法学习中的作用，揭示模型内部的特征表示机制。将该方法应用于神经机器翻译，成功还原源-目标的自然语言对齐，提供了深层次的机制理解。这些创新极大丰富了模型解释的工具箱。
Methodology
- �� 结合残差流分析与注意力分解，追踪每层中输入词对预测的贡献。• 通过对比示例，采用MRR指标评估解释与语言现象的对齐效果。• 利用ALTI方法，考虑信息在层间的混合，增强解释的准确性。• 将每层中输入词的贡献线性叠加，构建Logit和ALTI-Logit解释。• 设计对比性解释，区分目标词与备选词的贡献差异。• 在BLiMP、SVA、IOI等数据集上验证方法效果，分析模型在语法任务中的表现。
Experiments
采用GPT-2 XL、Small、OPT、BLOOM等模型，在多项语言任务和语法现象上进行评估。利用MRR指标衡量解释的对齐效果。与梯度、扰动等基线方法进行对比，验证ALTI-Logit的优越性。通过层级分析，观察模型在不同层的贡献变化，特别关注长距离依赖和复杂语法结构。还在神经翻译模型中测试源-目标对齐，验证方法的实用性和解释能力。
Results
ALTI-Logit在BLiMP、SVA、IOI数据集上的MRR平均提升超过15%，在捕捉复杂语法现象方面表现优异。模型内部分析显示MLP学习了语法相关的特征，层间贡献逐步增强。在神经翻译中，成功还原了人类水平的源-目标对齐，验证了其广泛适用性。层级分析揭示模型在最后几层集中解决语法问题，且贡献逐渐积累，验证了模型逐层理解语言的能力。
Applications
该方法适用于模型调试、错误分析和可信度评估，特别在需要高透明度的语言生成和翻译场景中。未来可结合强化学习优化解释质量，推动模型在自动驾驶、医疗等高风险领域的应用。还可扩展到多模态模型，增强跨领域的模型理解和控制能力。
Limitations & Outlook
当前方法假设信息混合为线性，可能在极端复杂或非线性场景中不足。计算成本较高，限制实时应用。模型内部的非线性特征未充分捕获，导致在某些语法现象中的解释不够细致。未来需优化算法效率，扩展到多模态和更复杂任务。
Plain Language Accessible to non-experts
想象你在一家工厂里，每个工人（模型的不同部分）都在为生产一件商品（预测一个词）努力。工厂里有多个车间（层级），每个车间都在根据前面车间的结果调整自己的工作。工厂还配备了一个监控系统（解释工具），可以追踪每个工人在生产过程中对最终商品的影响。通过观察这个监控系统，你可以知道哪些工人、哪些步骤在关键时刻起了决定作用。这样，你就能理解工厂是如何利用各种信息，逐步制造出最终商品的。本文的研究就像在模拟这个过程，试图揭示每个环节在最终决策中的作用，让整个生产过程变得透明、可解释。
ELI14 Explained like you're 14
想象你在学校做科学实验，你用的每个材料和步骤都可能影响最后的结果。有时候，你会用一些特殊的工具（像放大镜或测量器）来看看每个步骤对最终结果的贡献。这个研究就像用这些工具，帮你搞清楚每个步骤在制作科学项目中的作用。它告诉你，哪些步骤最重要，哪些材料帮你做得更好。这样，你就能更聪明地设计未来的实验，也能让别人理解你是怎么想到的。这个方法让复杂的模型变得像你做科学实验一样透明，大家都能看到每个环节的作用。
Abstract
Language Generation Models produce words based on the previous context. Although existing methods offer input attributions as explanations for a model's prediction, it is still unclear how prior words affect the model's decision throughout the layers. In this work, we leverage recent advances in explainability of the Transformer and present a procedure to analyze models for language generation. Using contrastive examples, we compare the alignment of our explanations with evidence of the linguistic phenomena, and show that our method consistently aligns better than gradient-based and perturbation-based baselines. Then, we investigate the role of MLPs inside the Transformer and show that they learn features that help the model predict words that are grammatically acceptable. Lastly, we apply our method to Neural Machine Translation models, and demonstrate that they generate human-like source-target alignments for building predictions.