DecoderLens: Layerwise Interpretation of Encoder-Decoder Transformers
DecoderLens introduces layerwise interpretation by cross-attending intermediate encoder states in encoder-decoder Transformers.
Key Findings
Methodology
DecoderLens builds on LogitLens by forcing the decoder to attend to intermediate encoder layer representations. It involves passing the encoder layer outputs through a nonlinear function, then feeding them into the decoder’s cross-attention during inference. This process produces natural language sequences that reflect the information stored at each layer without additional training. The approach is applicable across tasks like QA, logical reasoning, speech recognition, and translation, revealing how information evolves across layers. It leverages the residual stream in Transformers, enabling direct interpretation of intermediate states and their contribution to final outputs.
Key Results
- In Flan-T5 models, DecoderLens shows that intermediate layers, especially in capitalized prompts, outperform top layers with over 85% accuracy in geographic question answering, indicating early layers encode substantial factual knowledge. In logical reasoning tasks, intermediate layers effectively capture variable assignments, reaching 93% accuracy, demonstrating the progressive refinement of representations. For machine translation and speech recognition, intermediate layers produce outputs comparable to final layers, confirming the layered information flow. These results highlight the method’s ability to identify task-relevant information at different depths, providing insights into how models process complex tasks step-by-step.
- The experiments reveal that early layers tend to encode local, task-specific information, while middle layers integrate broader context, and final layers refine details. This layered understanding helps diagnose model behavior, improve interpretability, and guide model design for better transparency and robustness.
Significance
This work advances interpretability by providing a layerwise visualization of internal representations in encoder-decoder models, bridging the gap between black-box analysis and mechanistic understanding. It enables researchers to pinpoint where specific knowledge resides, understand how models progressively build solutions, and identify potential failure points. Such insights are crucial for developing trustworthy AI, especially in safety-critical applications. The method’s generality across tasks and models underscores its potential as a standard tool for deep model analysis, fostering transparency and accountability in AI systems.
Technical Contribution
The paper introduces a novel, training-free technique that exploits residual streams and cross-attention to interpret intermediate encoder states. Unlike prior methods limited to decoder-only models, DecoderLens extends interpretability to encoder-decoder architectures, providing full sequence generation from partial representations. It combines the residual stream’s properties with cross-attention mechanisms, enabling dynamic, layer-specific insights. This approach offers a new paradigm for mechanistic interpretability, with potential for integration into model debugging, pruning, and knowledge extraction workflows.
Novelty
DecoderLens is the first method to visualize and interpret the evolution of encoder representations in encoder-decoder models via cross-attention, extending LogitLens’ residual stream approach. Its key innovation is forcing the decoder to generate outputs based solely on intermediate encoder states, revealing how information is accumulated and refined across layers. This provides a more comprehensive understanding of model internal dynamics than previous techniques, which mainly focus on final outputs or static feature attribution. Its ability to produce plausible sequences from intermediate states marks a significant step forward in interpretability research.
Limitations
- The method relies on the residual stream’s properties, which may vary across architectures or training regimes, limiting its generalizability. In models without residual connections, effectiveness diminishes.
- Interpretability of intermediate outputs can be ambiguous when representations contain mixed or overlapping information, especially in complex tasks.
- Computational cost increases with model size and depth, making real-time analysis challenging for very large models. Further optimization is needed for scalability.
Future Work
Future research could extend DecoderLens to multimodal models integrating vision and language, explore dynamic interpretability during training, and develop automated tools for layer-wise analysis. Combining this approach with causal inference techniques could deepen understanding of decision pathways, ultimately leading to more transparent and controllable AI systems.
AI Executive Summary
Deep learning models, especially Transformer-based architectures, have achieved remarkable success across NLP, speech, and translation tasks. However, their internal mechanisms remain largely opaque, hindering trust and further optimization. Traditional interpretability methods, such as feature attribution or activation analysis, often provide coarse or static insights, leaving the dynamic evolution of internal representations largely unexplored. To address this, the present work introduces DecoderLens, a novel, layerwise interpretability technique inspired by LogitLens, tailored for encoder-decoder models.
DecoderLens leverages the residual stream inherent in Transformer architectures, enabling the decoder to attend to intermediate encoder states during inference. This process produces human-interpretable sequences that reflect the information stored at each layer, without requiring additional training. The method effectively visualizes how factual knowledge, logical variables, or translation information propagates through the network, revealing that early layers often encode local or task-specific features, while middle layers integrate broader context, and later layers refine the output.
Extensive experiments across diverse tasks—question answering, logical reasoning, speech recognition, and machine translation—demonstrate the method’s robustness. For instance, in geographic QA with Flan-T5, intermediate layers surpass top layers in accuracy under certain prompts, indicating early encoding of factual data. In logical tasks, intermediate layers successfully capture variable assignments, aligning with the theoretical understanding of progressive information refinement. In translation and speech tasks, intermediate layers produce outputs comparable to final layers, confirming the layered nature of information processing.
This approach significantly enhances our understanding of model internal dynamics, providing a powerful tool for debugging, interpretability, and safety assessment. It opens avenues for future work in multimodal interpretability, causal analysis, and model transparency, despite current limitations related to architecture dependence and computational costs. Overall, DecoderLens marks a substantial step toward transparent, trustworthy AI systems that can be better understood and controlled.
Deep Analysis
Background
Transformer模型自Vaswani等（2017）提出以来，推动了自然语言处理、语音识别和机器翻译的快速发展。代表性模型包括BERT（Devlin et al., 2019）、GPT（Radford et al., 2018-2023）等预训练架构。早期研究多关注模型性能提升，缺乏对内部表示的深入理解。近年来，解释方法如特征归因（Sundararajan et al., 2017）、激活分析（Jumelet et al., 2021）和机制性解释（Elhage et al., 2021）逐渐兴起。LogitLens（nostalgebraist, 2020）利用残差流将中间层映射到词汇空间，揭示了知识存储的层级结构。随着编码器-解码器架构的广泛应用，理解其逐层信息流成为新挑战。
Core Problem
尽管已有多种解释工具，但对编码器-解码器模型中信息在不同层级的演变过程仍缺乏直观、细粒度的理解。传统方法多关注模型最终输出或整体特征归因，难以揭示中间表示的具体任务作用。模型内部的多层交互机制复杂，缺乏可视化工具，限制了模型调试和优化的效率。特别是在多任务、多模态场景中，理解不同层级的知识流动成为关键瓶颈。这不仅影响模型的透明性，也制约其在实际应用中的可信度。
Innovation
本研究的核心创新在于提出DecoderLens方法，将中间编码层的表示引入解码器交叉注意机制，实现逐层可解释性。具体创新点包括：• 利用残差流，将中间层向量映射到解码器输入空间；• 强制解码器以中间层表示为交叉注意输入，生成对应的输出序列；• 无需训练，通过模型自身的残差流实现动态解释。这一方法突破了LogitLens仅适用于解码器的限制，扩展到编码器-解码器架构，提供了更丰富的机制性理解工具。
Methodology
- �� 选择预训练的编码器-解码器模型（如T5、NLLB、Whisper）；• 在正常解码过程中，将第i层编码器输出通过非线性变换f，映射到解码器交叉注意输入；• 让解码器以此中间表示生成输出，映射层级知识；• 不进行额外训练，直接利用残差流实现逐层解释；• 在问答、逻辑推理、翻译等任务中验证效果，分析不同层的输出内容。
Experiments
在问答、逻辑推理、语音识别和机器翻译任务中，验证DecoderLens的效果。问答任务中，分析Flan-T5模型不同层的地理知识存储；逻辑任务中，观察中间层变量赋值的演变；翻译任务中，评估中间层对翻译质量的贡献；语音识别中，分析中间层的转录准确率。每项任务都通过指标如准确率、BLEU、COMET等进行量化，验证不同层级的知识表达和信息流动。
Results
在多任务中，发现中间层在存储和处理任务关键信息方面表现优异。例如，Flan-T5在地理问答中，中间层正确率超过85%，优于顶层的75%；逻辑任务中，中间层成功捕获变量赋值，准确率达93%；翻译和语音识别中，中间层输出的质量与最终层相似，验证了逐层信息演变的假设。这些结果表明，模型在不同任务中逐层积累和细化信息，为理解其工作机制提供了实证依据。
Applications
该方法适用于模型调试、性能优化和安全性检测。通过逐层观察模型内部表示，有助于发现偏差、漏洞，优化模型结构和训练策略。在工业界，可用于提升模型透明度、实现知识蒸馏和模型压缩，增强模型的可信度和可控性。未来结合因果推断，能更深入理解模型决策路径，推动可信AI的发展。
Limitations & Outlook
该方法依赖残差流特性，可能在非标准架构或训练方式下效果受限。中间层表示仍可能含有模糊信息，导致输出难以完全解释模型机制。对大规模模型的逐层分析计算成本较高，实时应用存在挑战。未来需优化算法，提高效率，扩展到更多模型架构和任务中。
Plain Language Accessible to non-experts
想象一个工厂生产线，每个工序都在逐步完善产品。第一步是原材料准备，中间几道工序是组装、检测，最后一道是包装。每个工序都在做不同的事情，但都为最终产品服务。DecoderLens就像在每个工序中插入一个摄像头，观察每个步骤的产品状态。这样，我们可以知道在生产的哪个阶段，产品变得更完整、更接近最终形态。它帮助我们理解这个复杂的工厂是怎么一步步把原材料变成成品的。
ELI14 Explained like you're 14
你知道我们用手机玩游戏或发信息时，背后其实有个超级复杂的电脑在帮忙吗？这个电脑里有很多“工厂工人”，每个工人负责不同任务，比如拼图、涂色、检查。这个研究就像是给每个工人装个摄像头，让我们可以看到每一步他们在做什么。这样，我们就知道，工厂里的工人在不同阶段是怎么合作，把原材料变成漂亮的成品的。这个方法叫DecoderLens，它让我们可以逐层观察这个“工厂”的工作流程，了解每个环节的秘密。
Glossary
Transformer (变换器)
一种深度学习模型架构，利用自注意力机制处理序列数据，广泛应用于自然语言处理。
论文中提到的基础模型架构。
LogitLens (对数线索)
一种利用残差流将模型中间层映射到词汇空间的解释方法，用于分析模型内部知识存储。
作为DecoderLens的灵感来源。
Cross-Attention (交叉注意)
在Transformer中，解码器通过注意机制结合编码器中间层表示，实现信息融合。
DecoderLens强制解码器以中间编码层为交叉注意输入。
Residual Stream (残差流)
模型中每一层的输入和输出之间的跳跃连接，用于信息的逐层传递和积累。
DecoderLens利用残差流实现逐层解释。
Encoder-Decoder (编码器-解码器)
一种模型架构，编码器提取输入特征，解码器生成输出，广泛用于翻译、问答等任务。
本文分析的模型类型。
Open Questions Unanswered questions from this research
- 1 如何将DecoderLens扩展到多模态模型（如图像、视频）中，仍未充分研究。当前方法主要针对文本模型，跨模态理解仍是未来挑战。
- 2 模型中间层的语义解释仍存在模糊性，如何量化不同层级的知识表达，仍需深入探索。
Abstract
In recent years, many interpretability methods have been proposed to help interpret the internal states of Transformer-models, at different levels of precision and complexity. Here, to analyze encoder-decoder Transformers, we propose a simple, new method: DecoderLens. Inspired by the LogitLens (for decoder-only Transformers), this method involves allowing the decoder to cross-attend representations of intermediate encoder layers instead of using the final encoder output, as is normally done in encoder-decoder models. The method thus maps previously uninterpretable vector representations to human-interpretable sequences of words or symbols. We report results from the DecoderLens applied to models trained on question answering, logical reasoning, speech recognition and machine translation. The DecoderLens reveals several specific subtasks that are solved at low or intermediate layers, shedding new light on the information flow inside the encoder component of this important class of models.