On Identifiability in Transformers
This paper proves attention weights in Transformers are non-identifiable for sequences longer than head dimension, introduces effective attention, and shows token identity is mainly encoded in embedding angles.
Key Findings
Methodology
Using linear algebra, the authors analyze the rank of the attention matrix T, demonstrating that for sequence length exceeding head dimension, T's null space is non-trivial, leading to multiple attention distributions producing identical outputs. They introduce effective attention by projecting attention weights onto the null space, removing irrelevant components. Gradient attribution quantifies input contributions to embeddings across layers, revealing that token identity is largely encoded in embedding angles and gradually diminishes with depth. Experiments on Wikipedia data validate these theoretical insights, showing the non-uniqueness of attention weights and the persistence of token identity.
Key Results
- Attention weights become non-identifiable when sequence length exceeds head dimension (e.g., >64 tokens), with null space dimension increasing with length. Effective attention diverges from raw attention, highlighting irrelevant peaks that do not influence output. Word embeddings encode token identity mainly via angles, with over 93% recoverability in deep layers through linear mapping. Input contribution analysis shows strong mixing of token and context information, with local context dominating despite long-term dependencies. These findings challenge the interpretability of attention distributions and suggest new diagnostic tools.
Significance
This work fundamentally questions the interpretability of attention in Transformers, revealing its non-uniqueness and proposing effective attention as a more reliable diagnostic. It uncovers that token identity is mainly angle-encoded and gradually fades, providing insights into the internal information flow. These results impact both theoretical understanding and practical explainability, guiding future model design and interpretability methods. The findings address long-standing issues about attention's role in explanation, promoting more robust and transparent AI systems.
Technical Contribution
The paper introduces a formal proof of attention non-identifiability based on the rank-nullity theorem, demonstrating that for sequences longer than head dimension, multiple attention matrices can produce identical outputs. It proposes effective attention via null space projection, offering a practical diagnostic tool. Additionally, it employs gradient attribution to quantify token contributions, revealing the gradual loss of identity information and the dominance of local context. These contributions provide a rigorous theoretical framework and practical methods for analyzing Transformer internals, advancing interpretability research.
Novelty
This is the first comprehensive analysis of the structural non-uniqueness of attention weights in Transformers, combining linear algebra and gradient attribution. It challenges the common assumption that attention weights directly explain model decisions. The introduction of effective attention offers a novel diagnostic approach, and the empirical validation of token identity encoding via embedding angles provides new insights into internal representations, setting this work apart from prior interpretability studies.
Limitations
- The analysis primarily focuses on pre-trained BERT-Base, and its applicability to fine-tuned or other architectures remains to be validated. The non-identifiability becomes more prominent with longer sequences, but many practical tasks involve shorter inputs, limiting immediate impact. Effective attention's non-probabilistic nature may introduce interpretability issues. Gradient attribution methods are sensitive to noise, potentially affecting the accuracy of contribution estimates. Further research is needed to extend these findings to multi-task and multi-modal models, and to develop probabilistic or constrained variants of effective attention.
Future Work
Future research should explore the non-identifiability in other Transformer variants, including GPT and XLNet, especially under fine-tuning. Developing probabilistic versions of effective attention could improve interpretability. Investigating how token identity encoding varies across tasks and modalities will deepen understanding. Extending the analysis to longer sequences and multi-task settings, as well as integrating these tools into model training for better interpretability, are promising directions. Additionally, exploring how to enforce or leverage null space properties during training could lead to more transparent models.
AI Executive Summary
Transformer models have revolutionized NLP, yet their internal mechanisms remain opaque. Central to this opacity is the question of whether attention weights truly reflect model focus. This paper rigorously demonstrates that, for sequences longer than the attention head dimension, attention weights are non-identifiable—meaning multiple distributions can produce the same output. This challenges the common practice of interpreting attention as explanation. To address this, the authors introduce effective attention, which isolates the component of attention weights that genuinely influence the model's decision, providing a more reliable interpretive tool.
Simultaneously, the study investigates the internal representations of input tokens. Contrary to the assumption that token identities are preserved throughout the layers, the authors find that identity information is mainly encoded in the angle of embeddings and diminishes gradually with depth. Using gradient attribution, they quantify how input tokens contribute to hidden states, revealing a strong mixing of token and context information, especially in deeper layers. Despite this, most tokens remain identifiable, indicating that the model retains core input features even after complex transformations.
These findings have profound implications for model interpretability. They show that raw attention distributions are insufficient for explanation, and that more nuanced tools like effective attention are necessary. The insights into token identity encoding also inform how models process and retain information, guiding future architecture improvements. Limitations include the focus on BERT-Base and the challenge of extending these results to longer sequences or other architectures. Overall, this work advances our understanding of Transformer internals, paving the way for more transparent and trustworthy NLP models.
Deep Analysis
Background
近年来，Transformer架构成为NLP的主流模型，代表作包括Vaswani等的原始Transformer、BERT、GPT系列等。这些模型通过自注意力机制实现长距离依赖捕获，极大提升任务性能。然而，模型的黑箱特性引发对其内部机制的疑问，尤其是注意力分布的解释性问题。尽管注意力被广泛用作模型关注区域的代理，但其结构性非唯一性限制了其可信度。此前研究多关注注意力的相关性，但缺乏系统分析其结构性本质。词嵌入的身份保持也被假设为基本特性，但缺乏实证验证。本文在此背景下，系统分析了注意力的非唯一性和词嵌入的逐层变化，为模型理解提供新视角。
Core Problem
核心问题在于，Transformer中的注意力权重是否具有唯一性，是否可以作为可靠的解释依据。现有研究多假设注意力分布代表模型关注区域，但未考虑其非唯一性可能导致的误导。此外，词嵌入的身份保持问题也未被充分验证，影响模型的可解释性。长序列中注意力的非唯一性尤为严重，限制了模型透明度。解决这些问题对于提升模型的可信度和可调试性至关重要，但缺乏系统的理论分析和实证验证。
Innovation
本研究的创新点包括：1）通过线性代数分析，证明在序列长度超过注意力头维度时，注意力权重存在无限多解，揭示其非可识别性。2）引入有效注意力，剔除无关部分，改善模型解释的可靠性。3）利用梯度归因，量化输入词在不同层的贡献，揭示上下文信息的强混合机制。4）验证词嵌入角度编码身份信息，逐层减弱但仍可恢复，为理解深层Transformer提供新证据。这些创新突破了以往对注意力解释的局限，为模型透明性提供理论支撑。
Methodology
- �� 采用线性代数分析，计算注意力矩阵T的秩，证明在序列长度大于头维度（如64）时，T的零空间非平凡，导致注意力非唯一性。• 通过Null空间理论，分析注意力权重的结构性非唯一性。• 提出有效注意力，将注意力分解为影响模型输出的部分和无关部分，利用奇异值分解实现。• 采用梯度归因方法，计算输入词对中间层嵌入的贡献，量化信息混合程度。• 在Wikipedia数据集上验证注意力非唯一性和词嵌入身份保持，使用多层输出。
Experiments
使用Wikipedia样本，提取BERT不同层的输入和隐藏嵌入，训练线性映射和MLP模型进行词识别。分析注意力权重与有效注意力的相关性，验证非唯一性。通过梯度归因，量化输入贡献，观察不同层中词嵌入的身份保持情况。实验还包括序列长度变化对注意力可识别性的影响，验证Null空间随长度增长的趋势。采用多种相似度指标（余弦距离、L2距离）评估词嵌入的可识别性，确保结论的稳健性。
Results
发现当序列长度超过头维度（如64）时，注意力权重不可唯一确定，Null空间维度随长度增加而扩大，导致多重注意力分布对应相同输出。引入有效注意力后，发现部分注意力峰值在模型输出中无实质贡献，质疑其解释性。词嵌入角度编码身份信息，深层逐渐减弱，但通过线性映射仍能恢复93%以上的词身份。输入贡献分析显示，长层中信息被强烈混合，局部上下文占主导，词的唯一贡献仍较高。这些结果揭示了Transformer内部信息流的复杂性，为模型解释提供新工具。
Applications
该研究的发现对模型解释、调试和改进具有重要意义。有效注意力工具可用于识别模型中无关的注意力峰值，提升模型透明度。词嵌入身份保持分析有助于理解模型对输入的依赖程度，优化模型结构。未来可在问答、翻译等任务中应用，增强模型的可信性和可调试性。长远来看，这些技术推动Transformer模型在自动化决策、法律审查等敏感场景中的应用，促进AI的公平与透明。
Limitations & Outlook
分析假设模型为预训练BERT-Base，未考虑微调或其他架构差异，可能影响结论的普适性。注意力非唯一性在极长序列中更为显著，但在实际应用中，序列长度有限，影响解释效果。有效注意力虽改善解释，但其非概率性质可能引入误导。梯度归因方法对噪声敏感，可能影响贡献量化的准确性。未来需验证多任务、多模态场景的适用性，探索结合概率约束的改进方案。
Abstract
In this paper we delve deep in the Transformer architecture by investigating two of its core components: self-attention and contextual embeddings. In particular, we study the identifiability of attention weights and token embeddings, and the aggregation of context into hidden tokens. We show that, for sequences longer than the attention head dimension, attention weights are not identifiable. We propose effective attention as a complementary tool for improving explanatory interpretations based on attention. Furthermore, we show that input tokens retain to a large degree their identity across the model. We also find evidence suggesting that identity information is mainly encoded in the angle of the embeddings and gradually decreases with depth. Finally, we demonstrate strong mixing of input information in the generation of contextual embeddings by means of a novel quantification method based on gradient attribution. Overall, we show that self-attention distributions are not directly interpretable and present tools to better understand and further investigate Transformer models.