Why bother with geometry? On the relevance of linear decompositions of Transformer embeddings
This paper investigates linear decompositions of Transformer embeddings, showing high correlation with model performance at corpus level but high variability across runs and tasks.
Key Findings
Methodology
Using two linear decomposition approaches—sub-layer-wise and token-wise—the study analyzes decoder embeddings from machine translation models trained on datasets like Tatoeba Challenge. Geometric indicators such as cosine similarity and norm ratio are computed across training stages, multilingual settings, and random initializations. The analysis compares these metrics with performance scores (BLEU, COMET) and examines their stability across different decoding strategies (forced inference vs. beam search). Statistical correlation measures (Spearman’s ρ, Pearson’s r) quantify the relationship between geometry and performance, revealing insights into the stability and model-specific nature of the embeddings.
Key Results
- Correlation analysis shows that geometric indicators (cosine similarity, norm ratio) are highly correlated with corpus-level BLEU and COMET scores (ρ>0.9), but their sentence-level correlation drops significantly (ρ<0.6), indicating they reflect overall model behavior rather than individual sentence quality.
- Variability across different random initializations exceeds differences caused by training on different datasets, suggesting geometry encodes model-specific features rather than task-specific ones.
- Comparison of decoding strategies reveals near-identical geometric structures (correlation >0.98), questioning the interpretability of geometry differences as explanations of decoding behavior.
Significance
This work challenges the assumption that embedding geometry reliably explains Transformer performance. The high variability across training runs and the weak sentence-level correlations imply that geometric metrics are more indicative of model-specific traits than universal explanations. These findings urge caution in using geometric analyses as definitive interpretability tools and highlight the importance of considering training randomness and model idiosyncrasies. The results have implications for future research aiming to develop robust, generalizable interpretability frameworks for deep models, especially in NLP tasks like machine translation.
Technical Contribution
The paper introduces two mathematically grounded linear decomposition methods leveraging residual connections and attention mechanisms, enabling precise geometric analysis of embeddings. It systematically evaluates these methods across multiple training conditions and decoding strategies, correlating geometric metrics with performance scores. The study also demonstrates that despite high correlations at the corpus level, the geometric structures are highly variable across training runs, emphasizing the model-specific nature of these features. This dual approach advances the quantitative understanding of Transformer internal representations and their relation to model quality.
Novelty
This is the first comprehensive comparison of two linear decomposition techniques across diverse training conditions and multilingual settings, revealing the high variability and limited sentence-level predictability of geometric indicators. The work uniquely combines residual-based algebraic decompositions with performance correlation analysis, providing new insights into the stability and interpretability of Transformer embeddings, thus challenging prior assumptions about their universality.
Limitations
- The focus on machine translation limits the generalizability to other NLP tasks; different tasks may exhibit different geometric behaviors.
- The linear decomposition approach struggles with non-linear activation functions, potentially missing deeper non-linear features.
- High variability across training runs suggests geometric metrics are unstable, requiring further refinement or complementary methods for reliable interpretability.
Future Work
Future research will explore extending these geometric analyses to other NLP tasks such as question answering and summarization. Incorporating non-linear methods and deeper theoretical frameworks could improve stability. Additionally, studying the evolution of geometry during training and across different model architectures will deepen understanding of internal representations, aiming toward more robust interpretability tools.
AI Executive Summary
Transformer models have revolutionized natural language processing, yet their internal mechanisms remain elusive. Researchers have turned to geometric analysis—decomposing embeddings into interpretable factors—to shed light on how these models encode information. However, most studies focus on single models or tasks, leaving questions about the stability and universality of these geometric features unanswered. This paper critically examines two linear decomposition methods applied to machine translation decoders, analyzing their geometric properties across training stages, multilingual settings, and decoding strategies.
The findings reveal that while geometric indicators such as cosine similarity and norm ratios correlate strongly with overall model performance at the corpus level—often exceeding 0.9—they exhibit significant variability across different training runs and are less predictive at the sentence level. Notably, models trained with similar data and conditions can develop markedly different embedding geometries, emphasizing the model-specific nature of these features. Furthermore, comparisons between forced inference and beam search decoding show near-identical geometric structures, casting doubt on the interpretability of geometric differences as explanations of decoding behavior.
These results challenge the assumption that embedding geometry provides a universal explanation for model performance. Instead, they suggest that geometry reflects intrinsic model characteristics more than task-specific computations. The high variability and limited sentence-level correlation imply that geometric metrics should be used cautiously and in conjunction with other interpretability tools. Overall, this work advances our understanding of the internal representations of Transformer models, highlighting the need for more robust, stable, and generalizable interpretability frameworks in NLP.
Deep Analysis
Background
Transformer架构在自然语言处理领域取得了巨大成功，成为主流模型。早期研究如Vaswani等（2017）提出了注意力机制，开启了深度学习模型的几何分析路径。随后，学者们尝试用线性代数工具解析嵌入空间的结构特征，诸如Ethayarajh（2019）和Rogers等（2020）提出的几何指标，试图关联模型内部表示与输出性能。近年来，线性分解方法如Mickus等（2022）和Oh与Schuler（2023）提出的技术，为理解Transformer的内部机制提供了数学工具。这些研究主要关注模型的可解释性、性能关联和内部特征的稳定性，但缺乏对不同训练条件和随机初始化影响的系统验证。
Core Problem
尽管线性分解在模型解释中显示出一定潜力，但其稳定性和泛化性仍存疑。具体而言，几何指标是否能稳定反映模型性能？不同训练随机性是否导致显著的几何差异？解码策略是否影响几何结构？这些问题关系到几何分析的科学性和实用性。现有研究多忽略模型训练的随机性和个体差异，导致结论的普适性不足。解决这些问题对于提升模型可解释性、优化训练流程具有重要意义。
Innovation
本研究创新在于：1）系统比较两种线性分解方法在多训练状态、多语言环境下的几何表现；2）引入多种几何指标（范数比、余弦相似度）结合性能指标（BLEU、COMET）进行相关性分析；3）验证不同解码策略对几何结构的影响，发现其高度一致，质疑其作为模型行为解释的有效性。这些创新为理解Transformer内部机制提供了更全面的数学框架，也为模型调优提供了新的指标。
Methodology
- �� 采用两种线性分解方法（子层级分解和标记级分解），分析Transformer解码器的嵌入表示。• 在不同训练阶段（每1000步）采集模型输出，测量几何指标（余弦相似度、范数比）。• 比较不同随机初始化和多语言训练条件下的几何差异。• 评估解码策略（强制解码与束搜索）对几何结构的影响。• 计算指标与模型性能（BLEU、COMET）相关性，验证几何指标的代表性。• 使用统计方法（斯皮尔曼相关系数）分析指标稳定性和一致性。
Experiments
- �� 数据集：Tatoeba Challenge，涵盖俄英、多语到英等任务，训练样本最多5M句。• 模型：基于marian-MT库，训练72小时，保存多个检查点。• 评估：使用BLEU、COMET、chrF++指标，比较不同训练状态和随机初始化的模型。• 方法：在不同解码策略下采集嵌入，计算几何指标，分析其与性能的相关性。• 还进行不同任务和训练条件的对比，验证几何指标的稳定性。
Results
- �� 相关性分析显示，几何指标（余弦、范数比）在语料层面与BLEU、COMET高度相关（ρ>0.9），但在句子层面相关性明显降低（ρ<0.6），表明指标更反映模型整体特性。• 不同随机初始化模型的几何差异远大于不同任务，显示模型特异性强。• 解码策略对几何结构影响极小，两种策略的指标高度一致（相关系数>0.98），质疑其作为行为解释的有效性。这些结果强调几何指标的局限性及其在模型性能预测中的作用。
Applications
- �� 立即应用：可用于模型训练监控、性能预测和模型调优，帮助开发者理解模型内部状态。• 长期愿景：推动基于几何的模型解释体系，结合其他分析工具，提升深度学习模型的透明度和可信度，促进其在关键行业的应用。
Limitations & Outlook
- �� 研究主要集中在机器翻译任务，未覆盖其他NLP任务，限制了通用性。• 线性分解对非线性激活的处理有限，可能忽略深层非线性特征。• 几何指标变异性较大，稳定性不足，未来需结合多种指标和方法增强解释的鲁棒性。
Plain Language Accessible to non-experts
想象你在一个工厂里，工厂里有很多不同的机器，每台机器负责不同的任务。科学家们试图用一种特殊的“地图”来描述这些机器的工作方式，这个地图显示了每台机器的工作位置和关系。通过观察这个地图，他们希望理解工厂的整体运作。可是，研究发现，每次工厂重新启动，机器的位置会有很大变化，地图也会不同。虽然地图和工厂的性能有关，但每次变化都很大，不能完全说明工厂的效率。这个研究告诉我们，用“地图”理解复杂机器（模型）是有帮助的，但不能把它作为唯一的解释工具，因为每次启动和训练都可能让“地图”变得不同。
ELI14 Explained like you're 14
想象你在玩一个超级复杂的拼图游戏，每次你拼完后，拼图的样子都不一样。科学家们试图用一种特殊的“线条图”来描述拼图的每一块是怎么拼在一起的。他们发现，这些线条图和拼图的整体表现（比如得分）关系很大，但每次拼图的线条都可能变得不一样。有时候，换个拼图的顺序，线条也会变，但拼图的最终得分差不多。这就像是说，拼图的“线条”反映了拼图的整体水平，但不能完全说明每一块拼得好不好。科学家们意识到，这些线条图虽然有用，但不能单靠它们来判断拼图的好坏，因为每次拼图都可能不同。
Abstract
A recent body of work has demonstrated that Transformer embeddings can be linearly decomposed into well-defined sums of factors, that can in turn be related to specific network inputs or components. There is however still a dearth of work studying whether these mathematical reformulations are empirically meaningful. In the present work, we study representations from machine-translation decoders using two of such embedding decomposition methods. Our results indicate that, while decomposition-derived indicators effectively correlate with model performance, variation across different runs suggests a more nuanced take on this question. The high variability of our measurements indicate that geometry reflects model-specific characteristics more than it does sentence-specific computations, and that similar training conditions do not guarantee similar vector spaces.