What Does BERT Look At? An Analysis of BERT's Attention
This study analyzes BERT's attention heads, revealing their alignment with syntactic relations, and demonstrates dependency parsing with 77% accuracy using attention maps.
Key Findings
Methodology
The paper conducts a comprehensive analysis of BERT's 144 attention heads through visualization, statistical measures, and an attention-based probing classifier. Attention maps are visualized to identify heads focusing on special tokens like [SEP] and positional offsets. Gradient importance scores quantify each head's influence on model outputs, revealing syntactic relevance. A dependency parsing classifier is trained on attention distributions, achieving 77% UAS on Penn Treebank, validating the syntactic information embedded in attention. Clustering analysis uncovers behavior groups among heads, indicating internal diversity and redundancy, providing insights into how BERT encodes syntax.
Key Results
- Analysis shows that many attention heads specialize in particular syntactic roles, such as identifying direct objects, determiners, and prepositional objects, with accuracy exceeding 75%. The dependency parser based on attention maps reaches 77% UAS, outperforming baselines, confirming that attention captures significant syntactic structure. In coreference tasks, a specific head achieves over 65% accuracy, demonstrating the model's capacity to encode long-distance semantic relations. Attention entropy analysis reveals some heads focus narrowly, others broadly, indicating diverse information processing across layers.
- Attention to special tokens like [SEP] is prevalent, but further analysis suggests this may serve as a 'no-op' rather than segment aggregation. Gradient importance scores show that attention to [SEP] becomes negligible in later layers, supporting this hypothesis. The clustering of heads indicates behaviorally similar groups within layers, with some heads redundantly capturing the same relations, hinting at potential model pruning or efficiency improvements.
- The attention-based dependency parser significantly outperforms simple baselines and approaches the performance of supervised models, highlighting the rich syntactic information in attention maps. The analysis also uncovers that heads in the same layer tend to behave similarly, yet exhibit diverse specialization, reflecting a balance between redundancy and diversity in learned representations.
Significance
This research provides the first systematic evidence that BERT's attention mechanisms inherently encode syntactic structures, even without explicit supervision. It advances understanding of how deep transformers internalize language hierarchies, bridging the gap between model interpretability and performance. The findings suggest that attention maps can serve as effective, interpretable features for syntactic tasks, reducing reliance on annotated data. This insight paves the way for developing more transparent NLP models, fostering trust and explainability in AI systems, and guiding future architecture design to better leverage attention for linguistic understanding.
Technical Contribution
The paper introduces a multi-faceted analysis framework combining visualization, statistical measures, gradient importance, and a novel attention-based dependency parser. It systematically characterizes attention head behaviors, identifies their roles in syntactic and semantic tasks, and demonstrates that attention maps alone can achieve high dependency parsing accuracy. The clustering of heads reveals internal structure and redundancy, informing model pruning and interpretability. The proposed probing classifier, trained solely on attention distributions, establishes a new benchmark for syntactic information extraction from pre-trained models, offering a lightweight yet effective alternative to vector-based probing methods.
Novelty
This work is the first to systematically analyze BERT's attention heads in relation to syntactic and coreferential phenomena, establishing a direct link between attention patterns and linguistic structures. Unlike prior studies focusing on output analysis or vector probing, it leverages attention maps as primary features, demonstrating their rich syntactic content. The dependency parsing approach based solely on attention distributions is a novel contribution, showing that self-attention mechanisms inherently encode hierarchical language structures without explicit supervision, marking a significant step forward in interpretability research.
Limitations
- The analysis is limited to English BERT models, and cross-linguistic applicability remains to be validated, especially for languages with different syntactic typologies.
- While many heads encode syntactic relations, some important relations are weakly represented, and complex structures like nested clauses pose challenges.
- The dependency parser based on attention maps, though effective, does not incorporate contextual or multi-layer information, which could further improve accuracy. Additionally, the interpretability of individual heads remains limited, requiring more sophisticated analysis tools.
Future Work
Future research should extend this analysis to multilingual models, exploring how attention heads encode syntax across languages. Combining multi-head and multi-layer information could enhance parsing accuracy and interpretability. Investigating attention's role in other linguistic phenomena, such as semantics and discourse, is also promising. Moreover, integrating these insights into model training could lead to architectures that explicitly leverage syntactic structures, improving both performance and transparency.
AI Executive Summary
The advent of large pre-trained language models like BERT has revolutionized NLP, yet understanding their internal mechanisms remains a challenge. This study offers a comprehensive analysis of BERT's attention heads, revealing that many encode meaningful syntactic relations learned implicitly during self-supervised training. By visualizing attention patterns, measuring their importance, and developing an attention-based dependency parser, the authors demonstrate that attention maps contain rich linguistic information. Notably, the parser achieves 77% UAS on Penn Treebank, validating the hypothesis that attention mechanisms serve as a window into the model's syntactic understanding.
The analysis uncovers diverse behaviors among attention heads, with some focusing narrowly on specific relations like direct objects or determiners, while others attend broadly across sentences. Clustering analysis shows that heads within the same layer often behave similarly, yet exhibit distinct specialization, indicating a balance between redundancy and diversity. The prevalence of attention to special tokens like [SEP] is interpreted as a 'no-op' mechanism, with gradient importance scores supporting this view.
These findings have significant implications for NLP research. They suggest that models like BERT inherently learn hierarchical language structures without explicit supervision, providing a foundation for more interpretable and linguistically informed models. The ability to extract syntactic information directly from attention maps opens new avenues for unsupervised syntax learning, model pruning, and explainability. Despite these advances, the study acknowledges limitations, including language scope and complexity of sentence structures, pointing to future work in multilingual settings and multi-layer integration.
Overall, this research bridges the gap between deep learning and linguistic theory, offering insights into how models internalize language and paving the way for more transparent AI systems capable of nuanced language understanding.
Deep Dive
Glossary
Attention Mechanism (注意力机制)
一种神经网络组件，用于动态调整输入元素的重要性，帮助模型关注相关信息。技术上通过计算查询、键、值向量的加权平均实现。
本文分析了BERT中144个注意力头的行为，揭示其关注不同句法关系的模式。
Dependency Parsing (依存句法分析)
一种句法分析方法，用于识别句子中词与词之间的依存关系，构建句法树。技术上通过预测词之间的依存关系实现。
本文利用注意力映射训练依存关系探测器，验证注意力中蕴含的句法信息。
Self-Attention (自注意力)
Transformer中的核心机制，使每个词在编码时考虑句子中所有其他词的信息。通过计算词对之间的注意力权重实现。
本文分析了BERT中自注意力头的行为，揭示其在句法学习中的作用。
Gradient Importance (梯度重要性)
通过计算模型输出对某参数的梯度，衡量该参数对模型决策的贡献。
本文用梯度重要性分析注意力对模型输出的影响，验证其语法信息的作用。
Clustering Analysis (聚类分析)
一种无监督学习方法，将相似的对象归为一类，以发现数据中的潜在结构。
本文用聚类分析发现行为相似的注意力头，揭示模型内部的行为簇。
Open Questions Unanswered questions from this research
- 1 当前分析主要集中在英文BERT，跨语言的注意力行为差异尚未充分研究，尤其在语法结构差异明显的语言中。
- 2 注意力头的行为虽与语法关系相关，但对复杂句子和多层信息的捕获能力仍有限，未来需结合多层信息提升性能。
- 3 如何将注意力机制的语法表示有效融入下游任务，提升模型的可解释性和性能，仍是亟待解决的问题。
Applications
Immediate Applications
无监督句法分析工具
利用注意力映射实现依存关系识别，降低对标注数据的依赖，适用于低资源语言和快速原型开发。
模型可解释性增强
通过分析注意力头行为，提升模型在语法和语义任务中的透明度，增强用户信任与调试能力。
Long-term Vision
语法敏感的预训练模型
设计专门优化注意力机制的模型架构，使其在多语种、多任务环境中都能更好地捕获语言结构，推动AI理解能力的全面提升。
Abstract
Large pre-trained neural networks such as BERT have had great recent success in NLP, motivating a growing body of research investigating what aspects of language they are able to learn from unlabeled data. Most recent analysis has focused on model outputs (e.g., language model surprisal) or internal vector representations (e.g., probing classifiers). Complementary to these works, we propose methods for analyzing the attention mechanisms of pre-trained models and apply them to BERT. BERT's attention heads exhibit patterns such as attending to delimiter tokens, specific positional offsets, or broadly attending over the whole sentence, with heads in the same layer often exhibiting similar behaviors. We further show that certain attention heads correspond well to linguistic notions of syntax and coreference. For example, we find heads that attend to the direct objects of verbs, determiners of nouns, objects of prepositions, and coreferent mentions with remarkably high accuracy. Lastly, we propose an attention-based probing classifier and use it to further demonstrate that substantial syntactic information is captured in BERT's attention.