Self-Attention with Relative Position Representations
Introduces relative position representations into self-attention, improving machine translation BLEU scores by 1.3 and 0.3, replacing absolute encodings.
Key Findings
Methodology
This work extends the Transformer’s self-attention by incorporating learned edge features aV_{ij} and aK_{ij} to encode relative positions. The approach models sequence elements as nodes in a fully-connected graph, with edges representing relative distances clipped within a maximum distance k. The edge features are shared across attention heads and sequences, optimized via tensor reshaping for efficiency. The modified attention formulas integrate these features into value and compatibility calculations, enabling the model to learn relative positional information directly. Experiments on WMT 2014 English-German and English-French datasets demonstrate BLEU improvements of 1.3 and 0.3 respectively, validating the effectiveness of the method.
Key Results
- On the English-German task, the baseline absolute position model scored 26.5 BLEU, while the relative position model achieved 26.8 BLEU, and the large model reached 29.2 BLEU, showing significant gains. In English-French, improvements of 0.3-0.5 BLEU were observed. Ablation studies confirmed that encoding relative positions solely in the compatibility function sufficed for performance gains, with no additional benefit from combining absolute encodings.
- Varying the clipping distance k from 2 to 64 showed minimal impact on BLEU, indicating the model’s ability to propagate relative position information beyond the clipped range through multi-layer stacking. The approach maintained computational efficiency, with only a modest decrease in training speed (~7%), demonstrating scalability.
- The experiments highlight that relative position representations enhance translation quality, especially for longer sequences, and improve generalization across different sequence lengths and structures.
Significance
This research advances sequence modeling by replacing traditional absolute position encodings with learned relative position features, significantly improving translation performance and model robustness. The approach addresses the limitations of fixed positional assumptions, enabling better generalization to unseen sequence lengths and complex structures. Its efficient implementation paves the way for broader adoption in NLP and graph-based tasks, fostering more flexible and context-aware models that better mimic human understanding of relationships in language and data.
Technical Contribution
The paper introduces a novel relative position encoding mechanism within self-attention, utilizing learnable edge features and a clipping strategy to control complexity. The approach modifies key formulas for value aggregation and compatibility scoring, enabling the model to explicitly encode and leverage relative distances. Efficient tensor reshaping ensures scalability, and parameter sharing across heads and sequences reduces memory footprint. This work bridges sequence and graph modeling, opening avenues for relation-aware attention mechanisms in diverse data structures.
Novelty
This is the first work to embed learned relative position representations directly into the core self-attention formulas of Transformer models, replacing static absolute encodings. The edge feature approach, combined with clipping and sharing strategies, offers a new paradigm for dynamic, relation-based sequence modeling, setting it apart from prior fixed or sinusoidal encodings. Its scalability and simplicity make it a significant innovation in the field.
Limitations
- The method primarily targets linear sequences; extending to complex graph structures with labeled, directed edges requires further adaptation. The fixed clipping distance may limit the capture of very long-range dependencies in some cases.
- While efficient, the added edge features increase model complexity slightly, potentially impacting training speed in extremely large-scale settings. The current design assumes symmetry in relative positions, which may not suit all tasks.
- Further research is needed to incorporate nonlinear compatibility functions and explore richer edge representations to handle diverse data modalities and structural complexities.
Future Work
Future directions include extending the mechanism to arbitrary labeled, directed graphs, enabling relation-aware modeling beyond sequences. Incorporating nonlinear compatibility functions could enhance expressive power. Additionally, applying this framework to multimodal data and multi-task learning could broaden its impact. Exploring adaptive clipping strategies and dynamic edge representations may further improve long-range dependency modeling, making the approach more versatile across NLP and graph-based domains.
AI Executive Summary
Since its introduction in 2017, the Transformer architecture has revolutionized sequence modeling, primarily due to its self-attention mechanism capable of capturing long-range dependencies efficiently. However, the original design relies heavily on absolute position encodings, which limit the model’s ability to generalize across varying sequence lengths and structures. This paper presents a significant innovation by embedding learned relative position representations directly into the self-attention mechanism, effectively modeling the distances between sequence elements as edges in a fully-connected graph.
The core idea involves learning edge features aV_{ij} and aK_{ij} that encode the relative positions between tokens, clipped within a maximum distance k to control complexity. These features are integrated into the attention formulas, replacing or augmenting traditional positional encodings. The implementation leverages tensor reshaping and parameter sharing to ensure computational efficiency, enabling the model to scale to large datasets.
Experimental results on WMT 2014 English-German and English-French translation tasks demonstrate that the relative position approach yields BLEU score improvements of 1.3 and 0.3 respectively over the baseline absolute position models. Ablation studies confirm that encoding relative positions within the compatibility function alone suffices, simplifying the design without sacrificing performance.
This work significantly enhances the model’s ability to generalize to unseen sequence lengths and complex structural relationships, broadening the applicability of Transformer-based models. The proposed mechanism offers a flexible, scalable, and effective way to incorporate relational information, opening new avenues for sequence and graph data modeling. Future work aims to extend this framework to arbitrary graph structures and explore nonlinear relation modeling, promising further breakthroughs in relation-aware neural architectures.
Deep Dive
Abstract
Relying entirely on an attention mechanism, the Transformer introduced by Vaswani et al. (2017) achieves state-of-the-art results for machine translation. In contrast to recurrent and convolutional neural networks, it does not explicitly model relative or absolute position information in its structure. Instead, it requires adding representations of absolute positions to its inputs. In this work we present an alternative approach, extending the self-attention mechanism to efficiently consider representations of the relative positions, or distances between sequence elements. On the WMT 2014 English-to-German and English-to-French translation tasks, this approach yields improvements of 1.3 BLEU and 0.3 BLEU over absolute position representations, respectively. Notably, we observe that combining relative and absolute position representations yields no further improvement in translation quality. We describe an efficient implementation of our method and cast it as an instance of relation-aware self-attention mechanisms that can generalize to arbitrary graph-labeled inputs.