Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation

Since Vaswani et al. (2017) introduced the transformer architecture, handling long sequences has remained a challenge. Traditional position encodings, such as sinusoidal embeddings, enable models to understand token order but falter when sequences extend beyond training lengths, limiting their extrapolation capabilities. This bottleneck hampers applications requiring processing of lengthy documents, like legal analysis or scientific literature. To address this, the authors propose ALiBi—Attention with Linear Biases—a simple yet effective mechanism that biases attention scores based on token distance. ALiBi replaces explicit position embeddings with a fixed, head-specific linear bias, which inherently favors recent tokens. This design allows models trained on short sequences to reliably extrapolate to longer inputs without additional parameters or complex training procedures. Extensive experiments demonstrate that a 1.3 billion parameter transformer trained on 1024 tokens with ALiBi can process sequences up to 2048 tokens with perplexity comparable to models trained directly on longer sequences, but with 11% faster training and less memory consumption. The bias's recency preference also results in superior performance on benchmarks like WikiText-103, outperforming rotary and T5 bias methods. Importantly, ALiBi's simplicity facilitates easy integration into existing frameworks, making it a practical solution for scalable long-text modeling. The findings open new avenues for efficient, large-scale language models capable of understanding and generating extended texts, with potential impacts across NLP, AI research, and industry applications. Future work will focus on adaptive bias parameters, hybrid position encoding strategies, and broader multimodal applications, aiming to push the boundaries of long-sequence processing further.