Self-attention Does Not Need $O(n^2)$ Memory

Since its inception by Vaswani et al. (2017), the attention mechanism has revolutionized sequence modeling, especially within Transformer architectures. Early efforts like the Reformer (Kitaev et al., 2020) introduced sparse attention to reduce complexity, but still faced memory constraints at very long sequences. As models scaled up, the quadratic memory growth became a bottleneck, limiting applications in NLP, genomics, and beyond. Current solutions often rely on approximations or sparse methods, sacrificing some accuracy for efficiency. The need for an exact, memory-efficient attention algorithm remains urgent, especially with the advent of ultra-long sequence tasks. This paper addresses this gap by proposing a non-approximate method that drastically reduces memory consumption while maintaining full accuracy.

The core issue is that standard self-attention requires storing an n×n matrix of scores, leading to O(n^2) memory usage. For long sequences, this quickly exceeds hardware limits, making it infeasible. Although the time complexity remains quadratic, the memory bottleneck prevents scaling to longer inputs. Existing approaches like sparse attention or low-rank approximations compromise accuracy or introduce complexity. Therefore, developing a method that retains exactness while reducing memory is crucial for advancing large-scale sequence modeling.

The key innovations include: 1) delaying softmax normalization to the end, enabling incremental computation; 2) maintaining a running maximum score for numerical stability; 3) processing queries and keys in chunks to reduce memory from O(n^2) to O(log n). These techniques are combined into a practical algorithm that computes the same attention output as the standard method, but with vastly reduced memory footprint. The implementation leverages hardware-aware chunking and checkpointing to support backpropagation, making it suitable for training large models on existing accelerators.

• �� The algorithm computes dot products between query and key vectors sequentially, updating two variables: a weighted sum of values and a total weight. • It postpones the softmax normalization by accumulating exponentiated scores and weighted values, tracking the maximum score to prevent overflow. • For self-attention, queries are processed one at a time, with an index tracking the current position, ensuring O(log n) memory. • Hardware-aware chunking divides keys and values into blocks of size √n, processing each block independently and summarizing results. • During backpropagation, checkpointing is used to recompute intermediate states, avoiding full storage. • The implementation in JAX on TPU supports multi-head attention with minimal memory overhead.

The authors evaluate their method on sequence lengths from 28 up to 16384, measuring peak memory and runtime on TPUv3 hardware. Results show a consistent reduction in memory usage, up to 59-fold at length 16384, with runtime differences within 5%. They also train a Transformer for WMT translation, achieving BLEU scores nearly identical to standard attention. Additional experiments on differentiation demonstrate a 32-fold memory reduction during backpropagation, with acceptable slowdown due to checkpointing. These experiments confirm the method's scalability, accuracy, and efficiency across tasks and hardware.

At sequence length 16384, memory overhead drops from hundreds of MBs to about 1MB, a 59× reduction, with negligible impact on translation BLEU scores. The method maintains quadratic time complexity but enables processing much longer sequences than previously feasible. During training, the model's BLEU scores and convergence behavior are nearly identical to standard attention, validating the approach's correctness. The memory savings during backpropagation are even more significant, facilitating large-scale training on resource-limited devices. Overall, the results demonstrate a practical, scalable solution for long-sequence modeling.

This algorithm is ideal for tasks requiring processing of very long sequences, such as document summarization, genomic data analysis, and long-form language modeling. It enables researchers and practitioners to build larger models or process longer inputs without hardware upgrades. In industry, it can reduce costs and hardware requirements, making large-scale NLP applications more accessible. Its exact computation ensures high fidelity, suitable for sensitive applications like legal or medical text analysis, where approximation errors are unacceptable.

Despite memory benefits, the method retains O(n^2) time complexity, which can be computationally intensive for extremely long sequences. Numerical stability depends on max-value tracking, which may be challenged in extreme score distributions. The current implementation is optimized for TPUs and GPUs; broader hardware support and further optimization are needed. Future work should explore reducing time complexity, improving robustness, and extending to multi-modal data, ensuring wider applicability and efficiency.