Depth-Attention: Cross-Layer Value Mixing for Language Models
Depth-Attention enhances language models by cross-layer value mixing, improving accuracy by up to 2.3 points.
Key Findings
Methodology
Depth-Attention performs cross-layer selection within the attention module, allowing the current layer's query to attend to earlier layers' keys at the same position and mix their values into the current value. This method adds no parameters or persistent inference state.
Key Results
- On Qwen3 architecture, Depth-Attention achieves the lowest perplexity and highest average downstream accuracy at 1.5B and 3B scales, improving over vanilla Transformer by up to 2.3 points.
- Compared to strong baselines like mHC and Attention Residuals, Depth-Attention excels in perplexity and average accuracy with only 0.01% extra computation.
- Depth-Attention performs well across 360M to 3B parameters and extends to looped Transformers.
Significance
This research addresses the limitation of traditional Transformers in selectively reusing earlier layer representations, significantly improving language model performance while reducing inference computation and memory overhead.
Technical Contribution
Depth-Attention reuses standard attention queries, keys, and value-cache slots to achieve cross-layer value mixing without adding parameters or inference state, offering an efficient mechanism for cross-layer information flow.
Novelty
Depth-Attention is the first to perform cross-layer selection within the attention module, avoiding the extra hidden state overhead compared to existing methods, and providing a novel way of cross-layer information flow.
Limitations
- In certain scenarios, Depth-Attention may not fully leverage all earlier layer information, potentially limiting performance to theoretical best.
- The complexity of cross-layer selection might lead to reduced computational efficiency on some hardware architectures.
Future Work
Future research could explore the application of Depth-Attention in different model architectures and further optimize cross-layer selection mechanisms for enhanced performance.
AI Executive Summary
Depth-Attention is a novel method that performs cross-layer selection within the attention module, addressing the limitation of traditional Transformers in selectively reusing earlier layer representations. This method excels in Qwen3 architecture, enhancing language model performance while reducing inference computation and memory overhead.
Depth-Attention achieves cross-layer value mixing by reusing standard attention queries, keys, and value-cache slots, adding no parameters or inference state, and offering an efficient mechanism for cross-layer information flow.
Future research could explore the application of Depth-Attention in different model architectures and further optimize cross-layer selection mechanisms for enhanced performance.
Deep Analysis
Background
Transformer models use self-attention to dynamically select information across the sequence dimension, but information flows less selectively across depth. Standard decoders propagate earlier-layer information only through residuals, collapsing all previous layers into a single hidden state.
Core Problem
Existing cross-layer methods operate on hidden states outside attention, adding computation and memory overhead during inference. Depth-Attention aims to solve this by performing cross-layer selection within the attention module.
Innovation
Depth-Attention performs cross-layer selection within the attention module, allowing the current layer's query to attend to earlier layers' keys at the same position and mix their values into the current value. This method adds no parameters or persistent inference state.
Methodology
- �� Perform cross-layer selection within the attention module
- �� Current layer's query attends to earlier layers' keys
- �� Mix earlier layers' values into the current value
- �� Reuse standard attention queries, keys, and value-cache slots
Experiments
Conduct large-scale pretraining experiments on Qwen3 architecture, comparing Depth-Attention with vanilla Transformer decoders and representative cross-layer baselines.
Results
Depth-Attention achieves the lowest perplexity and highest average downstream accuracy at 1.5B and 3B scales, improving over vanilla Transformer by up to 2.3 points.
Applications
This method can be used to enhance language model performance, reduce inference computation and memory overhead, suitable for large-scale language model training and inference.
Limitations & Outlook
In certain scenarios, Depth-Attention may not fully leverage all earlier layer information, potentially limiting performance to theoretical best.
Plain Language Accessible to non-experts
Imagine a school where teachers teach different grades. Traditional models are like students who only focus on the current grade, unable to revisit previous lessons. Depth-Attention is like a smart student who not only focuses on current lessons but also selectively revisits past knowledge for better understanding and application.
ELI14 Explained like you're 14
Imagine you're playing a game with different levels. Traditional models are like players who only focus on the current level, unable to recall past experiences. Depth-Attention is like a smart player who not only focuses on current challenges but also revisits past experiences for better strategy!
Glossary
Depth-Attention
A method that performs cross-layer selection within the attention module by reusing standard attention queries, keys, and value-cache slots for cross-layer value mixing.
Used to enhance language model performance and reduce inference computation and memory overhead.
Perplexity
A measure of how well a language model predicts the next word; lower values indicate better model performance.
Used to evaluate language modeling quality.
Qwen3 Architecture
An architecture for language models that supports large-scale parameter training and inference.
Used to evaluate Depth-Attention's performance.
Residual Connection
A standard mechanism for passing information across layers in deep networks by adding each block's transformation to the inherited representation.
Used to optimize deep model training.
Self-Attention
A mechanism that allows each token to mix information across the sequence, but offers no comparable mechanism across depth.
Used for dynamic selection of sequence dimension information.
Open Questions Unanswered questions from this research
- 1 How to further optimize cross-layer selection mechanisms for enhanced performance?
- 2 What is the application effect of Depth-Attention in different model architectures?
Applications
Immediate Applications
Language Model Optimization
Enhance model performance with Depth-Attention, reducing inference computation and memory overhead. Suitable for large-scale language model training and inference.
Long-term Vision
Cross-Layer Information Flow
Explore the application of Depth-Attention in different model architectures and further optimize cross-layer selection mechanisms for enhanced performance.
Abstract
Self-attention selects information freely across the sequence, but across depth, Transformers merely add each layer's output to the residual stream, so later layers cannot selectively reuse earlier-layer representations. Recent cross-layer methods improve this flow but operate on hidden states outside attention, adding state beyond the key-value cache at inference--a cost that becomes increasingly salient as modern LLMs compress the cache with grouped-query and multi-head latent attention. We introduce Depth-Attention, which performs this selection inside the attention module itself: before a layer attends over the sequence, its query attends over the keys of earlier layers at the same token position and mixes their values into the value that self-attention then reads. Because Depth-Attention reuses the standard attention queries, keys, and value-cache slots, storing depth-mixed values in place of the original values, it adds no parameters and introduces no persistent inference state beyond the standard key-value cache--the same cache size as a vanilla decoder and less than hidden-state-based cross-layer methods. On Qwen3-style decoders at 1.5B and 3B parameters, Depth-Attention attains the lowest perplexity and the highest average downstream accuracy, improving over the vanilla Transformer by up to 2.3 accuracy points and surpassing strong cross-layer baselines in perplexity and average accuracy, while adding under 0.01% extra arithmetic FLOPs and no additional persistent inference state. The gains hold from 360M to 3B parameters and extend to looped Transformers.