DUET: Disaggregated Hybrid Mamba-Transformer LLMs with Prefill and Decode-Specific Packages
DUET accelerates Mamba-Transformer models by assigning prefill and decode phases to specialized packages, achieving 4x speed improvement.
Key Findings
Methodology
DUET framework assigns prefill and decode phases to specialized packages, using on-chip systems and high-bandwidth memory for efficient acceleration. The prefill package uses systolic array chiplets, while the decode package uses vector-unit arrays, supporting hybrid Mamba and attention layers.
Key Results
- On Nemotron-H-56B, Zamba2-7B, and Llama3-8B models, DUET achieves a 4x speed improvement in first token generation, 1.4x throughput increase, and 1.5x reduction in time between tokens.
- Compared to the B200 GPU, DUET consistently outperforms across all workloads, significantly improving prefill and decode efficiency.
- Eliminates the poor mapping of SSM recurrences and element-wise operations to matrix multiplication accelerators.
Significance
DUET framework is significant in both academia and industry, addressing performance bottlenecks in large language model inference, especially in hybrid Mamba-Transformer architectures. It provides new insights for future hardware design.
Technical Contribution
DUET achieves specialized acceleration for prefill and decode phases through hardware disaggregation, proposing configurable microarchitectural extensions that efficiently handle long-sequence SSM and standard matrix multiplications.
Novelty
DUET is the first to apply hardware disaggregation to hybrid Mamba-Transformer models, addressing the mismatch between computation and memory bandwidth.
Limitations
- DUET may still face memory bandwidth limitations when handling extremely long sequences.
- Adaptability to new model architectures needs further validation.
Future Work
Future research directions include optimizing the DUET framework to support more types of hybrid models and exploring more efficient hardware designs to further enhance performance.
AI Executive Summary
Large language models face a mismatch between computation and memory bandwidth during inference, and existing solutions fail to fully exploit these characteristics. The DUET framework addresses this bottleneck by assigning prefill and decode phases to specialized hardware packages. The prefill package uses systolic array chiplets, while the decode package uses vector-unit arrays, supporting hybrid Mamba and attention layers. Experimental results show that DUET achieves significant performance improvements across multiple models and workloads, particularly in first token generation speed and throughput. While DUET makes breakthroughs in hardware disaggregation, it still faces memory bandwidth limitations when handling extremely long sequences. Future research will continue to optimize the DUET framework to support more types of hybrid models and improve its adaptability.
Deep Analysis
Background
In recent years, large language models have made significant progress in natural language processing. However, their inference process is still limited by the mismatch between computation and memory bandwidth, especially in hybrid Mamba-Transformer architectures. Existing solutions often rely on homogeneous GPUs, leading to inefficient resource utilization.
Core Problem
The inference process of large language models is divided into a compute-intensive prefill phase and a memory bandwidth-intensive decode phase. Existing hardware architectures cannot simultaneously meet these two demands, leading to performance bottlenecks.
Innovation
DUET addresses the mismatch between computation and memory bandwidth by assigning prefill and decode phases to specialized packages. The prefill package uses systolic array chiplets, while the decode package uses vector-unit arrays, supporting hybrid Mamba and attention layers.
Methodology
- �� The prefill package uses systolic array chiplets to support large matrix multiplications and long-sequence SSMs.
- �� The decode package uses vector-unit arrays to support token-by-token SSM and vector-matrix multiplications.
- �� Both architectures are runtime-configurable to support hybrid models.
Experiments
Experiments were conducted on Nemotron-H-56B, Zamba2-7B, and Llama3-8B models, evaluated using workloads such as ArXiv, BWB, LongWriter, and Chat. DUET outperformed the B200 GPU across all workloads.
Results
Experimental results show that DUET achieves a 4x improvement in first token generation speed, a 1.4x increase in throughput, and a 1.5x reduction in time between tokens.
Applications
DUET is applicable to large language models requiring efficient inference, particularly in applications needing hybrid Mamba and Transformer layers, such as natural language generation and understanding.
Limitations & Outlook
While DUET makes breakthroughs in hardware disaggregation, it still faces memory bandwidth limitations when handling extremely long sequences. Future research will continue to optimize the DUET framework to support more types of hybrid models and improve its adaptability.
Plain Language Accessible to non-experts
Imagine a factory where the prefill phase is like preparing raw materials, requiring fast and efficient processing of large amounts of data. The decode phase is like assembling products, needing precise operations and efficient resource utilization. DUET is like designing two specialized production lines for this factory, one focused on quickly processing raw materials and the other on efficiently assembling products. This way, the overall efficiency of the factory is significantly improved.
ELI14 Explained like you're 14
Imagine playing a fast-paced game where the prefill phase is like collecting power-ups, and the decode phase is like using those power-ups to defeat enemies. DUET is like equipping you with two specialized tools, one to collect power-ups faster and the other to use them more effectively. This way, you can perform better in the game and defeat more enemies!
Glossary
DUET
A framework that assigns prefill and decode phases to specialized hardware packages.
Used to accelerate inference in hybrid Mamba-Transformer models.
Mamba
A hybrid architecture combining state-space models and self-attention mechanisms.
Used in DUET, alternating with Transformer layers.
SSM
A model used for efficient linear-time computation with a fixed-size state representation.
Used in DUET for long-sequence processing.
Systolic Array
A hardware architecture used for efficient matrix multiplications.
Used in the prefill package of DUET.
Vector-Unit Array
A hardware architecture used to accelerate token-wise SSM and vector-matrix multiplications.
Used in the decode package of DUET.
Open Questions Unanswered questions from this research
- 1 How can DUET be further optimized to support longer sequence processing?
- 2 How adaptable is DUET to new model architectures?
Applications
Immediate Applications
Natural Language Generation
DUET can be used to accelerate natural language generation tasks, especially in scenarios requiring hybrid models.
Long-term Vision
Intelligent Assistants
DUET can be used to develop more efficient intelligent assistants, supporting more complex dialogues and tasks.
Abstract
Large language models operate in distinct compute-bound prefill followed by memory bandwidth-bound decode phases. Hybrid Mamba-Transformer models inherit this asymmetry while adding state space model (SSM) recurrences and element-wise operations that map poorly to matmul-centric accelerators. This mismatch causes performance bottlenecks, showing that a homogeneous architecture cannot satisfy all requirements. We introduce DUET, a disaggregated accelerator that assigns prefill and decode phases to specialized packages. The Prefill package utilizes systolic array chiplets with off-package memory for efficient large matrix multiplications and long-sequence SSMs. The Decode package utilizes vector-unit arrays with high-bandwidth in-package memory to accelerate token-by-token SSM and vector-matrix multiplications. Both architectures are runtime-configurable to support hybrid models with mixed Mamba and attention layers. Evaluations on Nemotron-H-56B, Zamba2-7B, and Llama3-8B across four workloads show that DUET achieves 4x faster time to first token, 1.4x higher throughput, and 1.5x lower time between tokens over the B200 GPU.