NOVA: Technology-Architecture Co-Design of Near-Memory Processing for Attention-SSM-MoE Hybrid LLM Inference
NOVA combines 4F^2 VCT DRAM and dual-tier NMP architecture, boosting LLM inference by 4.5x.
Key Findings
Methodology
NOVA integrates 4F^2 vertical channel transistor (VCT) DRAM cells with a peri-over-cell (POC) structure, forming a dual-tier NMP architecture. Tier-1 targets low-to-mid Op/B operations, while Tier-2 targets mid-to-high Op/B operations. Parallel execution across tiers supports diverse compute patterns for hybrid LLMs.
Key Results
- NOVA achieves an average 4.5x throughput increase, 69.8% lower end-to-end latency, and 5x better energy efficiency on models like Nemotron3-Nano compared to a GPU baseline.
- With no loss in memory capacity, NOVA incurs only a 3.9% area overhead.
- NOVA effectively supports diverse compute patterns of hybrid LLMs, maximizing inference performance.
Significance
NOVA's co-design of technology and architecture overcomes the physical limits of traditional DRAM and the computational characteristic constraints of existing NMP architectures, significantly enhancing hybrid LLM inference performance and efficiency. This breakthrough offers a new solution for large-scale language model inference.
Technical Contribution
NOVA employs 4F^2 VCT DRAM cells to achieve higher memory density and uses a dual-tier NMP architecture to effectively support diverse compute patterns. This design overcomes the limitations of existing NMP architectures, providing new possibilities for hybrid LLM inference.
Novelty
NOVA is the first to combine 4F^2 VCT DRAM with a dual-tier NMP architecture, addressing memory capacity and computational characteristic issues in hybrid LLM inference. It achieves significant improvements in memory density and compute efficiency compared to traditional methods.
Limitations
- NOVA may be less efficient than dedicated hardware in extreme high Op/B computational scenarios.
- Bandwidth bottlenecks may occur in specific memory-intensive tasks.
Future Work
Future work could explore NOVA's application to larger-scale LLMs and further optimize its performance in extreme computational scenarios.
AI Executive Summary
The rapid evolution of large language models (LLMs) poses dual challenges for existing near-memory processing architectures: technological and architectural. NOVA addresses these by integrating 4F^2 VCT DRAM cells with a dual-tier NMP architecture.
Technologically, NOVA employs 4F^2 vertical channel transistor DRAM cells combined with a peri-over-cell structure to achieve higher memory density. Architecturally, NOVA's dual-tier NMP design supports diverse compute patterns in hybrid LLMs, maximizing inference performance.
Experimental results show that NOVA significantly enhances performance across multiple hybrid LLM models, achieving a 4.5x increase in throughput, 69.8% reduction in latency, and 5x improvement in energy efficiency. This breakthrough offers a new solution for future large-scale language model inference.
Deep Analysis
Background
In recent years, the rapid development of large language models (LLMs) has demanded higher computational and memory capabilities. Traditional DRAM technology faces physical limits at the 10nm node, while existing NMP architectures struggle to support the diverse computational characteristics of hybrid LLMs.
Core Problem
Hybrid LLM inference faces dual challenges of memory capacity and computational characteristics. Traditional DRAM technology struggles to meet the memory demands of large-scale MoE models, and existing NMP architectures cannot efficiently support heterogeneous computational characteristics.
Innovation
NOVA's co-design of technology and architecture employs 4F^2 VCT DRAM cells and a dual-tier NMP architecture to address memory density and computational characteristic issues. Compared to traditional methods, NOVA achieves significant improvements in memory density and compute efficiency.
Methodology
- �� Utilizes 4F^2 VCT DRAM cells to increase memory density.
- �� Employs a peri-over-cell structure to further enhance memory density.
- �� Dual-tier NMP architecture: Tier-1 targets low-to-mid Op/B operations, Tier-2 targets mid-to-high Op/B operations.
- �� Parallel execution across tiers supports diverse compute patterns for hybrid LLMs.
Experiments
Evaluated on multiple hybrid LLM models like Nemotron3-Nano, compared against a GPU baseline. Key metrics include throughput, latency, and energy efficiency.
Results
NOVA achieves an average 4.5x throughput increase, 69.8% lower end-to-end latency, and 5x better energy efficiency across multiple hybrid LLM models.
Applications
NOVA is applicable to large-scale hybrid LLM inference, particularly in memory-intensive and compute-intensive tasks, significantly enhancing performance and efficiency.
Limitations & Outlook
NOVA may be less efficient than dedicated hardware in extreme high Op/B computational scenarios. Bandwidth bottlenecks may occur in specific memory-intensive tasks.
Plain Language Accessible to non-experts
Imagine a factory where traditional production lines require moving raw materials from the warehouse to the production floor, similar to traditional computing architectures needing to move data from memory to processors. NOVA is like installing small production units directly in the warehouse, reducing transport time and increasing efficiency.
ELI14 Explained like you're 14
Hey there! Imagine you're playing a huge puzzle game. Normally, you'd need to move all the puzzle pieces to the table before you can start, but NOVA is like letting you solve the puzzle right in the box, so you don't have to move the pieces around. Isn't that cool?
Glossary
NMP (Near-Memory Processing)
A computing architecture that performs computations close to memory, reducing data transfer.
Used to enhance hybrid LLM inference performance.
DRAM (Dynamic Random-Access Memory)
A common memory technology known for its high density and low cost.
NOVA uses improved DRAM technology to increase memory density.
MoE (Mixture of Experts)
A model architecture using multiple expert networks to improve model flexibility and performance.
NOVA supports inference for large-scale MoE models.
Op/B (Arithmetic Intensity)
The ratio of computational operations to memory bandwidth, reflecting the nature of computational tasks.
NOVA supports tasks with varying Op/B through its dual-tier architecture.
VCT (Vertical Channel Transistor)
A new DRAM cell design that increases memory density through a vertical structure.
NOVA employs VCT technology to achieve higher memory density.
Open Questions Unanswered questions from this research
- 1 How can NOVA be applied to larger-scale LLMs?
- 2 How can NOVA's performance be optimized in extreme high Op/B scenarios?
Applications
Immediate Applications
Large-Scale LLM Inference
NOVA can be used to enhance the performance of large-scale hybrid LLM inference, suitable for scenarios requiring efficient processing of large data volumes.
Long-term Vision
New Computing Architectures
NOVA's success could drive future transformations in computing architectures, challenging the traditional separation of computation and memory design principles.
Abstract
The rapid evolution of hybrid large language models (LLMs), which interleave grouped-query-attention (GQA), state-space model (SSM), and Mixture-of-Experts (MoE) layers, introduces two fundamental challenges for near-memory processing (NMP) architectures. First, the Technology Wall: the conventional 6F^2 DRAM cell is approaching its physical scaling limits at 10nm-class nodes, making it difficult to meet the memory capacity demands of MoE models with hundreds of experts. Second, the Architecture Wall: existing NMP designs target narrow arithmetic intensity (Op/B) ranges and cannot efficiently support the heterogeneous compute characteristics of hybrid LLMs, spanning memory-bound SSM layers, compute-intensive GQA layers, and large Op/B variations across experts. We propose NOVA, a technology-architecture co-designed NMP system that overcomes both walls. On the technology side, NOVA combines a 4F^2 vertical channel transistor (VCT) DRAM cell with a peri-over-cell (POC) structure to achieve approximately 2x memory density at iso-area over conventional 6F^2-based DRAM, enabling continued scaling into sub-10nm nodes. On the architecture side, NOVA repurposes the POC peripheral-die (peri-die) to host processing units, forming a 2-tier NMP architecture: Tier-1 (peri-die NMP) for low-to-mid Op/B operations, and Tier-2 (base-die NMP) for mid-to-high Op/B operations. Parallel execution across tiers supports diverse compute patterns for hybrid LLMs, maximizing inference performance. Evaluated on state-of-the-art hybrid and MoE LLMs including Nemotron3-Nano, Nemotron3-Super, Falcon-H1R, and Qwen3, NOVA achieves on average 4.5x higher throughput, 69.8% lower end-to-end latency, and 5x better energy efficiency over a GPU baseline, with only 3.9% area overhead and no loss in memory capacity.