Pin Once, Swap Light: Subspace-Aligned Centroid-Residual Training for Efficient Ultra-LoRA Serving
SALT uses subspace-aligned training to recover high-rank accuracy with ultra-low-rank residuals, achieving 18.5% accuracy improvement.
Key Findings
Methodology
SALT is a hierarchical fine-tuning framework operating in three phases: First, a provider trains high-capacity domain centroids on public data using an alignment regularizer to unify task subspaces. Next, users fine-tune ultra-low-rank task residual adapters on these frozen centroids. Finally, during inference, the provider pins the centroid in GPU VRAM and dynamically swaps in each user's task residual on demand.
Key Results
- SALT recovers high-rank accuracy using r≤2 residuals across LLMs, achieving up to 18.5% absolute accuracy gains and reducing per-adapter memory by up to 16x.
- Integrated into vLLM, SALT improves serving throughput by up to 51% under PCIe bandwidth pressure and 28% under GPU VRAM constraints.
- On mathematical reasoning and coding tasks, SALT significantly enhances task performance while maintaining low memory usage.
Significance
SALT addresses the dilemma between efficiency and task performance in multi-tenant low-rank adapter systems by using subspace alignment and residual training. It not only boosts task performance but also significantly reduces memory usage, enabling more concurrent tenants under limited hardware resources.
Technical Contribution
SALT fundamentally differs from existing methods by decomposing adapter weight updates into high-capacity domain-shared centroids and ultra-low-rank residuals. It offers new theoretical guarantees and engineering possibilities, particularly in optimizing memory usage and improving service efficiency in multi-tenant environments.
Novelty
SALT is the first to achieve high-rank performance recovery under ultra-low-rank conditions through subspace-aligned training. Compared to existing methods, it provides more efficient memory management and task performance in multi-tenant environments.
Limitations
- SALT may require fallback to standard LoRA training when handling entirely unknown task data, as subspace alignment might not adapt to all tasks.
- In extreme cases, recalculating the shared basis may be necessary, increasing computational costs.
Future Work
Future research can explore SALT's application in more domains and further optimize subspace alignment mechanisms to support more complex tasks.
AI Executive Summary
Modern multi-tenant low-rank adapter serving systems face a dilemma between serving efficiency and task performance. Higher-rank adapters generally achieve better task performance, but their GPU VRAM footprint and Host-to-Device PCIe swapping overhead severely constrain scalability. Ultra-low-rank adapters minimize memory footprint and PCIe transfer overhead but suffer from performance degradation.
To address this issue, researchers propose Subspace-Aligned LoRA Training (SALT), a serving efficiency-aware hierarchical fine-tuning framework. First, providers jointly train high-capacity domain centroids on public data using a novel alignment regularizer to unify task subspaces. Next, users fine-tune ultra-low-rank task residual adapters on these frozen centroids. Finally, during inference, providers pin the centroid in GPU VRAM and dynamically swap in each user's task residual on demand.
Experimental results show that SALT recovers high-rank accuracy using r≤2 residuals across LLMs, achieving up to 18.5% absolute accuracy gains and reducing per-adapter memory by up to 16x. Integrated into vLLM, SALT improves serving throughput by up to 51% under PCIe bandwidth pressure and 28% under GPU VRAM constraints. This method not only boosts task performance but also significantly reduces memory usage, enabling more concurrent tenants under limited hardware resources.
Deep Dive
Abstract
Modern multi-tenant Low-Rank Adapters (LoRAs) serving systems concurrently host tens to hundreds of LoRA adapters. Though powerful, this introduces a critical system dilemma between serving efficiency and task performance: higher-rank adapters generally achieve better downstream task performance, but their GPU VRAM footprint and Host-to-Device PCIe swapping overhead severely constrain scalability. Conversely, ultra-low-rank adapters ($r \le 2$) minimize both VRAM footprint and PCIe transfer overhead, but suffer from downstream task performance degradation. To solve this problem, we propose Subspace-Aligned LoRA Training (SALT), a serving efficiency-aware hierarchical fine-tuning framework. Our solution operates in three phases. First, a provider jointly trains high-capacity domain centroids on public data within the domain using a novel alignment regularizer that coheres in-domain task subspaces into a unified basis. Next, users fine-tune ultra-low-rank task residual adapters on private data atop those frozen centroids. Finally, during inference, the provider pins the centroid in GPU VRAM and dynamically swaps in each user's task residual on demand. Across LLMs of varying scales, SALT recovers high-rank accuracy using $r \le 2$ residuals, achieving up to 18.5% absolute accuracy gains over state-of-the-art compression baselines and reducing per-adapter memory by up to 16x. When integrated into vLLM, SALT improves serving throughput by up to 51% under PCIe bandwidth pressure and 28% under GPU VRAM constraints for Llama-3.2-3B.