SOMA: Efficient Multi-turn LLM Serving via Small Language Model
SOMA enables efficient multi-turn LLM serving via small models, reducing costs and latency while maintaining high response quality.
Key Findings
Methodology
SOMA leverages early dialogue turns to extract semantic divergence between large and small models using soft prompts. It then applies LoRA-based fine-tuning to align the small model with the local context, enabling efficient inference without prompts. A gating mechanism ensures seamless switching and rollback on drift.
Key Results
- SOMA achieved 93.1% response similarity on datasets like ShareGPT and ReMeDi, outperforming RouteLLM by ~1%.
- On the MATH dataset, SOMA improved exact-match accuracy to 41.62%, a 7.74% increase over baselines.
- SOMA reduced large model invocation costs by ~40% in multi-turn dialogues while maintaining high contextual coherence.
Significance
SOMA addresses the trade-off between context retention and efficiency in multi-turn dialogues, significantly reducing computational and API costs. It provides a foundation for scalable and cost-effective multi-model dialogue systems in real-world applications.
Technical Contribution
SOMA introduces a novel approach to local manifold approximation using soft prompts and LoRA fine-tuning, reducing reliance on large models. It also incorporates semantic gating and rollback mechanisms to ensure response quality and context coherence.
Novelty
SOMA is the first framework to leverage early dialogue turns for local semantic manifold approximation, enabling small models to efficiently emulate large models in multi-turn settings.
Limitations
- Frequent topic shifts may require repeated rollbacks to the large model, increasing overhead.
- The initialization phase incurs computational costs, making it less suitable for very short dialogues.
- In highly complex dialogue tasks, small models may still underperform compared to large models.
Future Work
Future work could explore more efficient soft-prompt generation methods to reduce initialization costs and enhance small model performance in complex dialogue scenarios.
AI Executive Summary
Multi-turn dialogue is a critical application of large language models (LLMs), but existing methods struggle to balance context retention and computational efficiency. Current approaches often require reprocessing the entire dialogue history at each turn, leading to high latency and costs.
SOMA (Soft-prompts for lOcal Manifold Approximation) introduces a novel framework that uses early dialogue turns to estimate a local response manifold. By mining semantic divergence between large and small models with soft prompts, and fine-tuning the small model using LoRA, SOMA enables efficient multi-turn serving. A semantic gate ensures the small model takes over only when aligned, and a rollback mechanism handles topic drift.
Experiments demonstrate that SOMA achieves 93.1% response similarity across datasets like ShareGPT and ReMeDi, while reducing costs by ~40%. On the MATH dataset, it improves task accuracy by 7.74%. This work provides a scalable solution for cost-effective, high-quality multi-turn dialogue systems, with potential applications in customer service, education, and healthcare.
Deep Analysis
Background
Multi-turn dialogue systems are essential for applications like chat assistants and customer support. However, existing methods often require reprocessing the entire dialogue history at each turn, leading to high computational costs. Studies show that early turns are information-dense, while later turns are shorter but context-dependent, highlighting inefficiencies in current approaches.
Core Problem
The core challenge is balancing efficiency and quality in multi-turn dialogue. Using large models for every turn is costly, while small models often fail to maintain context coherence, leading to degraded response quality.
Innovation
SOMA introduces: 1) Soft prompt mining to identify semantic divergence between large and small models, 2) LoRA-based fine-tuning to align small models with local context, and 3) semantic gating and rollback mechanisms to ensure quality and efficiency.
Methodology
- �� Use the large model for early turns to establish context.
- �� Mine semantic divergence between large and small models using soft prompts.
- �� Fine-tune the small model with LoRA to align it with the local context.
- �� Implement a semantic gate to switch to the small model and a rollback mechanism for topic drift.
Experiments
Experiments were conducted on six datasets, including ShareGPT and ReMeDi, using LLaMA and Qwen model families. Baselines included History-Prefix, History-FT, and RouteLLM, among others.
Results
SOMA achieved 93.1% response similarity on ShareGPT and ReMeDi, outperforming baselines. On MATH, it improved exact-match accuracy by 7.74%. Additionally, SOMA reduced large model costs by ~40%.
Applications
SOMA is ideal for scenarios requiring efficient multi-turn dialogue, such as customer support, educational Q&A systems, and medical consultations, offering cost savings without compromising quality.
Limitations & Outlook
SOMA may require frequent rollbacks in scenarios with frequent topic shifts, increasing overhead. Additionally, its initialization costs may not suit very short dialogues.
Plain Language Accessible to non-experts
Imagine a restaurant kitchen. The large model is like a master chef who can handle complex orders but is expensive. The small model is like a junior chef, cheaper but less skilled. SOMA's approach is to let the master chef handle the complex early orders, then train the junior chef to take over simpler tasks. If the junior chef messes up, the master chef steps back in to ensure quality.
ELI14 Explained like you're 14
Think of playing a team video game. The large model is like a pro gamer who can handle tough challenges but uses lots of energy. The small model is like a newbie who needs guidance. SOMA is like a smart team captain: the pro starts the game, teaches the newbie, and lets them play the easy parts. If the newbie struggles, the pro jumps back in. Cool, right?
Glossary
SOMA
A framework using soft prompts and LoRA fine-tuning to adapt small models for efficient multi-turn dialogue.
Used to approximate large model behavior locally.
LoRA
A lightweight fine-tuning method using low-rank matrix updates.
Applied to align small models with dialogue context.
Soft Prompts
Learnable embeddings that guide a model's behavior in specific contexts.
Used to mine semantic divergence between models.
Semantic Gating
A mechanism to switch between models based on context alignment.
Ensures quality when switching to the small model.
Local Manifold
A region in the model's representation space capturing dialogue context.
Defines the local behavior of the dialogue state.
Open Questions Unanswered questions from this research
- 1 How can initialization costs be further reduced?
- 2 How to improve small model performance in complex dialogues?
- 3 How to optimize semantic gating to minimize rollbacks?
Applications
Immediate Applications
Customer Support
Use SOMA to reduce operational costs while maintaining high-quality customer interactions.
Educational Q&A
Leverage SOMA for efficient, accurate multi-turn dialogues in learning platforms.
Long-term Vision
Accessible AI Services
Lower costs of large models to enable widespread use in healthcare, education, and beyond.
Abstract
Large Language Models (LLMs) are increasingly deployed in multi-turn dialogue settings where preserving conversational context across turns is essential. A standard serving practice concatenates the full dialogue history at every turn, which reliably maintains coherence but incurs substantial cost in latency, memory, and API expenditure, especially when queries are routed to large proprietary models. Existing approaches often struggle to balance the trade-off between response quality and efficiency. We propose a framework that exploits the early turns of a session to estimate a local response manifold and then adapt a smaller surrogate model to this local region for the remainder of the conversation. Concretely, we learn soft prompts that maximize semantic divergence between the large and surrogate small language models' responses to surface least-aligned local directions, stabilize training with anti-degeneration control, and distill the mined cases into localized LoRA fine-tuning so the surrogate runs without prompts at inference. A simple gate enables a one-time switch with rollback on drift. We further provide a theoretical analysis for key components in SOMA. Extensive experiments show the effectiveness of SOMA. The source code is provided at: https://github.com/LabRAI/SOMA.