SLMs as Multi-Agent Routers: A Progressive SFT and Reinforcement Learning Approach
Trained small language models using progressive SFT and RL to enhance multi-agent retrieval relevance, achieving NDCG@10 of 0.918.
Key Findings
Methodology
The paper employs a combination of progressive supervised fine-tuning (SFT) and reinforcement learning (RL) to train small language models for agent selection and parameter generation. Initially, SFT learns query features, followed by RL optimizing selection based on retrieval quality. A hierarchical reward function integrates query-agent topic alignment and retrieval relevance to enhance model selection capabilities.
Key Results
- On a subset of agent-query mismatches, the trained model achieved an NDCG@10 of 0.918, significantly outperforming intent-based LLM baselines (Amazon Nova Lite and Claude Haiku 4.5) with scores of 0.539 and 0.490, respectively.
- Overall NDCG@10 was 0.771, with a mean selection latency of 120.1ms, an 82.4% reduction over Nova Lite.
- Experiments demonstrate that the RL-trained router identifies agent-query mismatches invisible to intent-only routing.
Significance
This research significantly improves the accuracy and efficiency of multi-agent retrieval systems by combining SFT and RL. The results show that agent selection grounded in retrieval quality signals can surpass traditional intent routing methods, addressing long-standing issues in agent selection with substantial academic and industrial implications.
Technical Contribution
Technical contributions include: 1) a novel method combining SFT and RL for agent selection; 2) optimization of selection decisions using a hierarchical reward function based on retrieval quality; 3) achieving efficient agent selection on small language models, significantly reducing selection latency and computational costs.
Novelty
This paper is the first to introduce retrieval quality signals into agent selection, optimizing decision-making through RL rather than relying solely on query intent. This approach is innovative in the field of agent selection, surpassing existing intent routing methods.
Limitations
- The model may perform poorly on extremely short queries due to insufficient context information.
- Requires a large amount of labeled data for initial SFT training, which is costly to obtain.
Future Work
Future work may explore more complex reward function designs to further improve selection accuracy. Additionally, research on applying this method to larger-scale agent pools is an important direction.
AI Executive Summary
The paper presents a novel method combining progressive supervised fine-tuning (SFT) and reinforcement learning (RL) for agent selection in multi-agent retrieval systems. Traditional intent routing methods rely solely on query topics and fail to detect agent-query mismatches. By introducing retrieval result quality signals, this method significantly improves selection accuracy.
In experiments, the trained model achieved an NDCG@10 of 0.918 on a subset of agent-query mismatches, significantly outperforming intent-based LLM baselines. Overall NDCG@10 was 0.771, with a mean selection latency of 120.1ms, an 82.4% reduction over Nova Lite.
This research has significant academic implications and provides an efficient solution for industrial applications. Future work will continue to optimize reward function design and explore applications in larger-scale agent pools.
Deep Analysis
Background
As AI systems increasingly rely on specialized retrieval agents to answer complex queries, selecting the most suitable agent becomes a critical determinant of retrieval quality. Traditional intent routing methods fail to detect agent-query mismatches, leading to decreased retrieval quality.
Core Problem
The core problem is how to select the most suitable agent in a multi-agent environment to ensure high-quality retrieval results. Existing methods rely solely on query intent and cannot utilize retrieval result quality signals.
Innovation
The innovations include: 1) combining SFT and RL for agent selection; 2) optimizing selection decisions using a hierarchical reward function; 3) achieving efficient agent selection on small language models.
Methodology
- �� Use SFT to learn query features, providing a foundation for selection
- �� Optimize selection decisions with RL using retrieval quality signals
- �� Employ a hierarchical reward function combining query-agent topic alignment and retrieval relevance
Experiments
The experimental design includes comparisons with multiple baseline models, evaluating metrics like NDCG@10 and selection latency. Experimental data comes from real query logs covering multiple domains.
Results
Results show the trained model performs exceptionally well on agent-query mismatches, achieving an NDCG@10 of 0.918, with overall NDCG@10 of 0.771 and significantly reduced selection latency.
Applications
This method can be used in multi-domain retrieval systems requiring efficient agent selection, especially in real-time applications requiring quick responses.
Limitations & Outlook
The model may perform poorly on extremely short queries and requires a large amount of labeled data for initial training. Future work will optimize reward function design and explore larger-scale applications.
Plain Language Accessible to non-experts
Imagine you're in a library with many different shelves, each with books on different topics. You need a book on biomedical science, but some shelves, though topically related, don't have the right content for you. Our model acts like a smart librarian, choosing shelves based not just on topic but also on the quality of the content. This way, it helps you find the most suitable book more effectively.
ELI14 Explained like you're 14
Imagine you're playing a game where you need to choose different characters to complete tasks. Each character has its own strengths, but sometimes the one you pick isn't right for the task. Our model is like a super-smart game assistant that picks characters based not just on the task type but also on how well they actually perform. This way, you can complete tasks faster and better!
Glossary
Small Language Model
A language model with fewer parameters, suitable for low-latency and low-cost applications.
Used for agent selection to provide faster response times.
Reinforcement Learning
A machine learning method that optimizes model decisions through reward signals.
Used to optimize agent selection decisions, improving retrieval result quality.
Supervised Fine-Tuning
Further training a model on labeled data to improve its performance on specific tasks.
Provides the initial foundation for agent selection.
NDCG@10
A metric for evaluating retrieval result quality, considering both relevance and ranking.
Used to assess model performance in agent selection.
Hierarchical Reward Function
A reward function combining multiple signals to optimize model decisions.
Used to combine query intent and retrieval result quality for agent selection.
Open Questions Unanswered questions from this research
- 1 How to apply this method to larger-scale agent pools remains to be further studied.
- 2 The complexity of reward function design may affect model training efficiency.
Applications
Immediate Applications
Real-time Retrieval Systems
Can be used in multi-domain retrieval systems requiring quick responses, improving retrieval relevance and efficiency.
Long-term Vision
Intelligent Information Retrieval
By continuously optimizing agent selection, it promotes the development of intelligent information retrieval technology, enhancing user experience.
Abstract
Specialised retrieval agents typically surface higher quality results than general-purpose search, but selecting the optimal agent for a given query remains an open problem. Current approaches route queries based on inferred topic or intent, however intent-based selection is fundamentally limited: it does not incorporate signal from retrieved content, and cannot detect when a topically aligned agent produces low-relevance results. We address this by training a small language model via supervised fine-tuning followed by reinforcement learning to jointly perform agent selection and structured parameter generation for downstream tool calls, using a hierarchical reward function grounded in retrieval relevance along with query-agent topic alignment. This enables the model to learn task-dependent agent suitability from retrieval performance: which agents reliably yield high-relevance results for which query distributions, and when to redirect queries away from specialised agents despite surface-level topical overlap. On a targeted subset of such agent-query mismatches, the trained model achieves an NDCG@10 of 0.918 compared to 0.539 and 0.490 for two LLM baselines (Amazon Nova Lite and Claude Haiku 4.5) that route on intent alone. Overall, it achieves a mean NDCG@10 of 0.771 (+0.177 over Nova Lite, +0.219 over Haiku) with a mean selection latency of 120.1ms, an 82.4% reduction over Nova Lite.