Beyond Query Memorization: Large Language Model Routing with Query Decomposition and Historical Matching
DecoR optimizes LLM routing via query decomposition and historical matching, achieving higher accuracy and reduced inference costs.
Key Findings
Methodology
DecoR redefines routing as a capability-based historical matching process. It includes query decomposition, hierarchical log-sifting, and empirical decision stages to enhance generalization.
Key Results
- DecoR achieves 95.59% accuracy in ID tasks with only 1.4x inference cost, outperforming existing methods.
- In OOD tasks, DecoR maintains stable performance with only a 3% accuracy drop, while other methods degrade significantly.
- Ablation studies show query decomposition contributes most to performance, with accuracy dropping by 2.1x when removed.
Significance
This study addresses poor generalization in OOD scenarios, offering a new approach to dynamic model selection while reducing redundant use of high-performance models.
Technical Contribution
Introduces a capability-based routing framework that avoids memorization traps; develops CodaSet benchmark for evaluating routing generalization.
Novelty
First to combine query decomposition and historical matching for LLM routing, significantly improving generalization and reducing inference costs compared to existing methods.
Limitations
- Relies heavily on historical logs; insufficient logs may impact performance.
- May require higher-performance models for extremely complex tasks.
Future Work
Future work could explore more efficient log-sifting algorithms and extend applications to dynamic model pools.
AI Executive Summary
Existing LLM routing methods often fail to generalize well to out-of-distribution (OOD) data and incur high inference costs. To address this, the paper introduces DecoR, a novel framework that optimizes routing decisions through query decomposition and historical matching.
DecoR operates in three stages: query decomposition extracts intrinsic task requirements; hierarchical log-sifting identifies relevant historical records; empirical decision balances performance and cost to select the optimal model. Experiments show DecoR outperforms existing methods across multiple benchmarks, achieving high accuracy and efficiency.
Beyond proposing a new routing method, the study develops CodaSet, a benchmark for evaluating routing generalization. Future work could refine log-sifting algorithms and explore applications in dynamic model pools.
Deep Analysis
Background
With the rise of LLMs, model routing has become critical for optimizing performance and cost. However, existing methods rely on direct query-to-model mappings, often leading to poor generalization.
Core Problem
The core challenge is balancing predictive performance and computational cost, especially in OOD scenarios where existing methods fail to generalize effectively.
Innovation
DecoR redefines routing by combining query decomposition and historical matching. Query decomposition extracts intrinsic requirements, avoiding semantic interference; historical matching selects logs based on capability needs, enhancing decision reliability.
Methodology
- �� Query decomposition: Transforms queries into structured profiles with skills, knowledge, and difficulty.
- �� Hierarchical log-sifting: Uses Jaccard similarity and BGE-M3 embeddings to filter relevant logs.
- �� Empirical decision: Balances normalized performance and cost scores to select the optimal model.
Experiments
Experiments use the CodaSet benchmark, including ID tasks (e.g., MMLU-Pro) and OOD tasks (e.g., Math500). Metrics include accuracy and inference cost, comparing multiple routing methods.
Results
DecoR achieves 95.59% accuracy in ID tasks with 1.4x cost; in OOD tasks, it maintains stable performance, outperforming other methods significantly.
Applications
Applicable for dynamic model selection to optimize inference efficiency, suitable for search engines, QA systems, etc.
Limitations & Outlook
Relies heavily on historical logs, limiting applications in new domains; extremely complex tasks may require higher-performance models.
Plain Language Accessible to non-experts
Imagine you're cooking in a kitchen, and DecoR acts like a smart assistant recommending the best tools based on your recipe. It first analyzes your dish (query decomposition), then finds similar recipes from past experiences (historical matching), and finally selects the most efficient tool balancing cost and effectiveness (empirical decision).
ELI14 Explained like you're 14
Think of DecoR like a teammate recommendation system in a game. It first analyzes your mission needs, like ranged attack or close combat (query decomposition). Then, it checks past missions to see which teammates performed best (historical matching). Finally, it picks the best teammate based on efficiency and win rate (empirical decision). Cool, right?
Glossary
Query Decomposition
Transforms queries into skill, knowledge, and difficulty requirements.
Used to extract intrinsic task needs, avoiding semantic interference.
Historical Matching
Matches capability needs with historical logs to support decisions.
Enhances routing reliability.
CodaSet
Benchmark for evaluating routing generalization.
Includes ID and OOD tasks for experimental validation.
Empirical Decision
Balances performance and cost to select the optimal model.
Used for final routing decisions.
Memorization Trap
Phenomenon where routers overfit training data, failing to generalize.
A key issue in existing methods.
Open Questions Unanswered questions from this research
- 1 How to improve performance without historical logs?
- 2 How to optimize routing for complex tasks?
Applications
Immediate Applications
Search Engine Optimization
Dynamically select models to improve query response efficiency.
QA Systems
Choose optimal models based on question complexity to reduce costs.
Long-term Vision
Dynamic Model Pool Management
Automate pool expansion and optimization for broader scenarios.
Abstract
Optimizing the trade-off among predictive performance and computational cost is a central focus in the deployment of Large Language Models (LLMs). Current routing methods primarily rely on direct mapping from queries to models based on surface-level features, making them susceptible to the memorization trap and leading to poor generalizability on out-of-distribution (OOD) data. In this paper, we propose DecoR, a novel routing framework that recasts the routing task as a matching process of sifting similar queries from historical logs, effectively mitigating the memorization trap. To enhance matching accuracy, we introduce a query capability deconstruction method that decouples linguistic surface forms from task-intrinsic requirements, directing matching toward capability dimensions to ground decisions in essential task attributes. Furthermore, we develop CodaSet, a comprehensive benchmark for assessing routing generalization, where experimental results demonstrate that DecoR maintains superior accuracy while substantially lowering inference costs across both in-distribution and OOD settings. All the codes and data are available at https://github.com/lvbotenbest/DecoR.