RouteProfile: Graph-Based Profiling for Cold-Start LLM Routing
RouteProfile uses graph-based profiling to enable cold-start LLM routing, achieving ~6% improvement in training-free scenarios.
Key Findings
Methodology
RouteProfile constructs heterogeneous graphs from public signals like model family, descriptions, benchmarks, and domains. It evaluates profile designs across four dimensions: organizational form, representation type, aggregation depth, and learning configuration.
Key Results
- Result 1: Structured profiles (e.g., TextGNN-4hop) achieve 0.580 in cold-start routing, ~9% better than flat baselines.
- Result 2: EmbGNN-3hop achieves NCIR of 0.411 with GraphRouter in new-LLM integration.
- Result 3: Model family metadata improves performance significantly, e.g., EmbGNN-3hop from 0.500 to 0.551.
Significance
This work addresses the challenge of integrating new LLMs without interaction data, reducing costs and delays in model adaptation. It enables scalable LLM ecosystems.
Technical Contribution
Introduces a graph-based framework for cold-start LLM routing, systematically explores graph structures, and highlights the importance of profile-router co-design.
Novelty
First to leverage public signals (e.g., technical reports) for graph-based profiling, avoiding costly interaction data and pioneering a new direction for cold-start routing.
Limitations
- Limitation 1: Relies on public signals, which may be sparse or incomplete.
- Limitation 2: Deep aggregation can cause over-smoothing, reducing node discriminability.
- Limitation 3: Does not fully address dynamic updates.
Future Work
Future work could explore dynamic graph updates and more efficient embedding generation to enhance adaptability and performance.
AI Executive Summary
As the LLM ecosystem grows, models vary widely in capabilities, costs, and latency. Efficiently selecting the right model is critical, but traditional routing methods rely on interaction data, making new-model integration costly and slow. RouteProfile addresses this by constructing heterogeneous graphs from public signals like model family, descriptions, and benchmarks, enabling effective cold-start routing.
The framework evaluates profile designs across four dimensions: organizational form, representation type, aggregation depth, and learning configuration. Experiments show that structured profiles outperform flat baselines, with TextGNN-4hop achieving 0.580 in cold-start routing. In new-model integration, EmbGNN-3hop achieves an NCIR of 0.411 with GraphRouter, demonstrating the potential of embedding-based profiles.
RouteProfile significantly reduces the cost and time of integrating new models, providing a scalable solution for the evolving LLM ecosystem. Future directions include dynamic graph updates and more efficient embedding methods to further enhance performance and adaptability.
Deep Analysis
Background
The rapid expansion of LLMs has led to diverse capabilities and costs. Traditional routing methods rely on query-response-reward data, making new-model integration slow and expensive.
Core Problem
The core challenge is selecting suitable models in cold-start scenarios. New models lack interaction data, requiring costly inference and retraining in traditional methods.
Innovation
RouteProfile introduces a graph-based approach, integrating public signals into structured profiles. Key innovations include: 1) leveraging model family metadata as a reliable signal, 2) a four-dimensional design framework, and 3) validating the importance of profile-router co-design.
Methodology
- �� Construct heterogeneous graphs with nodes for models, families, domains, and benchmarks.
- �� Use TextGNN and EmbGNN for text or embedding aggregation.
- �� Analyze aggregation depth via multi-hop message passing.
- �� Evaluate in cold-start and new-model integration scenarios.
Experiments
Experiments use 15 datasets and 25 models across knowledge, reasoning, math, and coding. Evaluations include cold-start routing (SimRouter) and new-model integration (GraphRouter). Metrics include average performance and NCIR.
Results
TextGNN-4hop achieves 0.580 in cold-start routing, while EmbGNN-3hop achieves 0.411 NCIR in new-model integration. Model family metadata significantly boosts performance.
Applications
Applicable to multi-model service platforms and dynamic task allocation systems requiring rapid new-model adaptation.
Limitations & Outlook
Relies on public signal quality, faces over-smoothing in deep aggregation, and lacks dynamic update mechanisms.
Plain Language Accessible to non-experts
Imagine a library where books represent models, and their categories, summaries, and ratings are public signals. RouteProfile acts like a smart librarian, analyzing these signals to recommend the best book without needing detailed feedback.
ELI14 Explained like you're 14
Imagine you're in a game with many characters, each with unique skills. RouteProfile is like a smart team captain who picks the best character for a mission by reading their backstories and stats, without testing them all first!
Glossary
Cold-Start
Selecting models or recommendations without historical interaction data.
Used to describe the routing problem for new LLMs.
Heterogeneous Graph
A graph with multiple types of nodes and edges.
Used to integrate model family, domain, and benchmark information.
Model Family
Metadata describing a model's architecture lineage and developer.
Used as a public signal for graph construction.
TextGNN
A text-based graph neural network for message passing and node updates.
Used for text aggregation in cold-start routing.
NCIR
New-LLM Correct Integration Rate, measuring correct routing to new models.
Evaluates new-model integration performance.
Open Questions Unanswered questions from this research
- 1 How to maintain high performance in sparse or dynamically updated signal scenarios?
- 2 Can embedding generation be further optimized for routing efficiency?
Applications
Immediate Applications
Multi-Model Service Platforms
Quickly adapt to new models, reducing deployment costs and improving user experience.
Dynamic Task Allocation
Efficiently select optimal models for real-time tasks, enhancing system performance.
Long-term Vision
Adaptive Model Ecosystems
Build intelligent systems that dynamically adapt to new models and tasks, advancing AI ecosystems.
Abstract
LLM routing is increasingly important for selecting suitable models under diverse user needs and deployment constraints, but its practical effectiveness depends on continual adaptation to emerging queries and newly released models. New-LLM integration is particularly challenging, as newly released models lack the query-response-reward interactions required for router training and cannot be profiled as directly as new queries via semantic embeddings. Existing profiles are limited: LLM-generated descriptions are often coarse, while interaction-based embeddings are costly to construct. To address this problem, we propose RouteProfile, a graph-based profiling framework that constructs LLM profiles from public signals in technical reports or model cards, including model family, model description, reported benchmark scores, and benchmark domains. RouteProfile organizes these heterogeneous signals into a graph and studies profile construction along four dimensions: organizational form, representation type, aggregation depth, and learning configuration. We evaluate RouteProfile in training-free cold-start routing and new-LLM integration settings. Experiments show that: (1) structured profiles outperform flat baselines in training-free cold-start routing; (2) model family metadata is more reliable than benchmark domain information; and (3) effective new-LLM integration requires profile-router co-design. Overall, our findings highlight the importance of profile design for enabling routing systems to adapt to the evolving model ecosystem.