Large Language Model Routing with Benchmark Datasets

The evolution of NLP has been driven by the development of large-scale pre-trained models such as BERT, GPT, and LLaMA, which have demonstrated remarkable capabilities across diverse tasks. Early benchmarks like UCI and ImageNet focused on single-task performance, but the advent of foundation models shifted evaluation towards multi-task benchmarks such as HELM, MMLU, and Open LLM Leaderboard. These benchmarks aggregate performance across multiple datasets, providing a comprehensive view of model capabilities. Despite these advances, the evaluation primarily relies on average scores, which do not reflect the nuanced needs of real-world applications, where task-specific model selection is critical. Researchers have explored transfer learning, few-shot learning, and model ensemble techniques to improve performance, but the challenge remains in efficiently selecting the best model for a new, unseen task, especially when deploying large models incurs high computational costs. Existing methods like model ensembling, output ranking, and confidence-based selection have limitations in scalability and robustness, particularly under distribution shifts. Consequently, there is a pressing need for methods that can leverage benchmark data to predict model performance on new tasks, enabling smarter, resource-efficient deployment.

The core problem addressed in this work is how to efficiently and accurately select the most suitable large language model for a new task in a multi-model environment. Traditional approaches involve generating outputs from all candidate models and then selecting the best based on evaluation metrics, which is computationally prohibitive at scale. Moreover, models often encounter distributional shifts when applied to new tasks, leading to unpredictable performance. The challenge is to develop a method that can predict model performance without exhaustive inference, using only lightweight features derived from benchmark data. This requires overcoming difficulties such as limited labeled data for new tasks, the heterogeneity of task domains, and the need for robustness against out-of-distribution data. Addressing these issues is crucial for deploying AI systems that are both cost-effective and adaptable to dynamic environments.

The main innovation of this paper is the formulation of multi-model selection as a set of binary classification problems trained on benchmark-derived data. Instead of generating model outputs for each input, the authors train correctness predictors (g_m) using sample features and performance labels, enabling rapid prediction of model correctness probabilities. The introduction of three scoring mechanisms—S1, S2, and S3—allows for flexible, robust model ranking, with S3 explicitly modeling OOD uncertainty via Bayesian inference and task descriptors. This approach bridges the gap between traditional evaluation metrics and real-time model selection, reducing inference costs and improving robustness. The work also establishes a theoretical connection to meta-learning, demonstrating that adaptive routing strategies can outperform static baselines, especially in the presence of distribution shifts. The simplicity of using sentence embeddings and kNN classifiers underscores the practicality and scalability of the approach.

• �� Data collection: Extract per-sample performance metrics (correctness, scores) from benchmark datasets (HELM, MixInstruct) for multiple models.
• �� Feature extraction: Use sentence-transformer embeddings to convert input samples into fixed-length feature vectors.
• �� Correctness predictor training: For each model m, train a binary classifier g_m (kNN with k=5) to predict whether the model will perform well on a sample, using the extracted features.
• �� Score design: Develop three scoring functions—S1 (predictor probability), S2 (thresholded correctness), S3 (Bayesian correction incorporating task descriptors)—to evaluate models during inference.
• �� Task modeling: Represent each task d by a descriptor u(d), measuring its distance to other tasks; fit a non-parametric regression model (Gaussian kernel smoothing) to estimate p(d′, m), the probability of correctness on new tasks.
• �� Model routing: For a new task d′, compute sample features, estimate p(d′, m), and calculate scores S1, S2, S3; select the model with the highest score, balancing performance and cost.

• �� Dataset: 29 diverse datasets from HELM and MixInstruct, covering question answering, text classification, reasoning, and instruction-following scenarios.
• �� Models: 18 open-source models, including Llama 2 variants (3B to 70B parameters), evaluated across multiple tasks.
• �� Evaluation metrics: Accuracy, Pearson, and Spearman correlations for model performance; inference cost measured by parameter count and number of model calls.
• �� Training details: Correctness predictors trained as kNN classifiers on sentence embeddings; hyperparameters set to k=5; training data derived from benchmark samples.
• �� Baselines: Mean model performance (BMA), log-likelihood scoring, and oracle scores based on true correctness probabilities.
• �� Procedure: For each experiment, 28 datasets serve as training benchmarks, with the remaining dataset as the test task; repeated multiple times for statistical robustness.

• �� The S3 scoring mechanism achieved an average accuracy of 69.4%, outperforming the mean model (68.8%) and closely approaching the oracle upper bound (73.5%) across 29 datasets.
• �� Incorporating the true correctness probability (oracle) boosted accuracy to 73.5%, confirming the importance of accurate correctness estimation.
• �� In instruction-following benchmarks, the S1 score achieved 73% per-instance accuracy, surpassing traditional log-likelihood methods, while significantly reducing inference costs from 347B parameters to a few billion calls.
• �� The results demonstrate that leveraging benchmark data to train correctness predictors enables effective, resource-efficient model selection, adaptable to diverse tasks and distribution shifts.

• �� Multi-task AI deployment: Enables dynamic, cost-effective selection of models in real-time systems such as chatbots, virtual assistants, and customer support platforms.
• �� Cloud-based AI services: Facilitates automated model scheduling, reducing computational costs and improving response quality.
• �� Research and development: Provides a scalable framework for evaluating and deploying new models across multiple domains, accelerating innovation.
• �� Future integration with active learning and multi-modal data can further enhance robustness and applicability in complex real-world scenarios.

• �� The correctness predictors achieve moderate accuracy (~59%), which may limit effectiveness in highly diverse or novel tasks, especially under severe distribution shifts.
• �� The reliance on simple embeddings and kNN classifiers may not scale well with increasing feature complexity or high-dimensional data.
• �� Dependence on benchmark data with detailed per-sample performance metrics restricts applicability in scenarios lacking such annotations, particularly for emerging tasks or domains.