Automatic Model Selection with Large Language Models for Reasoning

Recent advances in deep learning have propelled large language models (LLMs) like GPT-3, GPT-4, and Codex to the forefront of NLP research. These models excel at a variety of tasks, yet their reasoning capabilities, especially for complex problems, remain limited. Chain-of-Thought (CoT) prompting, introduced by Wei et al., enhances interpretability and generalization by decomposing problems into intermediate steps. Conversely, Program-Aided Language Models (PAL), exemplified by Gao et al., leverage structured code, such as Python, to improve numerical accuracy and logical rigor. Despite these innovations, single-model approaches often struggle with diverse problem types, prompting research into ensemble and selection strategies. Existing techniques like self-consistency improve performance but at high computational costs. This paper builds on these foundations, proposing a dynamic selection method that leverages LLMs’ contextual understanding to choose the most suitable reasoning strategy per instance, aiming to optimize accuracy and efficiency.

Current reasoning systems predominantly rely on static, single-strategy models, which are insufficient for the heterogeneity of real-world problems. CoT provides flexibility but can lack numerical precision, while PAL offers accuracy but limited adaptability. The challenge lies in effectively selecting the appropriate reasoning approach for each problem without extensive retraining or manual tuning. Existing ensemble methods like self-consistency improve results but incur significant computational overhead. Therefore, a scalable, adaptive mechanism is needed to dynamically choose the best reasoning strategy, balancing accuracy, interpretability, and computational efficiency. The core problem is designing a model selection framework that can operate reliably across diverse tasks, with minimal additional cost and complexity.

This work introduces a novel, LLM-based dynamic model selection framework that combines the strengths of CoT and PAL. Key innovations include: 1) Prompting LLMs to generate and compare reasoning chains from both models, 2) Employing probabilistic analysis to quantify the likelihood of correct model choice, 3) Theoretically establishing bounds on performance improvements even with imperfect selection, 4) Achieving state-of-the-art accuracy on benchmarks like GSM8K (96.8%) and SVAMP (93.7%), and 5) Demonstrating that integrating model selection with self-consistency reduces costs while boosting accuracy. Unlike prior static ensemble methods, this approach adaptively chooses the optimal reasoning path for each instance, leveraging LLMs’ contextual understanding to make informed decisions, thus significantly advancing the field of AI reasoning.

• �� Input: Reasoning question Q. • Generate: Use LLM to produce CoT reasoning chain C_CoT and answer A_CoT; simultaneously generate PAL reasoning chain C_PAL and answer A_PAL. • Compare: If A_CoT ≠ A_PAL, prompt LLM to evaluate which solution is better, providing a brief explanation S. • Select: Based on S, choose A_final as either A_CoT or A_PAL. • Theoretical analysis: Model the performance difference R(x) between models, and derive bounds on overall error reduction. • Empirical validation: Conduct experiments on eight datasets, compare accuracy with baselines, analyze cost-performance trade-offs. • Use few-shot exemplars to guide LLM in judgment, ensuring robustness across tasks.

The experimental setup involves eight datasets covering arithmetic and symbolic reasoning, including GSM8K, SVAMP, ASDIV, and Date Understanding. Baselines include Codex, ChatGPT, GPT-4, and open-source Llama 2 models. The evaluation metrics are accuracy, cost (tokens and dollars), and success rate of model selection. Hyperparameters like temperature are fixed to ensure comparability. Experiments compare pure models, ensemble methods, and the proposed model selection approach, analyzing performance gains, cost reductions, and robustness. Additional ablation studies evaluate the effect of different sample sizes, prompt designs, and model combinations, validating the theoretical insights.

The proposed method consistently outperforms individual models and static ensembles, achieving 96.8% accuracy on GSM8K with GPT-4, surpassing previous state-of-the-art of 95.5%. It improves accuracy on SVAMP and other datasets by 1-2%, even when the difference between models is large. Integrating with self-consistency, the approach reduces inference costs by up to 80%, while maintaining or improving accuracy. Experiments with open-source Llama 2 models demonstrate broad applicability, with accuracy gains of around 2%. The success rate of model selection exceeds 70% on most datasets, confirming the effectiveness of the LLM-based decision process. Theoretical bounds support the empirical findings, showing that even imperfect selection can lead to substantial performance improvements.

This framework can be directly applied to mathematical problem solving, logical reasoning, and automated programming tasks. It enables AI systems to adaptively choose the most suitable reasoning strategy, improving accuracy and robustness in diverse applications such as educational tools, scientific computing, and intelligent assistants. The approach also facilitates efficient resource utilization by reducing unnecessary computations, making it suitable for deployment in real-world scenarios where computational cost is critical. Long-term, this method can be extended to multimodal reasoning, autonomous decision-making, and complex industrial automation, transforming AI into more flexible and intelligent systems.

The method's success heavily depends on the calibration and reasoning ability of the LLM, which may vary across domains. In cases of poor calibration or unfamiliar tasks, selection accuracy may decline, affecting overall performance. Although costs are reduced compared to naive ensembles, large models still incur significant computational expenses, especially with multiple samples. The current validation is limited to text-based reasoning tasks; extending to multimodal data or real-world environments requires further research. Additionally, the approach assumes the availability of high-quality prompts and exemplars, which may not always be feasible in practice.