Who Thinks Best Depends on How Long You Let Them: Budget-Dependent Rankings in LLM Evaluation
This paper introduces budget-dependent evaluation of LLMs, revealing model rank reversals across token budgets (64-4096 tokens), with 14.1% of oracle gap captured by a budget-aware router.
Key Findings
Methodology
The study systematically assesses four models—ranging from 8B to 70B parameters—across three reasoning benchmarks (GSM8K, MATH-500, GPQA) at seven token budgets (64 to 4096). Using deterministic greedy decoding (T=0), the authors evaluate accuracy, analyze per-item correctness trajectories, and classify behaviors into categories: always correct, monotone-increasing, non-monotone, and always wrong. Statistical significance of ranking reversals is tested via McNemar tests. Additionally, they construct oracle ensembles to quantify model complementarity and train budget-aware XGBoost classifiers to predict correctness, enabling dynamic model routing. Control for truncation artifacts ensures robustness of findings.
Key Results
- A significant portion (3%-19%) of items exhibit non-monotone behavior, where accuracy decreases as the token budget increases, with this phenomenon being highly model-specific and rarely overlapping across models (overlap as low as 6-14%).
- Model rankings are not stable; they reverse across budgets on all three benchmarks. For example, on GSM8K, LLaMA-3.3 70B leads at 256 tokens (62.4%), but GPT-OSS 20B surpasses it at 4096 tokens (94.8%), with p<0.001.
- Oracle analysis shows that model complementarity peaks at constrained budgets, with the oracle outperforming the best single model by up to +27.8 percentage points, especially at lower budgets, indicating diverse strengths.
- A budget-aware routing system trained with XGBoost captures 14.1% of the oracle gap cross-domain, outperforming baseline methods. It performs best at moderate budgets (~1024 tokens), but its effectiveness diminishes at extreme budgets due to domain-specific budget-performance relationships.
Significance
The findings fundamentally challenge the conventional evaluation paradigm that treats model performance as invariant across inference conditions. Recognizing the budget dependency reveals that model rankings can fluctuate or even invert, which has profound implications for model selection, deployment, and benchmarking. It underscores the necessity of multi-budget evaluation protocols, especially for resource-constrained environments such as edge devices or real-time systems. Furthermore, understanding model complementarity and developing budget-aware routing strategies can unlock significant performance gains through model ensembles and dynamic scheduling, advancing the deployment of more efficient and adaptable AI systems.
Technical Contribution
This work pioneers a comprehensive analysis of how token generation budgets influence large language model performance. It introduces a behavioral taxonomy at the item level, quantifies model complementarity through oracle ensembles, and demonstrates the effectiveness of a budget-aware routing system trained with gradient boosting. The study also provides rigorous statistical validation of ranking reversals and highlights the importance of considering resource constraints in evaluation frameworks. These contributions collectively push forward the understanding of inference resource management in LLMs and open avenues for adaptive model deployment strategies.
Novelty
The paper is the first to systematically document and analyze the phenomenon of model ranking reversals driven by token budget variations across multiple models and benchmarks. Unlike prior work focusing solely on model architecture or training techniques, this research emphasizes the inference resource dimension, revealing that model performance is highly context-dependent. The introduction of a budget-sensitive evaluation protocol and a practical routing mechanism marks a significant step towards resource-aware AI deployment, setting a new standard for comprehensive model assessment.
Limitations
- Despite controlling for truncation, the high truncation rates at low budgets (e.g., 64 tokens) may bias performance metrics, complicating the interpretation of true reasoning ability versus artifact effects.
- The experiments are limited to three reasoning benchmarks, and the generality of the budget-dependent phenomena across other tasks, such as summarization or dialogue, remains to be validated.
- The proposed routing system, while effective within the evaluated domains, exhibits limited transferability across domains, indicating that budget-performance relationships are domain-specific and require further adaptation.
- Computational costs for extensive evaluation across multiple models and budgets are substantial, posing challenges for large-scale benchmarking and real-time deployment.
Future Work
Future research should explore the universality of budget-dependent behaviors across diverse NLP tasks, including multi-modal and multi-task settings. Incorporating internal model signals, such as logit distributions and hidden states, could enhance routing accuracy and robustness. Developing adaptive, reinforcement learning-based schedulers that dynamically allocate resources based on task complexity and real-time feedback is another promising direction. Additionally, extending these insights to multi-modal models and real-world applications, such as chatbots and autonomous agents, will be crucial for building resource-efficient, high-performing AI systems capable of operating under strict constraints.
AI Executive Summary
The rapid advancement of large language models (LLMs) has revolutionized natural language processing, yet the evaluation methodologies have largely remained static, assuming model performance stability across different inference conditions. This paper by Guedes de Souza and Panisson challenges this assumption by systematically investigating how varying the token generation budget influences model performance and ranking. Their comprehensive study evaluates four prominent models—LLaMA-3 8B, Qwen-3 32B, LLaMA-3.3 70B, and GPT-OSS 20B—across three reasoning benchmarks: GSM8K, MATH-500, and GPQA, at seven different token budgets ranging from 64 to 4096 tokens.
The core discovery is that model rankings are highly sensitive to the token budget, with all benchmarks exhibiting reversals in the top-performing model as the budget increases. For example, on GSM8K, the 70B LLaMA-3.3 model is best at 256 tokens, but GPT-OSS 20B surpasses it at 4096 tokens, with a statistically significant difference (p<0.001). More intriguingly, a notable fraction of items (3%-19%) show non-monotonic behavior, where performance deteriorates as more tokens are allocated, indicating overthinking phenomena that are model-specific and rarely overlapping across models.
To quantify the potential of leveraging model diversity, the authors construct oracle ensembles, revealing that the combined performance can exceed the best single model by up to 27.8 percentage points, especially under constrained budgets. Building on this, they develop a budget-aware routing system based on gradient boosting (XGBoost), trained on text and budget features, which can dynamically select the most suitable model for each sample. This system captures 14.1% of the oracle gap in cross-domain settings, demonstrating practical benefits of resource-aware model selection.
These findings have profound implications for AI evaluation and deployment. They suggest that a single accuracy metric is insufficient, advocating for multi-budget evaluation protocols that better reflect real-world resource constraints. Moreover, the demonstrated model complementarity and the effectiveness of budget-aware routing highlight opportunities for optimizing inference efficiency and accuracy through intelligent model scheduling.
Despite the promising results, limitations remain, including the impact of high truncation rates at low budgets and the domain-specific nature of budget-performance relationships. Future work should focus on extending these insights to broader tasks, integrating internal model signals, and developing adaptive resource management strategies. Overall, this research underscores the importance of considering inference resource constraints in both the evaluation and deployment of large language models, paving the way for more flexible, efficient, and context-aware AI systems.
Deep Analysis
Background
近年来，随着Transformer架构的不断优化和大规模预训练技术的发展，LLMs在自然语言理解和生成任务中取得了突破性进展。代表性模型如GPT系列、LLaMA、Qwen等在多个基准上展现出优异性能。早期研究主要关注模型架构创新、训练数据规模和微调策略，旨在提升模型的整体能力。然而，随着模型规模的扩大和应用场景的多样化，评估体系逐渐暴露出局限性。传统的评估方法通常在固定的推理长度或资源条件下进行，忽略了实际应用中资源限制对模型表现的影响。近年来，test-time compute scaling（如Chain-of-Thought prompting、搜索增强、预算强制机制）成为研究热点，试图在推理过程中动态调整计算资源以优化性能。与此同时，关于“过度思考”现象的研究揭示，更多推理步骤不一定带来更高准确率，甚至可能引入噪声，导致性能下降。这些研究共同推动了对模型推理行为的深入理解，但尚未系统性分析在不同生成预算下模型表现的变化规律，尤其是模型排名的逆转现象。本论文填补了这一空白，系统性地探讨了预算条件对模型性能的影响，为模型评估和调度提供了新的视角。
Core Problem
核心问题在于，现有模型评估体系假设模型性能在不同推理条件下具有一致性，然而实际中，模型在不同生成长度（即token预算）下的表现差异巨大，甚至出现逆转。这一现象严重影响模型的实际应用效果，尤其在资源受限环境（如移动端、边缘设备、实时系统）中，模型的性能表现极易受到预算限制的影响。具体而言，模型在较短预算下可能表现平平，但在较长预算中表现优异，反之亦然。这种不稳定性使得单一的性能指标无法全面反映模型的能力，也阻碍了模型的合理调度和融合。更复杂的问题在于，模型在不同预算下的行为具有高度非线性和样本依赖性，导致模型排名在不同条件下频繁逆转，影响模型选择的可靠性。解决这一问题，要求我们重新设计评估体系，将预算作为重要参数，建立预算条件下的模型性能曲线，从而实现更科学、更全面的模型评价体系。
Innovation
本研究的创新点主要体现在以下几个方面：
1. 发现模型在不同生成预算下的性能逆转现象，挑战了传统的模型稳定性假设，为模型评估引入了预算依赖的新维度。
2. 构建行为分类体系，将每个样本在不同预算下的正确性轨迹划分为始终正确、单调递增、非单调（过度思考）和始终错误，为理解模型推理行为提供了细粒度的分析工具。
3. 利用oracle集成分析，量化模型在不同预算下的互补性，揭示模型在资源受限时的协同潜力。
4. 设计了预算感知的模型路由器（基于XGBoost），通过学习预算特征显著提升跨域模型选择效果，验证了预算作为调度信号的有效性。这一机制为未来动态模型调度提供了技术基础。
Methodology
- �� 选择代表性模型：包括LLaMA-3 8B、Qwen-3 32B、LLaMA-3.3 70B，以及商业模型GPT-OSS 20B，确保参数规模覆盖较大范围。• 任务设计：采用GSM8K、MATH-500和GPQA三项推理基准，涵盖从基础数学到高阶科学问题，确保评估的多样性。• 生成预算设定：在7个不同的最大token数（64至4096）下，逐步评估每个模型在每个任务上的表现，确保数据的全面性。• 评估指标：采用精确匹配（exact match）作为正确性判定，结合截断信息，区分截断和非截断样本，确保评估的可靠性。• 样本分析：构建每个样本在不同预算下的正确性轨迹，分类为始终正确、单调递增、非单调和始终错误。• 统计检验：利用McNemar检验分析模型排名变化的显著性。• 模型互补性分析：通过oracle集成，计算不同模型在不同预算下的性能差异，量化模型互补潜力。• 预算感知路由：训练基于XGBoost的分类器，利用文本特征和预算信息进行模型选择。
Experiments
- �� 数据集：采用GSM8K（1319个小学数学题）、MATH-500（500个竞赛级数学题）和GPQA（198个研究生科学题），覆盖不同难度层级。• 模型：包括开源的LLaMA-3 8B、Qwen-3 32B、LLaMA-3.3 70B，以及商业模型GPT-OSS 20B，确保参数规模多样。• 评估策略：在7个预算水平下，采用贪婪解码（T=0）进行推理，确保输出的确定性。• 样本处理：通过正则表达式提取答案，采用严格的精确匹配评分，统计正确率。• 截断控制：分析截断率对性能的影响，确保非截断样本的可靠性。• 行为分类：根据正确性轨迹，将样本划分为不同行为类别，分析非单调行为的比例和分布。• 统计检验：利用McNemar检验模型排名变化的显著性。• oracle集成：构建每个样本的最优模型组合，评估模型互补性。• 路由器训练：利用文本特征和预算信息训练XGBoost分类器，进行模型选择。
Results
- �� 样本行为分析显示，非单调（过度思考）样本在不同模型中比例高达25.8%，且多为模型特异性，难以通过样本筛除解决。• 模型排名在所有基准上随预算变化而逆转，统计学上显著（p<0.01），例如在GSM8K中，70B模型在256 tokens时表现最佳（62.4%），而在4096 tokens时，20B模型表现更优（94.8%）。• oracle分析表明，模型互补性在有限预算下的潜力最大，最高达+27.8个百分点，说明不同模型在不同样本上的优势互补极大。• 预算感知路由器（基于XGBoost）在跨域任务中捕获了14.1%的oracle性能差距，显著优于传统文本特征选择方法，尤其在中等预算（如1024 tokens）表现突出。• 实验还验证了截断率对性能的影响，控制后非单调行为依然存在，说明过度思考是真实的推理失误。
Applications
- �� 立即应用：在模型部署中引入预算感知的调度机制，根据任务复杂度和资源限制动态选择模型，提高推理效率和准确率。• 资源有限环境：在移动端、边缘设备或实时系统中，合理设定推理预算，平衡模型性能与计算成本，确保系统的实时响应。• 多模型融合：结合oracle分析，设计多模型集成策略，充分利用模型互补性，提升整体推理能力。• 长期愿景：发展智能调度系统，结合强化学习和模型内部信号，实现自适应预算管理，推动大模型在复杂、多变环境中的智能应用。
Limitations & Outlook
- �� 由于模型在极低预算（如64 tokens）下大量截断，性能评估可能偏向于截断行为而非真实推理能力，影响结论的普适性。• 实验仅覆盖三项推理任务，尚未验证在多模态、多任务或更复杂场景中的表现，未来需扩展验证范围。• 预算感知路由器，虽然提升了模型选择效果，但在跨域迁移中表现有限，说明模型-预算关系具有较强的领域依赖性，需进一步研究泛化机制。• 计算成本较高，尤其在大模型和多预算水平下的评估，未来需优化评估流程和算法效率。
Plain Language Accessible to non-experts
Imagine you’re managing a busy kitchen with many different chefs (models), each capable of preparing dishes (answers). The challenge is deciding how long each chef should spend on a dish—sometimes, giving them only a short time (few tokens) results in a quick but possibly imperfect dish. Other times, letting them work longer (more tokens) might improve quality, but it could also cause them to overthink and make mistakes. This is similar to how language models generate text: more tokens don’t always mean better answers.
Researchers found that different chefs perform best at different time limits. Some are quick and accurate with less time, while others need more time to shine. Interestingly, the ranking of which chef is best can flip depending on how much time is allocated. For example, Chef A might be best with a short deadline, but Chef B outperforms when given more time.
To optimize this, the team built a smart kitchen manager (a routing system) that predicts which chef will do the best job based on the dish’s complexity and time limit. This system learns from past data and can choose the right chef for each order, saving time and improving quality. They found that this approach can capture about 14% of the potential improvement—significantly better than just always picking the biggest or fastest chef.
This research shows that in real life, how much time or resources you give to a task can dramatically change who does it best. Instead of sticking to a fixed plan, smartly adjusting resources based on the task can lead to better results, whether in a kitchen, a factory, or an AI system. It’s a reminder that flexibility and resource management are key to success in complex environments.
ELI14 Explained like you're 14
Imagine you’re playing a video game where you have different characters (models) to choose from, and each character can do a task (answer questions). The catch is, you get to decide how much time (or how many moves) each character can use. Sometimes, giving a character only a little time makes them perform well quickly, but other times, they need more time to do their best. Interestingly, which character is best can change depending on how much time you give them.
Scientists studied this idea and found that, just like in the game, models (characters) perform differently depending on how many tokens (moves) they are allowed to generate. They discovered that some models are better with less tokens, while others shine when given more. But here’s the twist: the ranking of models can flip as the token limit increases. For example, a smaller model might do better with fewer tokens, but a bigger one might outperform it when given more tokens.
To make things smarter, the researchers created a system that predicts which model will do best for each question based on how complex it is and how many tokens are allowed. This way, the system can pick the right model for each task, saving time and improving accuracy. They found that this approach can capture about 14% of the maximum possible improvement, which is pretty good.
This study teaches us that in real life, how much effort or time we give to a problem can change who solves it best. Instead of always using the same approach, being flexible and choosing the right method based on the situation can lead to better results—whether in AI, games, or everyday tasks. It’s like having a smart assistant that knows exactly when to use each tool for the best outcome.
Abstract
Standard evaluation of large language models assumes stable model rankings across inference conditions. We challenge this assumption by varying the token generation budget, i.e., the maximum tokens a model may produce, across seven levels (64--4,096), evaluating four models on three reasoning benchmarks (56,476 inferences). We report four findings: (i) 3--19% of items exhibit non-monotone behavior (accuracy decreasing with more budget), even after controlling for truncation, and this phenomenon is model-specific (cross-model overlap: 6--14%). (ii) Model rankings reverse across budgets on all benchmarks ($p {<} 0.01$, McNemar). (iii) Oracle analysis reveals model complementarity up to $+27.8$pp, most pronounced at constrained budgets. (iv) A budget-aware router captures 14.1% of the oracle gap cross-domain; budget features help within-domain ($+1.6$ to $+5.7$pp) but are domain-specific and hurt transfer ($-1.2$pp). These results argue for budget-conditioned evaluation protocols.
References (16)
Chain of Thought Prompting Elicits Reasoning in Large Language Models
Jason Wei, Xuezhi Wang, Dale Schuurmans et al.
GPQA: A Graduate-Level Google-Proof Q&A Benchmark
David Rein, Betty Li Hou, Asa Cooper Stickland et al.
Routing to the Expert: Efficient Reward-guided Ensemble of Large Language Models
Keming Lu, Hongyi Yuan, Runji Lin et al.
The Llama 3 Herd of Models
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey et al.
Large Language Model Routing with Benchmark Datasets
Tal Shnitzer, Anthony Ou, M'irian Silva et al.
Measuring Mathematical Problem Solving With the MATH Dataset
Dan Hendrycks, Collin Burns, Saurav Kadavath et al.
Dynabench: Rethinking Benchmarking in NLP
Douwe Kiela, Max Bartolo, Yixin Nie et al.
Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
C. Snell, Jaehoon Lee, Kelvin Xu et al.
Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs
Xingyu Chen, Jiahao Xu, Tian Liang et al.
Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models
Yang Sui, Yu-Neng Chuang, Guanchu Wang et al.
LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion
Dongfu Jiang, Xiang Ren, Bill Yuchen Lin
Judging LLM-as-a-judge with MT-Bench and Chatbot Arena
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng et al.
NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark
Oscar Sainz, Jon Ander Campos, Iker García-Ferrero et al.
Holistic Evaluation of Language Models
Percy Liang, Rishi Bommasani, Tony Lee et al.
Training Verifiers to Solve Math Word Problems
K. Cobbe, Vineet Kosaraju, Mo Bavarian et al.
s1: Simple test-time scaling
Niklas Muennighoff, Zitong Yang, Weijia Shi et al.