OckBench: Measuring the Efficiency of LLM Reasoning
OckBench jointly evaluates accuracy and token efficiency, revealing large disparities in reasoning density among models.
Key Findings
Methodology
OckBench integrates reasoning datasets across math, coding, and science, employing a Differentiation Filter to select tasks with high token variance. It compares models at matched accuracy levels, measuring full output token counts, and introduces OckScore—a metric penalizing verbosity logarithmically. The evaluation involves 37 models, including proprietary and open-source, using greedy decoding for stability, with precise answer extraction and unit tests. Analysis of token distribution, model capacity, and efficiency gaps highlights the importance of reasoning density. Optimization strategies include model interpolation and difficulty-aware reinforcement learning, aimed at enhancing token efficiency without sacrificing accuracy.
Key Results
- Results show open-source models like DeepSeek-V4-Flash consume 26 times more tokens than GPT-5.4 at similar accuracy (82.5% vs. 82.0%), exemplifying efficiency gaps. Increasing model size from 9B to 397B parameters improves accuracy from 21.5% to 67.5%, while reducing average token use by 4.4×, demonstrating capacity scaling benefits.
- Introducing Per-Token Intelligence, the study emphasizes that models should maximize information density, reducing unnecessary reasoning tokens. The evaluation reveals a persistent efficiency gap between open and proprietary models, with proprietary models achieving higher reasoning density.
- Optimization techniques like model interpolation and reinforcement learning effectively improve reasoning density, as evidenced by increased OckScore, indicating that models can be trained or fine-tuned to produce more concise, accurate reasoning paths.
Significance
This work shifts the evaluation paradigm from pure accuracy to efficiency, emphasizing token economy as a core aspect of model intelligence. It provides a standardized, hardware-agnostic benchmark to quantify and compare reasoning efficiency, guiding future model development toward more resource-effective solutions. The findings have broad implications for deploying large models in cost-sensitive and latency-critical applications, fostering sustainable AI practices and enabling scalable deployment in real-world scenarios.
Technical Contribution
The paper introduces OckScore, a unified metric combining accuracy and token efficiency, and a Differentiation Filter to select tasks exposing efficiency gaps. It formalizes Per-Token Intelligence, emphasizing reasoning density. The benchmark spans multiple domains, facilitating cross-model comparison. Additionally, it proposes training-free model interpolation and difficulty-aware reinforcement learning to optimize reasoning paths, pushing the frontier of efficient inference. These contributions collectively provide a comprehensive framework for evaluating and improving reasoning density in large language models.
Novelty
This is the first comprehensive framework to jointly evaluate accuracy and token efficiency across diverse reasoning tasks, explicitly exposing efficiency gaps between open-source and proprietary models. The concept of Per-Token Intelligence emphasizes reasoning density as a key metric, shifting focus from mere output correctness to the quality of reasoning paths. The Differentiation Filter ensures the benchmark challenges models on tasks where efficiency differences are most pronounced, enabling targeted improvements. These innovations collectively advance the state-of-the-art in model evaluation and optimization.
Limitations
- The evaluation relies on specific datasets and tasks, which may not fully represent all real-world scenarios. Broader task coverage is needed.
- Optimization strategies like reinforcement learning require significant computational resources and may not generalize across all model architectures.
- Token efficiency improvements are constrained by hardware limitations, such as context window size and processing speed, which need further hardware-aware optimization.
Future Work
Future research will extend the benchmark to include multi-modal and multi-task scenarios, exploring adaptive inference strategies and hardware-aware optimization. Developing lightweight training techniques and scalable fine-tuning methods will further enhance token efficiency. Additionally, expanding the dataset pool and refining the Per-Token Intelligence concept will help create more robust, generalizable models capable of high-density reasoning across diverse applications.
AI Executive Summary
Large language models (LLMs) like GPT-5 and Gemini 3 have revolutionized automated reasoning and code generation, yet their evaluation remains incomplete. Existing benchmarks predominantly measure accuracy and output quality, neglecting the critical aspect of token efficiency—how concisely models reason and respond. This oversight leads to significant inefficiencies, with models reaching similar accuracy levels but differing in token consumption by up to 26 times. Such disparities inflate deployment costs, latency, and energy consumption, especially in resource-constrained environments.
To address this gap, the authors introduce OckBench, a novel, hardware-agnostic benchmark that jointly assesses accuracy and token efficiency across reasoning and coding tasks. By employing a Differentiation Filter, OckBench selects tasks that reveal substantial token consumption variance, ensuring the evaluation focuses on the efficiency frontier. The core metric, OckScore, logarithmically penalizes unnecessary verbosity while prioritizing correctness. Extensive experiments on 37 models, including proprietary and open-source, demonstrate that open models lag significantly in reasoning density, despite closing the accuracy gap.
A key concept introduced is Per-Token Intelligence, emphasizing the importance of dense, concise reasoning. Results show that larger models tend to produce shorter, more efficient responses, whereas smaller, verbose models incur an 'Overthinking Tax,' increasing deployment costs. The study also explores optimization pathways, such as model interpolation and difficulty-aware reinforcement learning, to enhance reasoning density.
Overall, this work advocates a paradigm shift: evaluation metrics should penalize unnecessary token use, fostering the development of more resource-efficient models. The implications extend to real-world deployment, where reducing token consumption directly translates to lower costs, faster responses, and greener AI systems. This research paves the way for a new standard in model assessment, emphasizing quality, efficiency, and practical deployability in the era of large-scale AI.
Deep Dive
Abstract
Large language models (LLMs) such as GPT-5 and Gemini 3 have pushed the frontier of automated reasoning and code generation. Yet current benchmarks emphasize accuracy and output quality, neglecting a critical dimension: efficiency of token usage. The token efficiency is highly variable in practical. Models solving the same problem with similar accuracy can exhibit up to a \textbf{5.0$\times$} difference in token length, leading to massive gap of model reasoning ability. Such variance exposes significant redundancy, highlighting the critical need for a standardized benchmark to quantify the gap of token efficiency. Thus, we introduce OckBench, the first benchmark that jointly measures accuracy and token efficiency across reasoning and coding tasks. Our evaluation reveals that token efficiency remains largely unoptimized across current models, significantly inflating serving costs and latency. These findings provide a concrete roadmap for the community to optimize the latent reasoning ability, token efficiency. Ultimately, we argue for an evaluation paradigm shift: tokens must not be multiplied beyond necessity. Our benchmarks are available at https://ockbench.github.io/.