Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading

The evolution of AI, particularly large language models like GPT and BERT, has led to impressive short-term task performance. Early benchmarks such as Terminal-Bench and SWE-Bench primarily assessed final outcomes, neglecting the importance of intermediate steps in complex workflows. In practical scenarios—such as scientific data analysis, software debugging, or multimodal dataset auditing—success depends on sustained, multi-stage reasoning and iterative correction. Existing evaluation methods lack the granularity to measure ongoing progress, limiting progress tracking and model development. Recent efforts have begun exploring multi-step tasks, but these are often domain-specific or limited in scope. This work builds on prior benchmarks by introducing a unified, multi-domain, long-horizon evaluation framework with dense rewards, aiming to better reflect real-world AI capabilities and challenges.

Current benchmarks inadequately assess models' ability to handle long, complex workflows requiring hundreds of steps, continuous planning, and verification. This gap hampers progress toward deploying AI in real-world applications like scientific research, industrial automation, and software engineering, where sustained effort and error correction are critical. The core challenge lies in designing an evaluation system that captures incremental progress, provides meaningful feedback, and encourages models to maintain long-term coherence. Without such metrics, models may appear successful despite only partial or superficial understanding, impeding targeted improvements. Addressing this problem is essential for advancing AI systems capable of managing real-world, long-duration tasks reliably.

The key innovations include: 1) a dense, subtask-based scoring system that assigns partial credit for intermediate progress, 2) a containerized environment ensuring reproducibility and task diversity, 3) multi-domain task design covering scientific, engineering, and multimodal workflows, 4) multi-round evaluation capturing sustained performance over hundreds of steps. These elements collectively enable a comprehensive, fine-grained assessment of long-horizon capabilities, surpassing traditional binary success metrics. The framework emphasizes debugging, verification, and iterative improvement, aligning evaluation with real-world workflow demands. This approach facilitates detailed diagnostics of model strengths and weaknesses, guiding future research.

• �� Construct diverse long-horizon tasks based on real-world workflows, ensuring complexity and relevance. • Use containerized environments with reference solutions and simulation engines for consistent execution. • Decompose each task into meaningful subtasks with specific checks, enabling partial scoring. • Implement dense reward signals based on subtask completion levels, encouraging continuous progress. • Conduct multiple runs per model, collecting success rates, token counts, and execution times. • Analyze near-misses and failure modes to identify bottlenecks in planning, debugging, and verification. • Use cost analysis to evaluate efficiency, balancing success and resource expenditure. This systematic approach provides a detailed view of models' long-term reasoning and persistence capabilities.

The evaluation involved 17 models, including GPT-5 series, Grok 4.5, and DeepSeek V4 Pro, tested across tasks like experiment reproduction, scientific computing, and multimodal analysis. Each model performed multiple runs per task, with metrics including success rate at different reward thresholds, average normalized reward, token usage, and wall-clock time. The experimental setup emphasized dense subtask scoring, near-miss analysis, and cost efficiency. Hyperparameters such as maximum episode count (~239) and time limits (~90 minutes) were standardized. Ablation studies examined the impact of reward density and subtask granularity, confirming their importance for nuanced performance measurement. Results highlighted persistent challenges in long-horizon reasoning, debugging, and verification.

The top-performing model Grok 4.5 achieved a success rate of 28.3% at R≥0.95, with an average success rate of 6.4% across all models. Most models demonstrated partial progress, with many near-misses reaching R between 0.75 and 0.95, but full task completion remained rare. Cost analysis showed Grok 4.5's per-task expense around $11, with other models ranging from $3.6 to $26. The dense reward system effectively differentiated models' ability to sustain progress, exposing common failure modes such as premature stopping and verification lapses. These findings underscore the significant gap between current model capabilities and the demands of real-world long workflows, emphasizing the need for further advancements in planning, debugging, and self-assessment.

This benchmark can be applied in scientific research, industrial automation, software debugging, and multimodal data analysis, where long-term, multi-step reasoning is essential. It provides a standardized framework for evaluating and improving AI systems' ability to handle extended workflows, facilitating development of more robust, persistent models. The detailed feedback from dense rewards guides targeted enhancements, accelerating progress toward real-world deployment. Additionally, the dataset and evaluation tools support community-driven benchmarking and model comparison, fostering innovation in long-horizon AI capabilities.

The current tasks rely heavily on simulated environments, which may not fully capture real-world complexities like environmental noise, unpredictable failures, or resource constraints. The high computational cost limits scalability and practical deployment. Models still struggle with maintaining coherence over extended periods, especially in debugging and verification phases. Future work must address environment realism, reduce resource demands, and develop models with stronger long-term reasoning and self-assessment abilities. Additionally, expanding task diversity and real-world applicability remains an ongoing challenge.