Beyond Final Scores: A Systematic Evaluation of Agents for Long-Horizon AI Research and Development
Proposed a framework to evaluate long-horizon AI research, analyzing 7 models across 36 tasks.
Key Findings
Methodology
The paper presents a systematic evaluation framework using rule-based metrics to analyze solution framing, execution, and feedback control in long-horizon tasks. It highlights the limitations of current agents in autonomous research by comparing experience reuse across tasks.
Key Results
- Result 1: Opus-4.7 achieved the best avg@3 and best@3 scores of 0.739 and 0.790 across 36 tasks.
- Result 2: GPT-5.5 excelled in execution with a score of 0.958 but lagged in feedback control at 0.858.
- Result 3: Experience reuse significantly impacted inter-task performance, with DeepSeek-V4-Pro's avg@3 increasing by 0.093 and Gemini-3.1-Pro's decreasing by 0.017.
Significance
This study reveals the limitations of current AI agents in long-horizon research, emphasizing the impact of experience reuse and harness design on performance. It provides concrete directions for future model training and system design.
Technical Contribution
The technical contribution lies in providing a fine-grained evaluation method that identifies specific bottlenecks in different research stages, surpassing traditional final score evaluations.
Novelty
This is the first systematic evaluation of long-horizon AI research agents, highlighting the deficiencies in current models' autonomous research capabilities.
Limitations
- Limitation 1: Significant performance variability across runs makes consistent reproduction challenging.
- Limitation 2: Genuine methodological innovation remains rare, relying mainly on existing techniques.
Future Work
Future research can focus on improving model training strategies, enhancing inference-time strategies, optimizing experience management, and harness design to boost autonomous research capabilities.
AI Executive Summary
Autonomous agents in long-horizon AI research are becoming increasingly important, but existing evaluation methods often focus solely on final scores, overlooking process details and the effects of experience reuse. This paper proposes a new evaluation framework that analyzes solution framing, execution, and feedback control, revealing the performance and limitations of current agents in long-horizon tasks.
The study shows that current AI agents resemble engineering optimizers more than fully autonomous researchers. While they can formulate and implement practical solutions, their performance varies significantly across runs, and genuine methodological innovation remains scarce. The analysis also finds that experience reuse can aid or mislead subsequent decisions, and harness design affects performance stability.
These findings provide concrete directions for improving model training, inference strategies, experience management, and harness design, advancing long-horizon AI research development.
Deep Analysis
Background
As AI technology evolves, long-horizon autonomous research is becoming feasible. Existing studies mostly focus on short-term tasks, lacking systematic evaluation of AI agents' performance in long-horizon tasks. This paper fills this gap by analyzing 7 frontier models across 36 long-horizon tasks, revealing current AI agents' capabilities and shortcomings.
Core Problem
The core problem in long-horizon AI research is effectively evaluating agent performance. Traditional final score evaluations fail to reveal specific performance during the research process and the effects of experience reuse, making it difficult to identify areas for improvement.
Innovation
The innovation lies in proposing a new evaluation framework that provides a fine-grained analysis of agent performance in long-horizon tasks. By comparing experience reuse across tasks, it highlights the limitations of current agents in autonomous research.
Methodology
- �� Solution Framing: Analyzes the directions pursued by agents in tasks.
- �� Execution: Evaluates agents' ability to translate plans into executable results.
- �� Feedback Control: Examines how agents use feedback to adjust subsequent decisions.
- �� Experience Reuse: Analyzes the effect of experience reuse by comparing performance across tasks.
Experiments
The experimental design includes 36 long-horizon tasks covering four categories: Model Development, System Optimization, Puzzle & Challenge, and CUDA. Each task provides an initial artifact and an expert reference solution, with agents required to optimize the artifact within a set time.
Results
Results show that Opus-4.7 achieved the best avg@3 and best@3 scores of 0.739 and 0.790. GPT-5.5 excelled in execution but lagged in feedback control. Experience reuse significantly impacted inter-task performance, with DeepSeek-V4-Pro's avg@3 increasing by 0.093 and Gemini-3.1-Pro's decreasing by 0.017.
Applications
Applications include improving AI model training strategies, enhancing inference-time strategies, optimizing experience management, and harness design to boost autonomous research capabilities.
Limitations & Outlook
Current models exhibit significant performance variability across runs, making consistent reproduction challenging. Genuine methodological innovation remains rare, relying mainly on existing techniques. Future research should focus on improving model training strategies and harness design.
Plain Language Accessible to non-experts
Imagine a kitchen with several chefs who need to create the best dishes within a limited time. Each chef has their own approach; some are good at quickly finding the best ingredients, while others adjust their cooking process to achieve the best taste. The AI agents in this paper are like these chefs, needing to constantly try and adjust in long-horizon tasks to find the best solutions. By analyzing these chefs' performances at different stages, we can better understand their strengths and weaknesses, providing directions for future improvements.
ELI14 Explained like you're 14
Imagine you're playing a long game where you need to keep upgrading and adjusting your strategy to win. AI agents are like the characters in the game, needing to constantly try and adjust in long-horizon tasks to find the best solutions. By analyzing these characters' performances at different stages, we can better understand their strengths and weaknesses, providing directions for future improvements. Isn't that cool?
Glossary
Long-Horizon AI
Refers to AI conducting autonomous research in long-term tasks.
Used to evaluate AI agents' performance in long-horizon tasks.
Solution Framing
Analyzes the directions pursued by agents in tasks.
Used to evaluate agents' performance in tasks.
Execution
Evaluates agents' ability to translate plans into executable results.
Used to analyze agents' execution capability.
Feedback Control
Examines how agents use feedback to adjust subsequent decisions.
Used to evaluate agents' feedback utilization capability.
Experience Reuse
Analyzes the effect of experience reuse by comparing performance across tasks.
Used to analyze the impact of experience reuse on agent performance.
Open Questions Unanswered questions from this research
- 1 How to improve AI agents' stability across runs and reduce performance variability.
- 2 How to achieve genuine methodological innovation in long-horizon tasks.
Applications
Immediate Applications
Model Training Optimization
Improve training strategies to enhance AI agents' performance in long-horizon tasks.
Long-term Vision
Autonomous Research Systems
Develop AI systems capable of conducting autonomous research in long-horizon tasks, achieving true autonomous innovation.
Abstract
Autonomous agents are increasingly capable of improving models, systems, and other technical artifacts through long-horizon experimentation. To understand the current state of this capability, however, evaluation must go beyond final scores, which neither reveal where progress is gained or lost nor indicate whether accumulated experience improves later decisions. We therefore present a systematic evaluation of seven frontier models on 36 long-horizon tasks based on a new framework that uses rule-based metrics to characterize within-run behavior through Solution Framing, Execution, and Feedback Control and controlled comparisons to assess experience reuse within and across tasks. The results show that current agents operate more like engineering optimizers than fully autonomous researchers: they can formulate and implement practical solutions, but their performance varies substantially across runs, their strongest solutions mainly adapt or combine established techniques, and genuine methodological novelty remains rare. Detailed analysis reveals that observed performance is shaped by multiple factors, including distinct process bottlenecks behind similar final outcomes, experience reuse that can help or mislead subsequent decisions, and harness designs that affect performance stability. These findings suggest concrete directions for improving model training, inference-time strategies, experience management, and harness design.