RoadmapBench: Evaluating Long-Horizon Agentic Software Development Across Version Upgrades

Recent progress in large language models (LLMs) such as OpenAI Codex, DeepMind AlphaCode, and Anthropic Claude has demonstrated impressive capabilities in code synthesis and bug fixing. However, these achievements are mostly confined to short-term, isolated tasks like function generation or defect repair, evaluated on benchmarks like HumanEval or MBPP. Real-world software engineering involves long-term, multi-stage processes, including version upgrades, feature additions, and maintenance across multiple files and modules. Existing benchmarks lack the scope to evaluate models' ability to handle such complex workflows, leading to a gap between research progress and practical deployment needs. As software systems grow in complexity, there is an urgent demand for evaluation frameworks that reflect real engineering challenges, including multi-objective coordination, API comprehension, and long-term reasoning.

Current AI models struggle with long-horizon software development tasks that involve multiple interdependent objectives, spanning across numerous files and versions. Traditional benchmarks do not capture the multi-target, multi-stage nature of actual version upgrades, which require understanding complex contextual changes, API semantics, and integration challenges. This gap hampers the development of truly autonomous coding agents capable of managing real-world projects. The core problem is designing an evaluation framework that accurately reflects these complexities, providing meaningful metrics for partial progress, and enabling systematic comparison of models' capabilities in sustained, multi-target development scenarios.

This work introduces RoadmapBench, a benchmark based on real open-source version upgrades, transforming diffs and release notes into structured, multi-target roadmaps. It combines static validation with rollout-based quality control, ensuring task authenticity and robustness. The benchmark covers diverse domains and languages, with tasks involving thousands of lines of code changes, multiple subtasks, and behavioral specifications. The evaluation metrics include weighted subtask completion and partial progress scores, providing nuanced insights into models' abilities beyond binary success. This approach bridges the gap between short-term defect fixing and real-world software evolution, setting a new standard for AI assessment in engineering contexts.

• �� Repository mining: select repositories with active releases and detailed documentation across five languages. • Version pairing: identify consecutive version pairs with substantial code changes and feature narratives. • Task construction: extract diffs, generate multi-target instructions, and adapt test suites to verify behavioral changes. • Static validation: check specification completeness, traceability, and test validity, repairing issues iteratively. • Quality control: run models in a sandboxed environment, attempt to implement each subtask within 2 hours, and attribute failures to task defects or model limitations. • Metrics: compute weighted subtask completion and overall resolution rates, analyze tool usage, step efficiency, and domain difficulty. • Evaluation: assess 13 models, including Claude-Opus-4.7, GPT-5.4, and others, under consistent conditions, recording detailed performance metrics.

The dataset comprises 115 tasks from 17 repositories, covering domains like ML, web, ORM, infrastructure, and UI. Models evaluated include Claude-Opus-4.7, GPT-5.4, Gemini-3.1-Pro, among others. Each task is executed in a Docker environment with a 2-hour limit, ensuring no access to target code or version control beyond the source snapshot. Metrics include task resolution rate, partial progress score, average interaction turns, and output token count. Experiments analyze domain-specific performance, step efficiency, tool invocation patterns, and failure modes. Ablation studies compare scaffold sensitivity and the impact of different validation strategies, providing comprehensive insights into model strengths and weaknesses.

The top model, Claude-Opus-4.7, resolves 39.1% of tasks, while Seed-2.0-Pro resolves only 5.2%. Performance varies significantly across domains, with ML&Data being the most challenging, where models resolve nearly no tasks. The Completion Score, reflecting partial progress, is consistently higher than the full resolution rate, indicating models often partially succeed but fail at later stages. Tool usage analysis shows models predominantly rely on code editing and API calls, with efficiency decreasing as task complexity increases. The results highlight that current models are far from mastering real-world software evolution, especially in multi-file, multi-target scenarios.

This benchmark enables rigorous evaluation of AI models for practical software engineering tasks such as automated version upgrades, feature development, and maintenance. It can guide industry adoption by identifying models capable of handling complex, multi-stage projects, reducing manual effort and errors. In the long term, it supports the development of autonomous development assistants that can manage entire software lifecycles, from initial design to deployment, across diverse domains and languages. The framework also facilitates research into multi-objective reasoning, long-term planning, and multi-file code understanding, accelerating AI-driven automation in software engineering.

The current scope is limited to 17 repositories and 5 languages, restricting diversity. Models still struggle with multi-target coordination and understanding long sequences, indicating fundamental limitations. The 2-hour evaluation window may not fully capture models' potential in extended interactions. Future work should include more repositories, richer task complexity, and multi-modal inputs to better reflect industrial scenarios. Additionally, integrating reinforcement learning and planning modules could enhance long-term reasoning capabilities.