CodeMidas: Scaling Agentic Coding RL Environments from Code Itself
CodeMidas builds RL environments from source code, enhancing MiMo-V2.5 performance across five benchmarks.
Key Findings
Methodology
CodeMidas constructs executable RL environments using source code. It involves task design, test construction, execution consistency checks, and post-rollout filtering, exploring codebase functionality to formulate behavioral specifications.
Key Results
- Training MiMo-V2.5 on DeepSWE benchmark improved performance by 11.7%, ProgramBench by 17%, and Terminal-Bench v2.1 by 8.5%.
- Increasing high-quality training tasks significantly boosts performance, with the 3k task set outperforming the unfiltered 8k task set.
- Post-RL training, agents exhibit better codebase exploration and diverse self-verification behaviors.
Significance
CodeMidas transforms source code into scalable RL environments, addressing the limitations of existing methods reliant on development artifacts, thereby expanding task diversity.
Technical Contribution
CodeMidas offers a method to construct RL environments directly from source code, reducing dependency on development records and enhancing task diversity and verification reliability.
Novelty
CodeMidas is the first to build RL environments from source code itself, differing from traditional methods reliant on development artifacts, providing broader task coverage.
Limitations
- CodeMidas may face challenges with complex codebases lacking clear interfaces.
- The task filtering process might overlook potentially valuable tasks.
Future Work
Future work could explore optimizing the task filtering process and handling more complex codebases.
AI Executive Summary
CodeMidas is an innovative reinforcement learning environment construction method that creates diverse task sets from source code itself. Existing methods typically rely on development artifacts like issues and commits, limiting the range of extractable tasks. CodeMidas formulates behavioral specifications by exploring codebase functionality and constructs tests by executing the original code, thereby creating executable RL environments. Experimental results show significant performance improvements for MiMo-V2.5 across multiple benchmarks, notably DeepSWE, ProgramBench, and Terminal-Bench v2.1. This method not only enhances agent codebase exploration capabilities but also promotes diverse self-verification behaviors. This research offers new insights into RL environment construction with broad application potential.
Deep Analysis
Background
In recent years, reinforcement learning has made significant strides in code generation. Traditional methods rely on development records like issues and commits, limiting task diversity. CodeMidas addresses this by constructing RL environments directly from source code.
Core Problem
Existing methods rely on development artifacts, limiting task diversity and verification reliability. The challenge is how to construct RL environments directly from source code.
Innovation
CodeMidas constructs RL environments from source code, offering broader task coverage. Its innovation lies in using codebase functionality to formulate behavioral specifications and constructing tests by executing the original code.
Methodology
- �� Task Design: Identify codebase functionality and formulate behavioral specifications.
- �� Test Construction: Construct tests by executing the original code.
- �� Execution Consistency Checks: Ensure task execution stability.
- �� Post-rollout Filtering: Filter tasks using agent rollouts.
Experiments
Experiments trained MiMo-V2.5 on 5,545 tasks, evaluating performance across five benchmarks. Training used the GRPO algorithm, observing agent performance improvements on various tasks.
Results
On the DeepSWE benchmark, MiMo-V2.5's pass rate increased from 10.0% to 21.7%; on ProgramBench, the Almost Solved score rose from 4.5% to 21.5%.
Applications
CodeMidas can be applied in automated code generation, software development, and test optimization, particularly in scenarios requiring diverse task sets.
Limitations & Outlook
CodeMidas may face challenges with complex codebases, and the task filtering process might overlook valuable tasks. Future work can explore optimizing the filtering process.
Plain Language Accessible to non-experts
Imagine a kitchen where CodeMidas is the smart chef. It creates a variety of dishes (tasks) from the ingredients (source code). Traditional chefs rely on recipes (development records), but CodeMidas draws inspiration directly from the ingredients, creating more diverse dishes. Through continuous trial and adjustment, CodeMidas can make dishes that better suit the customer's taste.
ELI14 Explained like you're 14
Hey, imagine you're playing a game, and CodeMidas is your super helper. It finds various tasks from the game code, like discovering hidden treasures on the game map. Traditional methods rely on game guides, but CodeMidas finds clues directly from the game, helping you level up faster! Isn't that cool?
Glossary
Reinforcement Learning
A machine learning method that trains agents through rewards and penalties.
Used to train code generation agents.
Source Code
The original code files of software, containing the implementation of functionalities.
Foundation for constructing RL environments.
MiMo-V2.5
A reinforcement learning model for code generation.
Trained on CodeMidas task set.
GRPO
An optimization algorithm used for training reinforcement learning models.
Used to train MiMo-V2.5.
Task Filtering
Filtering tasks through agent rollouts and verification results.
Ensures task quality and diversity.
Open Questions Unanswered questions from this research
- 1 Handling complex codebases lacking clear interfaces remains a challenge.
- 2 The task filtering process might overlook potentially valuable tasks.
Applications
Immediate Applications
Automated Code Generation
Developers can use CodeMidas to generate diverse code tasks, improving development efficiency.
Long-term Vision
Software Development Optimization
CodeMidas has the potential to transform software development processes by automating task generation to enhance productivity.
Abstract
Training capable coding agents via reinforcement learning (RL) requires diverse tasks with reliable verifiers. Open-source codebases offer a rich source of such tasks, while existing methods typically rely on development artifacts such as issues and commits, limiting the range of tasks that can be extracted. To better scale RL environments, we present CodeMidas, an agentic pipeline that turns implemented functionality in existing codebases into executable RL environments using source code as its only task-specific input. CodeMidas allocates agentic compute to every stage of environment construction: agents explore implemented functionality to formulate behavioral specifications, construct tests grounded in execution of the original code, and validate and filter candidate tasks through execution checks and repeated solution rollouts. The resulting dataset has 5,545 training tasks from 3,185 open-source codebases spanning 23 programming languages and 15 technical domains. Training MiMo-V2.5 on these tasks with GRPO improves performance on all five diverse benchmarks, covering issue repair (DeepSWE + 11.7%), whole-program construction (ProgramBench +17%), and terminal work (Terminal-Bench v2.1 +8.5%). Ablations show that increasing the number of high-quality training tasks improves performance. Trajectory analysis shows the RL-trained agent demonstrates better behaviors like increasing codebase exploration and more diverse self-verification. These results establish source code as a scalable foundation for constructing RL environments that improve coding agents across diverse software tasks.