Towards Execution-Grounded Automated AI Research

The evolution of automated AI research has seen notable milestones such as AutoML, NAS, and recent large language models like GPT-4 and GPT-5. These advances have enabled significant progress in model optimization and architecture search. However, manual intervention remains essential for hypothesis formulation, experiment design, and validation, limiting scalability. Recent efforts like AutoGPT and AutoML Zero have attempted to automate parts of the research cycle, but challenges persist in automating open-ended scientific exploration, especially in generating and validating novel ideas efficiently. The need for scalable, high-throughput systems that can automatically implement and evaluate ideas in real-world environments remains unmet, hindering rapid scientific progress.

The core challenge addressed is how to automate the entire cycle of scientific idea generation, implementation, and validation. Existing methods rely heavily on human expertise, which constrains scalability and speed. Automating idea realization requires a system capable of understanding natural language proposals, translating them into executable code, running experiments, and interpreting results—all at scale. Achieving this in open-ended research scenarios, where ideas are diverse and complex, is particularly difficult. Moreover, enabling models to learn from execution feedback to improve future ideas remains an open problem, especially given the risk of mode collapse in reinforcement learning approaches.

The key innovations include: 1) an automated idea executor that can implement and evaluate hundreds of ideas in parallel, significantly increasing throughput; 2) a novel execution-guided evolutionary search framework that iteratively refines ideas based on execution feedback, demonstrating high sample efficiency; 3) critical analysis of reinforcement learning's limitations in open-ended settings, revealing mode collapse issues; 4) integration of large frontier LLMs as both ideators and executors, enabling fully automated idea generation and validation. These innovations collectively push the frontier of autonomous scientific discovery, enabling scalable, high-quality research automation.

• �� Develop an automated executor system with three core components: Implementer (generates code diffs and applies changes), Scheduler (allocates resources), Worker (runs experiments and uploads results).• Construct two research environments: pretraining (based on nanoGPT, optimizing validation loss within fixed time) and posttraining (based on GRPO, optimizing accuracy).• Sample ideas from frontier LLMs like Claude-4.5-Opus, Claude-4.5-Sonnet, and GPT-5, and generate code implementations automatically.• Use the executor to run experiments in parallel, collecting performance metrics such as accuracy and training time.• Apply evolutionary search: sample, mutate, and select ideas based on execution results, balancing exploration and exploitation across epochs.• Incorporate reinforcement learning: use execution reward signals to fine-tune the ideator model, monitoring for mode collapse and diversity loss.

The experiments involved large-scale GPU clusters running hundreds of ideas sampled from top-tier LLMs. In the pretraining environment, the optimized recipe reduced training time from 35.9 to 19.7 minutes, outperforming NanoGPT baseline. In post-training, accuracy improved from 48.0% to 69.4%. The success rate of executing ideas exceeded 90%, validating the system's robustness. Different models showed varying scalability, with Claude-4.5-Opus demonstrating clear scaling trends, while GPT-5 and Claude-4.5-Sonnet saturated early. The experiments also compared evolutionary search with baseline algorithms and human experts, showing superior performance of the former in efficiency and results.

The automated system achieved a 69.4% accuracy in post-training, surpassing the baseline of 48.0%. It shortened pretraining time from 35.9 to 19.7 minutes, outperforming NanoGPT. Over 90% of ideas sampled from frontier LLMs were successfully implemented and tested. Evolutionary search proved highly sample-efficient, outperforming standard algorithms within ten epochs. Reinforcement learning improved average rewards but failed to enhance maximum rewards due to mode collapse. The results demonstrate the feasibility and effectiveness of fully automated idea validation in realistic research environments.

This framework can be applied to accelerate model development, hyperparameter tuning, and algorithm discovery across AI research. It enables large-scale hypothesis testing, reducing manual effort and time. In industry, it can streamline tasks like automated model compression, architecture search, and scientific hypothesis validation. The system’s scalability makes it suitable for complex, multi-modal research scenarios, fostering autonomous AI-driven innovation in fields like healthcare, materials science, and natural language processing.

The current system primarily targets GPU-intensive tasks, limiting applicability to broader scientific domains. Reinforcement learning faces challenges like mode collapse, reducing idea diversity. High computational costs restrict widespread deployment. The novelty and utility of generated ideas require further validation, especially in real-world applications. Future work should focus on improving algorithm robustness, reducing resource demands, and extending to multi-modal, multi-task research environments.