AIBuildAI-2: A Knowledge-Enhanced Agent for Automatically Building AI Models
AIBuildAI-2 uses hierarchical knowledge system to automate AI model building, achieving 70.7% medal rate.
Key Findings
Methodology
AIBuildAI-2 employs a hierarchical knowledge system with high-level instructions and low-level documents, dynamically retrieved based on task context. It integrates multi-agent architecture (designer, coder, tuner) with external knowledge sources, built from web data, cleaned, structured, and continuously refined through self-experience analysis. The system leverages algorithms like XGBoost, LightGBM, and CheMeleon for multi-task prediction, with knowledge retrieval optimized via context-aware mechanisms. The process involves web crawling, knowledge distillation, and experience-based updates, enabling adaptive, high-performance model development.
Key Results
- On MLE-Bench, AIBuildAI-2 ranks first with a 70.7% medal rate, outperforming baselines like R&D-Agent and AIRA-dojo. It maintains strong performance across task complexities, demonstrating robustness. In heart disease prediction, it achieves an AUC of 0.95529, placing in the top 6.6%, matching expert-level performance with ensemble models (CatBoost, XGBoost, LightGBM).
- In drug ADMET prediction, combining graph neural networks (CheMeleon) and traditional ML yields an average absolute relative error of 0.63, ranking top 40. The continual knowledge update mechanism enhances adaptability and accuracy, surpassing static knowledge-based systems.
- The self-updating knowledge system, analyzed via run traces, significantly boosts model performance and generalization, validating the core innovation of dynamic knowledge integration.
Significance
This work advances autonomous AI development, lowering barriers for non-experts and accelerating scientific discovery. The hierarchical, self-evolving knowledge system addresses static knowledge limitations, enabling models to adapt rapidly to new techniques and data. It bridges the gap between human expertise and automation, fostering broader adoption in research and industry, and setting new standards for AI automation efficiency.
Technical Contribution
Key contributions include the hierarchical knowledge architecture, dynamic retrieval mechanisms, and self-updating strategies driven by large language models. The multi-agent framework orchestrates end-to-end model development, grounded in external expertise. The system’s ability to analyze experience, update knowledge, and improve performance represents a significant leap over prior static or search-based methods, offering a scalable, adaptable platform for autonomous AI.
Novelty
This is the first integration of a hierarchical, continually evolving knowledge system with multi-agent autonomous modeling. Unlike static databases or simple retrieval, it dynamically updates and refines knowledge based on real experience, enabling continuous learning and adaptation. This approach fundamentally enhances the autonomy and robustness of AI model building.
Limitations
- Dependence on web data quality may introduce noise, affecting knowledge accuracy. Handling noisy or outdated sources remains challenging.
- In highly novel or complex tasks, the knowledge base may lack sufficient coverage, limiting performance.
- Computational costs for continuous knowledge updating and multi-agent coordination are high, impacting scalability.
Future Work
Future directions include integrating reinforcement learning for better knowledge retrieval policies, expanding multi-modal knowledge sources, and optimizing system efficiency. Developing domain-specific knowledge modules and improving scalability will further enhance autonomous AI capabilities, pushing toward fully self-sufficient AI research assistants.
AI Executive Summary
AIBuildAI-2 marks a significant milestone in automated AI model development. Traditional approaches rely heavily on manual design, hyperparameter tuning, and iterative experimentation, which are time-consuming and require specialized expertise. This limits accessibility for many scientific researchers and hampers rapid innovation. To address this, the authors propose a hierarchical knowledge system that dynamically retrieves relevant external expertise, combined with a multi-agent architecture that automates the entire modeling pipeline.
The core innovation lies in organizing AI development knowledge into high-level instructions and detailed documents, which are continually refined through self-experience analysis. This setup allows the system to adaptively update its strategies, effectively bridging the gap between static knowledge and evolving best practices. The multi-agent framework—comprising design, coding, and tuning agents—works collaboratively, guided by the knowledge system, to generate high-performance models across diverse tasks.
Experimental results demonstrate the system’s effectiveness. On the MLE-Bench, a comprehensive benchmark for end-to-end AI development, AIBuildAI-2 achieved a 70.7% medal rate, outperforming all baselines. It also matched expert human performance in a heart disease prediction challenge, with an AUC of 0.95529, and excelled in drug ADMET property prediction, ranking in the top 40 out of 103 teams.
These results highlight the potential of combining hierarchical, self-updating knowledge with autonomous agents to democratize AI development. The approach reduces reliance on expert intervention, accelerates research cycles, and adapts swiftly to new scientific advancements. Looking ahead, further integration of reinforcement learning, multi-modal data, and efficiency optimizations will push the boundaries of fully autonomous AI research systems, transforming how scientific discovery and industrial automation are conducted.
Deep Dive
Abstract
AI models underpin data-centric applications from image and text processing to scientific discovery in biology, physics, and chemistry. Yet developing them remains heavily manual, requiring practitioners to design architectures, build training pipelines, and iteratively refine solutions, making it challenging for natural scientists without specialized AI engineering expertise to build the high-performing models their research demands. To reduce this burden and broaden access to AI for scientific discovery, agents that automatically build AI models have been proposed. However, the performance of these agents is largely limited by the parametric knowledge of their underlying large language models, which is static, often outdated, and sparse on practical AI model engineering know-how. To address this limitation, we introduce AIBuildAI-2, a knowledge-enhanced agent with an external, evolving knowledge system for automatically building AI models. The knowledge system of AIBuildAI-2 is hierarchical, organizing curated AI development knowledge into high-level knowledge instructions over topical categories and low-level knowledge documents under each category, from which the agent dynamically loads only the context relevant to its current state and the AI task being solved, grounding each design and implementation decision in concrete, externally verifiable expertise. The system is initialized by collecting and cleaning AI-development-related documents from the web and organizing them into the corresponding categories, and continually evolves from the agent's own experience by distilling each completed run on an AI task into structured takeaways that are written back into the knowledge system. AIBuildAI-2 achieves state-of-the-art results, ranking first on MLE-Bench with a 70.7% medal rate and placing in the top 6.6% among 4,370 human-expert teams in a heart disease prediction competition.