FT-Dojo: Towards Autonomous LLM Fine-Tuning with Language Agents
FT-Dojo automates LLM fine-tuning with FT-Agent, excelling in 10 out of 13 tasks.
Key Findings
Methodology
FT-Dojo is an interactive benchmark for autonomous LLM fine-tuning. The FT-Agent framework uses structured iteration planning, fail-fast validation, and multi-level feedback analysis to optimize data and training strategies. This design makes data construction and fine-tuning configuration primary optimization targets.
Key Results
- FT-Agent excels in 10 out of 13 tasks, achieving 11.11% accuracy in the AIME 2025 task, where other baselines scored 0%.
- In Chemistry Molecule Editing, FT-Agent achieved 54.44% accuracy, surpassing OpenHands' 40.00%.
- In Table QA Visualization, FT-Agent achieved a Pass@1 of 29.33%, outperforming OpenHands' 24.00%.
Significance
The introduction of FT-Dojo and FT-Agent provides new benchmarks and tools for automating LLM fine-tuning, significantly reducing the need for manual intervention. This research offers new perspectives on model adaptability in specific domains, advancing the field of automated machine learning.
Technical Contribution
FT-Dojo standardizes task interfaces, data repositories, and feedback protocols, while FT-Agent introduces structured iteration and fail-fast validation mechanisms, significantly improving fine-tuning efficiency and effectiveness. Compared to existing methods, FT-Agent excels in handling complex tasks.
Novelty
FT-Dojo is the first interactive benchmark focused on LLM fine-tuning, and FT-Agent achieves breakthrough performance across multiple tasks through automated iterative optimization.
Limitations
- FT-Agent still struggles with causal diagnosis and long-term planning, potentially leading to resource waste.
- In some tasks, FT-Agent's performance is still below manual fine-tuning.
- Limited understanding of complex feedback signals.
Future Work
Future research could explore FT-Agent's application in more domains and improve its causal diagnosis and long-term planning capabilities to further enhance automated fine-tuning efficiency and effectiveness.
AI Executive Summary
FT-Dojo is an emerging interactive benchmark environment designed to automate large language model (LLM) fine-tuning through FT-Agent. The current fine-tuning process often requires significant human effort, but FT-Dojo simplifies this by standardizing task interfaces and feedback protocols.
FT-Agent is an agent framework specifically designed for fine-tuning, utilizing structured iteration planning, fail-fast validation, and multi-level feedback analysis to optimize data and training strategies. Experimental results show that FT-Agent excels in 10 out of 13 tasks, achieving breakthrough performance in the AIME 2025 task.
While FT-Agent performs exceptionally well in multiple tasks, it still has limitations in causal diagnosis and long-term planning. Future research will focus on enhancing FT-Agent's diagnostic and planning capabilities to further advance the application and development of automated fine-tuning.
Deep Analysis
Background
In recent years, large language models (LLMs) have achieved significant advancements across various fields. However, applying these models to specific domains still requires complex fine-tuning processes, which often demand substantial human effort and expertise. Existing fine-tuning methods largely rely on manual data curation and training configuration, which are inefficient and difficult to scale.
Core Problem
The current LLM fine-tuning process is complex and time-consuming, involving data curation, training configuration, and model behavior diagnosis. Automating this process to improve efficiency and effectiveness is the core problem of this research.
Innovation
FT-Dojo simplifies the fine-tuning process by standardizing task interfaces and feedback protocols. FT-Agent introduces structured iteration planning and fail-fast validation mechanisms, significantly improving fine-tuning efficiency. Compared to traditional methods, FT-Agent is better equipped to handle complex tasks.
Methodology
- �� FT-Dojo standardizes task interfaces and feedback protocols, simplifying the fine-tuning process.
- �� FT-Agent uses structured iteration planning to optimize data and training strategies.
- �� Fail-fast validation mechanisms improve fine-tuning efficiency, reducing resource waste.
- �� Multi-level feedback analysis helps the agent better understand and optimize model behavior.
Experiments
Experiments were conducted on 13 tasks in the FT-Dojo suite, covering fields such as mathematics, chemistry, and finance. Each experiment was run on a single GPU, limited to 12 hours. The Qwen2.5-7B-Instruct model was used as the base model, with training samples capped at 2,000.
Results
FT-Agent achieved the best performance in 10 out of 13 tasks, notably achieving 11.11% accuracy in the AIME 2025 task, where other baselines scored 0%. In Chemistry Molecule Editing, FT-Agent achieved 54.44% accuracy, surpassing OpenHands' 40.00%.
Applications
FT-Dojo and FT-Agent can be used for automated model fine-tuning in specific domains, reducing manual intervention and improving efficiency. Their applications in mathematics, chemistry, and finance show promising potential.
Limitations & Outlook
While FT-Agent performs exceptionally well in multiple tasks, it still has limitations in causal diagnosis and long-term planning. Future research will focus on enhancing FT-Agent's diagnostic and planning capabilities.
Plain Language Accessible to non-experts
Imagine you're in a kitchen preparing a grand meal. You need to choose ingredients, set cooking times, and adjust seasonings based on taste feedback. FT-Dojo acts like a smart kitchen assistant, helping you automatically select ingredients and adjust cooking parameters. FT-Agent is like an experienced chef, quickly identifying issues and adjusting strategies to ensure each dish reaches its best flavor. In this way, FT-Dojo and FT-Agent greatly simplify the complex cooking process, just as they simplify the fine-tuning process for LLMs.
ELI14 Explained like you're 14
Imagine you're playing a complex game where you need to constantly adjust strategies to win. FT-Dojo is like a super-smart game assistant, helping you automatically choose the best strategies. FT-Agent is the game expert, quickly spotting problems and adjusting strategies to ensure you achieve the best performance in the game. In this way, FT-Dojo and FT-Agent make complex games simple, just as they simplify the fine-tuning process for LLMs.
Glossary
FT-Dojo
An interactive benchmark environment for automating LLM fine-tuning, standardizing task interfaces and feedback protocols.
FT-Dojo is used to evaluate agent performance in LLM fine-tuning.
FT-Agent
An agent framework specifically designed for fine-tuning, utilizing structured iteration planning and fail-fast validation mechanisms.
FT-Agent is used in FT-Dojo to optimize data and training strategies.
LLM (Large Language Model)
A large-scale neural network model capable of generating and understanding natural language.
LLMs are used in various fields for natural language processing tasks.
CoT (Chain of Thought)
A technique used to explain and enhance model reasoning capabilities by providing intermediate reasoning steps to improve model performance.
In the AIME 2025 task, CoT is used to generate training data.
OpenHands
A general-purpose code execution and automation system suitable for various tasks.
OpenHands is used as a baseline for comparison in FT-Dojo.
Open Questions Unanswered questions from this research
- 1 How to improve FT-Agent's performance in causal diagnosis and long-term planning?
- 2 How to apply FT-Dojo and FT-Agent in more domains?
- 3 How to further reduce resource waste in the automated fine-tuning process?
Applications
Immediate Applications
Domain-specific Fine-tuning
FT-Dojo and FT-Agent can be used for automated model fine-tuning in specific domains, reducing manual intervention and improving efficiency.
Long-term Vision
Comprehensive Automated Fine-tuning
By improving FT-Agent's diagnostic and planning capabilities, achieve broader applications of automated fine-tuning.
Abstract
Fine-tuning large language models for vertical domains remains labor-intensive, requiring practitioners to curate data, configure training, and iteratively diagnose model behavior. Despite growing interest in autonomous machine learning and language agents, end-to-end LLM fine-tuning has not been systematically studied as an interactive agent task. We introduce FT-Dojo, an interactive benchmark environment for autonomous LLM fine-tuning, comprising 13 tasks across 5 domains. Rather than a new collection of static datasets, FT-Dojo standardizes a task interface, shared raw-data repository, sandboxed execution environment, structured feedback protocol, and held-out evaluation procedure. We further develop FT-Agent, a fine-tuning-oriented autonomous framework that uses structured iteration planning, fail-fast validation, and multi-level feedback analysis to refine data and training strategies. Experiments show that FT-Agent provides a strong initial baseline, achieving the best performance on 10 out of 13 tasks, with additional controlled comparisons against frontier agents, open-source planning backbones, and multi-run statistics supporting the main findings. Case studies show that agents can recover from failures through cumulative learning, while still exposing limitations in causal diagnosis and long-horizon planning. The implementation is available at https://github.com/microsoft/rd-agent.