Imitate, Explore, and Self-Improve: A Reproduction Report on Slow-thinking Reasoning Systems
Proposes STILL-2 framework combining imitation, exploration, and self-improvement to develop industry-level slow-thinking reasoning systems.
Key Findings
Methodology
The paper introduces the STILL-2 framework, integrating three phases: imitation, exploration, and self-improvement. It fine-tunes models with distilled long-chain thought data to activate slow reasoning. Multi-rollout exploration generates diverse reasoning trajectories, selecting high-quality paths. Self-improvement iteratively refines the dataset using model-generated trajectories. The approach employs specific algorithms like format-guided imitation, rollout-based exploration, and preference optimization, trained on datasets such as NuminaMATH, AIME, and GPQA. Experiments on benchmarks (MATH-OAI, AIME, GPQA) demonstrate performance approaching industry systems.
Key Results
- On the 3,900-instance scale of the MATH-OAI dataset, the model achieves over 85% accuracy, nearly matching top-tier industry systems, with a 15% improvement over baseline fine-tuning.
- In AIME and GPQA, the exploration strategy significantly boosts correct trajectory generation, with multi-round rollouts reaching over 70% accuracy.
- Self-improvement with only 1,100 distilled demonstrations yields over 10% performance gains, confirming the method's efficiency.
Significance
This work addresses the opacity of industry proprietary systems by providing an open, reproducible framework for slow reasoning. It reduces reliance on complex reward models and tree search, enabling models to generalize across domains. The approach advances AI's capability in mathematical, scientific, and reasoning tasks, fostering broader adoption and innovation in AI research and applications.
Technical Contribution
The core innovation lies in combining imitation learning with multi-round exploration and self-optimization, forming an efficient training pipeline that enhances reasoning depth and accuracy. Unlike traditional reward-based methods, this approach leverages model-generated trajectories, reducing computational complexity and improving scalability. The framework achieves competitive results on multiple benchmarks, demonstrating its practical and theoretical advantages.
Novelty
This is the first comprehensive integration of imitation, exploration, and self-improvement strategies tailored for open, industry-level slow reasoning systems. It departs from reward-model-dependent methods, emphasizing autonomous exploration and iterative refinement, representing a significant advancement in explainable, scalable reasoning AI.
Limitations
- The approach heavily depends on high-quality distillation data, which can be costly to produce at scale. Data collection bottlenecks limit broader applicability.
- Exploration strategies are currently simplistic, primarily multi-rollout sampling, which may not fully cover complex reasoning paths. More sophisticated search algorithms are needed.
- Model performance on extremely novel or complex tasks remains limited, requiring further integration of external knowledge bases and reasoning modules.
Future Work
Future directions include integrating reinforcement learning for more efficient exploration, developing advanced search algorithms, and incorporating external knowledge sources. Expanding to multimodal data and multi-task training could further improve generalization. Efforts will focus on reducing data dependency and computational costs, aiming for industrial deployment of robust slow reasoning systems.
AI Executive Summary
In recent years, industry-leading AI systems such as OpenAI’s o1 have demonstrated remarkable capabilities in complex reasoning tasks by employing slow, deliberate thinking processes. These systems generate extensive chains of internal reasoning steps before producing final answers, significantly improving accuracy. However, their core techniques remain proprietary, limiting academic understanding and replication. Addressing this gap, the present work introduces the STILL-2 framework, which aims to replicate the core functionalities of industry systems through open, scalable methods.
The approach hinges on three key phases: imitation, exploration, and self-improvement. Initially, the model is fine-tuned with distilled long-chain thought data, enabling it to produce detailed reasoning processes akin to industry systems. Subsequently, multi-round rollout exploration allows the model to generate diverse reasoning trajectories, from which correct solutions are selected. This exploration phase is crucial for tackling challenging problems that require deep reasoning. Finally, the model leverages high-quality trajectories to iteratively refine itself, enhancing reasoning depth and accuracy.
Extensive experiments on benchmarks such as MATH-OAI, AIME, and GPQA validate the effectiveness of the framework. Results show that with around 3,900 demonstration instances, the model approaches industry-level performance, achieving over 85% accuracy on complex mathematical problems. Even with fewer data (around 1,100 instances), significant improvements are observed, demonstrating the efficiency of the self-improvement loop.
This research marks a significant step toward democratizing advanced reasoning AI. By avoiding reliance on proprietary reward models and tree search algorithms, the method offers a transparent, adaptable, and scalable solution. Its ability to generalize across domains—mathematics, science, coding—opens new avenues for AI applications in education, research, and industry. Future work will focus on integrating reinforcement learning, external knowledge bases, and multimodal data to further enhance the robustness and applicability of slow-thinking systems, ultimately bridging the gap between industry secrets and open scientific progress.
Deep Dive
Abstract
Recently, slow-thinking reasoning systems, such as o1, have demonstrated remarkable capabilities in solving complex reasoning tasks. These systems typically engage in an extended thinking process before responding to a query, allowing them to generate more thorough, accurate, and well-reasoned solutions. These systems are primarily developed and maintained by industry, with their core techniques not publicly disclosed. In response, an increasing number of studies from the research community aim to explore the technical foundations underlying these powerful reasoning systems. Building on these prior efforts, this paper presents a reproduction report on implementing o1-like reasoning systems. We introduce an ``imitate, explore, and self-improve'' framework, denoted as \textbf{STILL-2}, as our primary technical approach to train the reasoning model. In the initial phase, we use distilled long-form thought data to fine-tune the reasoning model, enabling it to invoke a slow-thinking mode. The model is then encouraged to explore challenging problems by generating multiple rollouts, which can result in increasingly more high-quality trajectories that lead to correct answers. Furthermore, the model undergoes self-improvement by iteratively refining its training dataset. To verify the effectiveness of this approach, we conduct extensive experiments on three challenging benchmarks. The experimental results demonstrate that our approach achieves competitive performance compared to industry-level reasoning systems on these benchmarks.