WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct
WizardMath employs Evol-Instruct and reinforcement learning, achieving state-of-the-art math reasoning with 92.8% pass@1 on GSM8k and 58.6% on MATH.
Key Findings
Methodology
This work introduces Reinforcement Learning from Evol-Instruct Feedback (RLEIF), combining Math Evol-Instruct for automatic instruction generation with dual reward models (IRM and PRM). The process involves multi-round instruction evolution via GPT-4, which generates diverse math problems by upward and downward pathways. These instructions are ranked and evaluated automatically, then used to fine-tune models through PPO reinforcement learning, optimizing both instruction quality and reasoning step correctness. The approach eliminates reliance on external tools, enabling efficient training. Experiments on GSM8k and MATH benchmarks demonstrate significant performance improvements over existing open-source and proprietary models, with the 70B WizardMath surpassing GPT-4 early versions.
Key Results
- WizardMath-70B achieves 92.8% pass@1 on GSM8k and 58.6% on MATH, outperforming MetaMath-70B (82.3%, 26.6%) and GPT-3.5-Turbo (81.6%, 43.1%) by substantial margins, with high data efficiency.
- WizardMath-Mistral-7B surpasses MetaMath-Mistral-7B (77.9%) on GSM8k (90.7%) and exceeds GPT-4-0314 (52.6%) on MATH (55.4%), validating the scalability and effectiveness of the method.
- Incorporating IRM and PRM during PPO training yields an additional 6-8% performance gain, confirming the critical role of reward models in guiding complex reasoning improvements.
Significance
This research addresses the longstanding challenge of enhancing open-source large models' mathematical reasoning. By automating instruction generation and leveraging process-aware reinforcement learning, it significantly boosts multi-step reasoning performance. The approach reduces reliance on human-labeled data, improves data efficiency, and demonstrates broad applicability across various model architectures. It paves the way for AI systems capable of solving high-level scientific and educational problems, impacting both academia and industry, especially in intelligent tutoring, scientific research, and automated problem solving.
Technical Contribution
The core innovation lies in the RLEIF framework, integrating multi-path instruction evolution with dual reward models to automate training. The method introduces a novel combination of upward and downward instruction pathways, enabling diverse and challenging problem generation. It employs GPT-4 for automatic ranking and correctness assessment, and PPO reinforcement learning for end-to-end optimization. This approach surpasses traditional fine-tuning by explicitly modeling reasoning processes and leveraging process supervision, providing theoretical guarantees of improved reasoning depth and robustness. The framework is adaptable to various base models, demonstrating high scalability.
Novelty
This is the first work to combine automated instruction evolution with process-aware reward modeling in a fully AI-driven reinforcement learning pipeline for mathematical reasoning. Unlike prior methods relying on human annotations or limited datasets, RLEIF dynamically generates diverse, curriculum-like instructions and optimizes models without external tools. The dual reward models ensure both instruction quality and reasoning correctness, addressing hallucination issues and improving interpretability. This comprehensive automation and创新机制 represent a significant leap in scalable, high-performance mathematical reasoning for open-source large models.
Limitations
- 依赖GPT-4进行指令排序和评估，存在偏差和局限性，未来需引入多模态或多模型融合以增强鲁棒性。
- 模型在极端复杂或超出训练范围的问题上仍存在推理失误，演化路径可能限制泛化能力。
- 训练成本较高，尤其是在多轮演化和强化学习过程中，需优化算法和硬件资源配置。
Future Work
未来将结合多模态信息丰富指令内容，拓展到更复杂的科学推理任务。计划引入多模型集成和更先进的奖励机制，以提升模型泛化和推理深度。同时，推动模型在教育、科研等实际场景中的应用，探索跨领域迁移和多任务学习的可能性。
AI Executive Summary
Large language models (LLMs) have revolutionized natural language processing, but their mathematical reasoning abilities remain limited, especially in open-source variants. Existing techniques like Chain-of-Thought prompting improve step-by-step reasoning but still fall short on complex problems requiring multi-step inference. To address this, the present study introduces WizardMath, a novel framework that combines automated instruction evolution with reinforcement learning guided by dual reward models.
The core innovation is the Math Evol-Instruct mechanism, which employs upward and downward pathways to generate diverse, curriculum-like mathematical problems. These instructions are automatically ranked and evaluated using GPT-4, forming the basis for training data. The reward models—Instruction Reward Model (IRM) and Process-supervised Reward Model (PRM)—assess instruction quality and reasoning correctness, respectively. The entire process is integrated into a reinforcement learning pipeline using Proximal Policy Optimization (PPO), enabling the model to learn complex reasoning strategies without external Python tools.
Extensive experiments on GSM8k and MATH benchmarks demonstrate that WizardMath-70B achieves 92.8% pass@1 on GSM8k and 58.6% on MATH, outperforming state-of-the-art open-source models like MetaMath and Mistral, as well as proprietary models such as GPT-3.5-Turbo. Notably, the smaller WizardMath-Mistral-7B surpasses many larger models, validating the scalability and efficiency of the approach. The ablation studies confirm that the reward models significantly enhance reasoning performance, with a combined gain of 6-8%.
This work marks a significant step forward in AI-driven mathematical reasoning, reducing reliance on human-labeled data and external tools. Its broad applicability promises to impact educational technology, scientific research, and automated problem solving, fostering more intelligent and autonomous AI systems. Future directions include multi-modal instruction generation, multi-task learning, and real-world deployment, aiming to push the boundaries of AI reasoning further.
Deep Dive
Abstract
Large language models (LLMs), such as GPT-4, have shown remarkable performance in natural language processing (NLP) tasks, including challenging mathematical reasoning. However, most existing open-source models are only pre-trained on large-scale internet data and without math-related optimization. In this paper, we present WizardMath, which enhances the mathematical CoT reasoning abilities of LLMs without using external python tools, by applying our proposed Reinforcement Learning from Evol-Instruct Feedback (RLEIF) method to the domain of math. Through extensive experiments on two mathematical reasoning benchmarks, namely GSM8k and MATH, we reveal the extraordinary capabilities of our model. Remarkably, WizardMath-Mistral 7B surpasses top-tier open-source LLMs by a substantial margin with higher data efficiency. Furthermore, WizardMath 70B even outperforms GPT-3.5-Turbo, Claude 2, Gemini Pro and GPT-4-early-version. Additionally, our preliminary exploration highlights the pivotal role of instruction evolution and process supervision in achieving exceptional math performance. For more details refer to https://github.com/nlpxucan/WizardLM