SynthRL: Scaling Visual Reasoning with Verifiable Data Synthesis
SynthRL uses verifiable data synthesis to generate challenging training samples, boosting visual reasoning model performance by +1-2% on benchmarks.
Key Findings
Methodology
SynthRL employs a three-stage pipeline: first, it assesses question difficulty via Monte Carlo rollouts using the target model; second, it leverages a powerful vision-language model (e.g., Gemini-2.5) to generate more challenging question variants while preserving answers; third, it applies a strict verification process, re-evaluating the synthesized questions with the verifier model to ensure correctness and increased difficulty. This process enables scalable augmentation of datasets—e.g., from 8K seed questions to over 11K with 3.3K verified harder questions—significantly improving model generalization across five out-of-domain visual math reasoning benchmarks.
Key Results
- Models trained on the synthesized dataset A-MMK12 outperform baseline models trained solely on seed data, with average accuracy gains of +1.9% on MathVerse (from 51.6% to 53.5%) and +2.0% on WeMath (from 70.6% to 72.6%) at 8K data scale. The improvements are more pronounced on challenging samples, with reasoning steps increasing by 33%, indicating deeper problem complexity. Performance gains grow with larger data scales, reaching a +1.0% overall boost at 8K samples.
- A detailed analysis shows that synthesized questions have a lower pass rate (mean 6.33 vs. 15.10 for seed questions), confirming increased difficulty. The distribution of reasoning steps shifts toward more intermediate levels, and the difficulty ranking confirms the method's effectiveness in generating harder problems that enhance model reasoning capabilities.
- Across five benchmarks, the approach consistently improves out-of-domain generalization, especially on medium and hard difficulty levels, demonstrating its robustness and scalability.
Significance
This work addresses the critical challenge of data scarcity and quality in training deep visual reasoning models. By automating the synthesis of high-quality, verifiable, and increasingly challenging training samples, SynthRL significantly enhances model robustness and generalization. Its scalable pipeline reduces reliance on manual annotation, enabling large-scale dataset expansion without sacrificing correctness. The approach bridges the gap between data quantity and quality, paving the way for more intelligent and reliable AI systems capable of complex reasoning. Its verification mechanism ensures data integrity, making it suitable for industrial deployment where correctness is paramount.
Technical Contribution
SynthRL introduces a novel combination of difficulty-aware seed selection, leveraging Monte Carlo rollouts, with a powerful generative model (Gemini-2.5) for data synthesis. The core innovation lies in the strict verification process, which re-evaluates generated questions to guarantee correctness and increased difficulty, providing theoretical guarantees for data quality. This integrated pipeline allows for scalable, automated dataset augmentation tailored for reinforcement learning with verifiable rewards, enabling models to learn from richer, more challenging examples. The framework advances the state-of-the-art in automatic data generation for complex reasoning tasks, addressing both scalability and data integrity issues.
Novelty
This is the first comprehensive framework combining model-based difficulty assessment, automated challenging question generation, and strict answer verification to produce high-quality, verifiable training data at scale. Unlike prior work limited to manual annotation or simple augmentation, SynthRL systematically enhances data complexity while ensuring correctness, thus directly addressing the core bottleneck in training deep reasoning models. Its end-to-end automation and theoretical guarantees set it apart from existing methods, establishing a new paradigm for scalable, reliable data synthesis in multimodal reasoning.
Limitations
- The approach heavily depends on the accuracy of the target and verifier models; inaccuracies in these models could lead to incorrect or less challenging samples, especially in highly complex or ambiguous cases.
- Computational costs are significant, as multiple rounds of synthesis and verification are required, which may hinder real-time or large-scale industrial deployment without further optimization.
- The current pipeline primarily focuses on visual math reasoning; extending it to other reasoning domains or multimodal tasks may require additional adaptation and validation.
Future Work
Future research will explore integrating multi-modal feedback to further diversify and deepen generated samples, as well as employing reinforcement learning to optimize the synthesis process itself. Enhancing efficiency through model distillation or parallelization will be crucial for industrial-scale deployment. Additionally, extending SynthRL to other reasoning domains, such as commonsense or narrative understanding, could broaden its applicability. Developing adaptive verification thresholds and multi-objective optimization strategies will further improve data quality and diversity.
AI Executive Summary
In recent years, vision-language models (VLMs) have achieved remarkable progress in understanding and reasoning across visual and textual modalities. Yet, their performance on complex, multi-step reasoning tasks remains limited by the scarcity and quality of training data. Traditional data augmentation methods—manual annotation, paraphrasing, or simple synthetic generation—often fall short in providing the depth and challenge necessary for robust reasoning. Addressing this bottleneck, the present work introduces SynthRL, an innovative framework that automates the synthesis of high-quality, verifiable, and increasingly challenging training samples.
SynthRL operates through a three-stage pipeline: first, it assesses the difficulty of seed questions using Monte Carlo rollouts with the target model, selecting questions that are reliably answered but not too easy; second, it employs a powerful vision-language model—such as Gemini-2.5—to generate more challenging variants of these questions, requiring deeper reasoning while maintaining the original answers; third, it rigorously verifies the generated questions by re-evaluating them with a dedicated verifier model, ensuring answer consistency and increased difficulty. This process enables the scalable expansion of datasets, exemplified by augmenting 8K seed questions into over 11K high-quality, harder questions.
Experimental results on five out-of-domain visual math reasoning benchmarks demonstrate the effectiveness of SynthRL. Models trained with the synthesized data outperform baseline models trained solely on seed data, with average accuracy improvements of approximately 1-2%. Notably, the gains are most significant on challenging samples, which require more complex reasoning chains, validating the method’s capacity to promote deeper understanding. The benefits grow with larger datasets, reaching a +1.0% overall improvement at 8K scale, confirming the scalability and robustness of the approach.
This work marks a significant step forward in automated data generation for complex reasoning tasks. By combining model-based difficulty assessment, advanced generative models, and strict verification, SynthRL provides a reliable, scalable solution to enhance the training of visual reasoning models. Its implications extend to various applications, including automated education, intelligent tutoring, and real-world AI deployment, where high-quality, challenging data is crucial. Looking ahead, future efforts will focus on optimizing efficiency, broadening domain coverage, and integrating multi-modal feedback to further push the boundaries of AI reasoning capabilities.
Deep Analysis
Background
视觉语言模型（VLMs）在过去几年中实现了快速发展，从基础的多模态融合到复杂推理能力的提升。代表性工作如Alayrac等（2022）提出的基础融合技术，Li等（2023b）优化的视觉指令调优，以及Shi等（2024）提出的数学推理模型，推动了模型在视觉理解和推理方面的突破。然而，尽管GPT-4o（2024）和Gemini（2023）等模型展现出强大的视觉理解能力，但在多步骤深层推理任务中仍存在性能瓶颈，尤其是在复杂推断和多模态信息整合方面。强化学习（RL）逐渐被引入以增强模型推理深度，尤其在几何、计数等子领域取得一定成功，但训练数据的不足和样本有限仍是主要限制。
Core Problem
当前视觉推理模型在面对复杂、多步骤问题时表现不足，主要源于训练数据的局限性。传统数据集规模有限，难以覆盖推理的多样性和深度，导致模型在实际应用中泛化能力不足。人工标注成本高昂，难以大规模生成高质量、多样化的训练样本。如何在保证答案正确的前提下，自动生成具有挑战性和多样性的训练数据，成为亟待解决的问题。现有方法多依赖人工设计或简单增强，缺乏系统性和可验证性，限制了模型推理能力的提升。
Innovation
本研究的创新点在于提出一套完整的自动合成流程：首先利用蒙特卡洛滚动分析评估样本难度，筛选出适合合成的中等难度问题；其次采用强大视觉语言模型（如Gemini-2.5）自动生成更具挑战性的问题变体，保持答案一致；最后引入严格的验证机制，确保合成样本的正确性和难度递增。该流程实现了高效、可验证的自动数据扩展，突破了传统人工标注的瓶颈，为模型在复杂推理任务中的泛化提供了丰富的训练资源。与现有方法相比，SynthRL在保证答案正确的基础上，显著提升了样本的难度和多样性，为深层推理模型的训练提供了新的技术路径。
Methodology
- �� 样本筛选：基于蒙特卡洛滚动（N=16）分析模型对样本的预测一致性，计算每个问题的通过次数，筛选出中等难度（pass≥12）的问题作为合成对象。
- �� 样本合成：利用Gemini-2.5模型，输入原始图片和问题，生成更具挑战性的问题变体，保持原答案不变。采用特定提示模板引导模型生成深层推理问题。
- �� 验证机制：对合成问题再次进行蒙特卡洛滚动，确保答案一致（pass≥4），同时验证难度提升（pass数减少至少∆hard=2），保证样本的正确性和难度递增。
- �� 数据扩展：将验证通过的样本加入训练集，形成扩展数据集（如A-MMK12），提升模型推理能力。
- �� 实验验证：在五个视觉数学推理基准上测试模型性能，比较仅用种子数据和结合合成数据的模型表现。
Experiments
采用MMK12作为种子数据集，经过筛选和合成后，得到A-MMK12扩展集。训练模型基于Qwen2.5-VL-7B，采用EasyR1框架，设置合理的超参数（学习率1e-6，批量128等）。在五个基准（MathVerse、MathVision等）上评估模型性能，比较仅用种子数据和结合合成数据的模型表现。通过不同规模（2K、4K、8K）训练，分析性能变化和难度适应性。还进行了消融实验，验证验证机制的重要性。
Results
合成数据显著提升模型在五个基准的平均准确率，8K规模时从57.0%提升至58.0%。在MathVerse和WeMath等关键任务上，分别提升1.9%和2.0%。合成样本的推理步骤数增加33%，表明问题复杂度提升。中高难度样本的性能改善尤为明显，验证了合成策略在深层推理中的有效性。随着数据规模扩大，效果逐步增强，显示出合成方法与传统数据扩展的互补优势。
Applications
该方法可广泛应用于需要深层推理能力的视觉问答、自动化教育、智能辅导等场景。通过自动生成高难度样本，降低人工成本，提升模型在复杂环境下的表现。未来结合多模态信息融合和强化学习优化，将推动智能系统在实际复杂任务中的应用落地。
Limitations & Outlook
依赖高性能视觉语言模型的准确性，模型偏差可能影响合成样本的质量。验证机制在极端复杂场景下可能不足，存在误判风险。计算成本较高，需优化效率以适应大规模应用。未来需增强多模态融合能力，提升样本多样性和难度控制的精度。
Plain Language Accessible to non-experts
想象你在一家工厂工作，工厂的任务是生产各种不同的产品。以前，工厂只用固定的模具和原材料，生产的产品样式和难度都差不多。现在，工厂引入了一台智能机器人，可以根据指令自动设计出更复杂、更难做的产品，同时还能保证这些产品的质量。这个机器人会先分析已有的产品，判断哪些比较简单，然后用自己的技术设计出更难的版本，但仍然保证它们符合质量标准。最后，工厂会检查这些新产品，确保它们既符合要求，又比原来更复杂。这样，工厂就能不断生产出更丰富、更难的产品，帮助工人学习和提升技能。这个过程就像SynthRL一样，自动生成更难的问题，确保答案正确，帮助模型变得更聪明。
ELI14 Explained like you're 14
想象你在学校里学数学题，老师给你一些简单的题目让你练习。可是，有时候题太简单，你学不到更多东西。于是，你的老师决定用一台神奇的机器帮你出更难的题，但这些题还是能用你学过的知识解决。这个机器会先看你以前做过的题，判断哪些题你能轻松做对，然后用它的“魔法”把这些题变得更复杂，但答案还是一样。最后，老师会检查这些新题，确保它们既难又正确。这样，你就可以不断挑战自己，学到更深的知识。这就像SynthRL，它自动帮模型出更难的问题，让它变得更聪明、更会推理。
Abstract
Vision-language models (VLMs) trained via reinforcement learning with verifiable reward (RLVR) have shown notable progress in scaling test-time compute effectively. In this work, we investigate how synthesized RL data can further improve RLVR. To this end, we propose \textbf{SynthRL}-a scalable and guaranteed pipeline for automatic data scaling in reasoning-oriented RL training. SynthRL comprises three key stages: (1) selecting seed questions with appropriate distribution, (2) augmenting them into more challenging variants while preserving the original answers, and (3) a guaranteed verification stage that ensures near-perfect correctness and difficulty enhancement. Our empirical experiments demonstrate SynthRL's scalability and effectiveness. When applied to the MMK12 dataset, SynthRL synthesizes over 3.3K additional verifiable, challenging questions from approximately 8K seed samples. Models trained with our synthesized data achieve consistent gains across five out-of-domain visual math reasoning benchmarks, with a significant improvement over baseline models trained on seed data alone. Notably, detailed analysis reveals that the gains are more pronounced on the most challenging evaluation samples, highlighting SynthRL's effectiveness in eliciting deeper and more complex reasoning patterns.