LMM-R1: Empowering 3B LMMs with Strong Reasoning Abilities Through Two-Stage Rule-Based RL
LMM-R1 employs a two-stage rule-based RL, boosting 3B multimodal models' reasoning by 4.83% on average, enabling effective transfer from text to vision tasks.
Key Findings
Methodology
LMM-R1 utilizes a two-phase training framework: first, Foundational Reasoning Enhancement (FRE) employs rule-based RL on large-scale text reasoning datasets (e.g., DeepScaler-Preview-Dataset, VerMulti) to strengthen logical inference. This stage constructs a solid reasoning base. Next, Multimodal Generalization Training (MGT) continues RL on geometry, perception, and agent tasks (e.g., Sokoban, football) to transfer reasoning skills to visual contexts. The approach combines PPO with symbolic verification rewards, ensuring structured reasoning chains. This design addresses data scarcity and model capacity limits, enabling small models to perform complex reasoning effectively.
Key Results
- On multimodal benchmarks, MGT-Geo improves accuracy by 3.21% over the baseline, reaching 41.80% on geometric tasks. FRE-Text boosts pure reasoning tasks (MATH500) by 2%, GPQA by 6.57%, confirming reasoning enhancement. Direct multi-modal RL training degrades performance, highlighting the importance of foundational reasoning. Across visual QA, scientific reasoning, and agent tasks, models show an average gain of 4.83%, demonstrating successful transfer. The football task improves by 3.63%, validating real-world applicability.
- Results indicate that pretraining with text reasoning (FRE-Text) enhances core inference, while subsequent multi-modal training (MGT) broadens visual understanding. The two-stage process outperforms direct multi-modal training, which often causes reasoning degradation. The approach effectively balances reasoning and perception capabilities, making small models more versatile.
- In agent scenarios like Sokoban and football, models demonstrate planning, goal recognition, and strategic decision-making, confirming the method's practical potential. Overall, LMM-R1 achieves significant performance boosts, establishing a new paradigm for data-efficient reasoning in small multimodal models.
Significance
This work addresses the critical challenge of enhancing reasoning in small multimodal models without relying on costly large-scale multimodal data. By leveraging text-only reasoning datasets and a structured two-phase training, it enables models with limited parameters to perform complex inference tasks across diverse domains. The approach reduces data dependency, lowers training costs, and improves generalization, offering a scalable solution for deploying intelligent systems in real-world applications such as education, robotics, and autonomous vehicles. The methodology's novelty lies in its strategic transfer of reasoning skills from language to vision, opening new avenues for efficient AI development. Its success demonstrates that high reasoning performance is achievable even in resource-constrained settings, potentially transforming how multimodal AI systems are designed and trained.
Technical Contribution
The core technical innovation of LMM-R1 lies in integrating rule-based reinforcement learning with a two-stage training process tailored for small parameter models. The first stage, FRE, uses structured textual reasoning data to build a robust logical foundation, employing PPO combined with symbolic verification rewards to enforce reasoning chain correctness. The second stage, MGT, extends these capabilities to diverse multimodal tasks, including geometric, perception, and agent-based problems, through continued RL training. This design effectively mitigates data scarcity and model capacity limitations, enabling the transfer of reasoning skills. The framework introduces a novel synergy between symbolic verification and rule RL, ensuring reasoning chains are both accurate and interpretable, which is a significant departure from conventional end-to-end deep learning approaches.
Novelty
This research is pioneering in applying a two-stage rule-based RL framework specifically to small (3B) multimodal models for reasoning enhancement. Unlike prior works that rely heavily on large datasets or end-to-end supervised learning, this approach leverages high-quality text reasoning data to establish a reasoning backbone, then generalizes it to visual tasks. The integration of symbolic verification within reinforcement learning ensures structured, correct reasoning chains, a novel mechanism in this context. The method demonstrates that small models can achieve near state-of-the-art reasoning performance without extensive multimodal data, marking a significant advancement in data-efficient AI. Its unique focus on transfer learning from language to vision sets it apart from existing multimodal training paradigms.
Limitations
- Despite improvements, the model struggles with extremely complex reasoning chains, especially when visual data is ambiguous or limited, indicating that the current training data and symbolic verification may not fully capture all reasoning nuances.
- The symbolic verification process incurs high computational costs, which could hinder large-scale deployment or real-time applications, especially in resource-constrained environments.
- The framework's effectiveness has been validated mainly on geometric, perception, and agent tasks; its generalization to other complex reasoning domains remains to be tested and optimized.
Future Work
Future research will explore integrating external knowledge bases and self-supervised learning to further enhance reasoning robustness. Developing more efficient symbolic verification algorithms could reduce computational overhead. Extending训练到更大规模模型和多任务场景，验证其在实际复杂环境中的适应性。此外，结合多模态预训练与迁移学习策略，提升模型在多样化应用中的表现，推动多模态推理向更深层次发展。
AI Executive Summary
Deep Dive
Plain Language Accessible to non-experts
想象你在一家厨房做饭。你有很多不同的食材，比如蔬菜、肉和调料。每次做菜都需要按照一定的步骤，比如先切菜，再炒，然后加调料。有时候，菜谱很复杂，需要你理解每一步的逻辑，才能做出美味的菜。现在，假设你有一个聪明的助手，它一开始只会用文字说明怎么做菜（比如“先切菜”），但它还不懂看图片。你告诉它一些基本的做菜规则，它就学会了用文字推理。然后，你让它看一些图片，比如菜的样子、厨具的摆放，帮它理解不同的场景。最后，它不仅能用文字做菜，还能看懂图片，帮你做出各种复杂的菜。这就像LMM-R1，它先用文字学会推理，再用视觉信息扩展能力，变得更聪明、更能理解复杂场景。
Abstract
Enhancing reasoning in Large Multimodal Models (LMMs) faces unique challenges from the complex interplay between visual perception and logical reasoning, particularly in compact 3B-parameter architectures where architectural constraints limit reasoning capacity and modality alignment. While rule-based reinforcement learning (RL) excels in text-only domains, its multimodal extension confronts two critical barriers: (1) data limitations due to ambiguous answers and scarce complex reasoning examples, and (2) degraded foundational reasoning induced by multimodal pretraining. To address these challenges, we propose \textbf{LMM-R1}, a two-stage framework adapting rule-based RL for multimodal reasoning through \textbf{Foundational Reasoning Enhancement (FRE)} followed by \textbf{Multimodal Generalization Training (MGT)}. The FRE stage first strengthens reasoning abilities using text-only data with rule-based RL, then the MGT stage generalizes these reasoning capabilities to multimodal domains. Experiments on Qwen2.5-VL-Instruct-3B demonstrate that LMM-R1 achieves 4.83\% and 4.5\% average improvements over baselines in multimodal and text-only benchmarks, respectively, with a 3.63\% gain in complex Football Game tasks. These results validate that text-based reasoning enhancement enables effective multimodal generalization, offering a data-efficient paradigm that bypasses costly high-quality multimodal training data.