MM-PRM: Enhancing Multimodal Mathematical Reasoning with Scalable Step-Level Supervision
MM-PRM uses MCTS to create 700k+ step labels, raising multimodal math accuracy by up to 12.06 points.
Key Findings
Methodology
The framework has three stages: MM-Policy is fine-tuned on over four million structured mathematical examples; an OmegaPRM-style Monte Carlo Tree Search uses 10,000 MM-K12 seed problems, hierarchical rollouts, and binary search to locate the first erroneous step, yielding about 747,000 labels; MM-PRM then learns from continuous Monte Carlo success probabilities and reranks 16 candidate trajectories under Best-of-N inference.
Key Results
- With MM-PRM, MM-Policy improves from 33.92% to 42.80% on MM-K12, from 15.41% to 24.00% on OlympiadBench, and from 62.93% to 67.60% on MathVista, indicating that step selection suppresses localized logical failures.
- The reward model transfers across generators: InternVL2.5-8B rises from 27.01% to 37.80% on MM-K12, while InternVL2.5-78B improves from 50.18% to 54.47% on MathVerse. Thus, the PRM is not restricted to its training policy.
- Training analysis favors a small learning rate and soft labels; 4e-6 is the best reported PRM learning rate. Continuous labels preserve uncertainty caused by problem difficulty and rollout variability better than binary thresholding.
Significance
MM-PRM moves multimodal mathematical reasoning from judging only final answers to auditing complete reasoning chains. It addresses the false-positive case in which a flawed derivation happens to reach the right answer and offers a scalable alternative to human step annotation. Academically, it connects visual reasoning, process supervision, and search. In industry, it can improve an existing generator through candidate generation and reranking, without retraining the entire large model.
Technical Contribution
The main contributions are a multimodal adaptation of OmegaPRM, an MCTS state tree that preserves image-text context, step values estimated from downstream answer success, and a <prm>-token classifier. At step i, Yes/No logits yield p(i), optimized with soft cross-entropy: −[ŷlogp+(1−ŷ)log(1−p)]. This couples automated annotation, dense criticism, and Best-of-N inference in one pipeline.
Novelty
Compared with human-labeled PRM800K, rollout-based MathShepherd, and text-focused OmegaPRM, this work systematically applies MCTS process supervision to visual mathematics and demonstrates that 10,000 K-12 seed questions can produce more than 700,000 step annotations. The reported gains transfer across datasets and model scales, although the paper does not establish universal superiority over every contemporary PRM.
Limitations
- The supervision comes mainly from K-12 problems and MM-Policy trajectories, so coverage of university-level proofs, very long derivations, and unfamiliar visual concepts remains limited. Distribution shift may reduce both label quality and ranking reliability.
- Best-of-N depends on candidate quality and diversity. Sixteen generations increase inference cost, and the PRM can only select among existing paths; it cannot necessarily repair a mistake when every candidate is wrong.
- Monte Carlo labels reflect the policy distribution rather than formal mathematical truth and may inherit generator bias.
Future Work
Future work should add difficult, multilingual, and proof-oriented data; integrate formal verifiers, tool use, and reinforcement learning; and study efficient search, adaptive candidate budgets, and calibration. Evaluation on substantially different architectures is also needed to test whether the observed transfer is genuinely model-agnostic.
AI Executive Summary
Multimodal large language models can interpret diagrams and produce mathematical answers, yet their reasoning chains often contain silent failures. A model may guess the right result after an invalid step, while an outcome-only reward model sees no problem. Human labeling of every intermediate step is expensive and difficult to scale. MM-PRM addresses this gap by treating reasoning as an auditable sequence rather than a single final prediction.
The authors first assemble data from R-CoT, MAVIS, MathV360K, NuminaMath, and DART-Math, then train MM-Policy on InternVL2.5-8B. From 10,000 MM-K12 problems—5,000 fill-in-the-blank and 5,000 multiple-choice questions—they generate candidate solutions. An OmegaPRM-style Monte Carlo Tree Search repeatedly rolls out partial prefixes, estimates the probability of reaching the correct answer, and uses binary search to identify the earliest failure. The process produces about 747,000 step-level labels. MM-PRM predicts a soft correctness score after every step marked by <prm>.
Under Best-of-N inference, 16 candidate paths are generated and the highest-scoring path is selected. MM-Policy rises from 33.92% to 42.80% on MM-K12, from 15.41% to 24.00% on OlympiadBench, and from 62.93% to 67.60% on MathVista. InternVL2.5-8B improves from 27.01% to 37.80% on MM-K12. A qualitative example gives correct early steps scores of 0.83 and 0.68, but assigns 0.02 to the decisive error. The approach therefore improves robustness, while still paying for search and remaining dependent on candidate diversity, verifiable answers, and policy-model biases.
Deep Analysis
Background
Chain-of-Thought and Self-Consistency have strengthened language-model mathematics, but multimodal systems still break logical chains. PRM800K uses costly human annotations; MathShepherd estimates step quality through repeated rollouts; OmegaPRM uses MCTS for more structured supervision. MM-PRM extends this direction to visual mathematics while emphasizing automation, soft labels, and transfer across generators.
Core Problem
Final-answer supervision cannot distinguish a sound derivation from a lucky guess and provides little information about where a solution failed. Multimodal problems combine visual interpretation, text, and long reasoning chains. Human labeling is expensive, while naive Monte Carlo estimates have high variance, motivating an automated search method that preserves the full visual context.
Innovation
The paper contributes four linked ideas: MM-K12, with 10,000 seed and 500 test problems; a multimodal OmegaPRM-style divide-and-conquer MCTS pipeline producing about 747,000 labels; continuous Monte Carlo targets instead of hard thresholds; and a <prm>-based step classifier used for Best-of-N reranking. Together, these make process supervision scalable without human step labeling.
Methodology
- �� Policy construction: clean over four million examples, use Qwen2.5-72B-Instruct to format <step> and <answer> traces, and fine-tune InternVL2.5-8B into MM-Policy.
- �� Search annotation: generate multiple trajectories for MM-K12; MCTS stores states, actions, and Monte Carlo statistics, while hierarchical rollouts and binary search locate the first failure.
- �� PRM learning: insert <prm> after every step, compute p(i) from Yes/No logits, and train against MC(x<t)∈[0,1] using soft cross-entropy.
- �� Selection: sample N=16 paths and aggregate scores with Min, Average, SumLog, SumLogOdds, or related functions.
Experiments
Evaluation covers MM-K12, OlympiadBench OE_MM_maths_en_COMP, MathVista testmini, MathVerse testmini, and MathVision test. MM-Policy is trained for one epoch with batch size 128 and learning rate 4e-5; the PRM uses one epoch, batch size 512, and learning rate 4e-6. MCTS uses temperature 1.0, top-k 50, top-p 0.9, c_puct 0.125, up to 200 search steps, or 1,000 rollouts per problem.
Results
For MM-Policy, gains across the five benchmarks are +8.88, +8.59, +4.67, +3.28, and +5.37 points. InternVL2.5-38B improves on MM-K12 from 40.34% to 52.40%; InternVL2.5-78B improves on MathVerse from 50.18% to 54.47%. In the geometry example, valid early steps score 0.83 and 0.68, whereas the erroneous third step receives 0.02, demonstrating localized error detection.
Applications
The method fits diagram questions, geometry, educational tutoring, and visual question answering where multiple candidate solutions can be generated and answers can be checked. Deployment requires a capable generator, verifiable outcomes or partial validators, and additional sampling budget. Educational products could provide step-specific feedback instead of merely marking answers wrong.
Limitations & Outlook
The framework assumes a verifiable final answer and sufficiently diverse policy-generated candidates; open-ended proofs, ambiguous images, and non-unique answers are harder. MCTS and Best-of-N increase latency and computation, while soft Monte Carlo targets may inherit policy bias. Future systems should combine formal proof checking, tool use, adaptive search budgets, repair mechanisms, and broader cross-model evaluation.
Plain Language Accessible to non-experts
Imagine a teacher grading homework. A student first submits 16 different solution drafts. A conventional grader checks only the final number, so a lucky guess can pass even when the reasoning is broken. MM-PRM instead reads each line and asks whether the work can still lead to a correct answer from that point.
The system performs this check automatically. From a partial solution, it tries many possible continuations. If most continuations eventually reach the correct answer, that step receives a high score; if they usually fail, it receives a low score. A tree-search procedure quickly narrows down where the first mistake appeared. In this way, 10,000 image-based problems generate roughly 747,000 step-level judgments without people labeling every line.
The final answer is chosen from the best-scoring draft, not simply the first response. MM-Policy improves from 33.92% to 42.80% on MM-K12. Still, the teacher cannot invent a perfect draft if all 16 drafts are wrong, and checking many drafts costs extra time and computing power. The system is therefore a careful selector, not an all-powerful solver.
ELI14 Explained like you're 14
Think of a math game with several checkpoints. A normal robot only checks whether you reached the finish line. That sounds fine, right? Not really! You might take a wrong turn, get lucky near the end, and still win. Or one mistake at checkpoint three might ruin everything afterward.
MM-PRM acts like a smart teammate. It asks the robot to write 16 possible solutions, then checks them checkpoint by checkpoint. After each step it asks, “If we continue from here, how likely are we to reach the correct answer?” Steps that usually lead to success get high scores; suspicious steps get low scores. A search process helps find the earliest bad turn instead of blaming the whole solution.
The researchers trained this checker with 10,000 picture-based math problems and automatically created about 747,000 step records. On MM-K12, accuracy rose from 33.92% to 42.80%, like moving from roughly three correct answers out of ten to more than four. On the harder OlympiadBench, it rose from 15.41% to 24.00%.
But it is not magic. The checker can choose only among the robot’s 16 drafts. If every draft is wrong, it has no good option. Generating and checking all those attempts also takes extra time. Future versions could use calculators, diagrams, theorem tools, or even rewrite a bad step themselves!
Glossary
Process Reward Model
A model that evaluates intermediate reasoning steps rather than only the final answer. Technically, it predicts a step-level correctness or continuation-success score.
MM-PRM scores every step after a <prm> marker.
Monte Carlo Tree Search
A search algorithm that repeatedly simulates continuations and updates statistics in a tree. It balances exploring alternatives with exploiting promising branches.
It locates early errors and creates process labels.
Best-of-N
An inference strategy that samples N candidate responses and selects the one with the highest evaluator score.
The experiments use N=16.
Soft label
A continuous target between zero and one rather than a binary correct/incorrect label. It preserves uncertainty in the underlying estimate.
The target is the Monte Carlo probability of reaching the correct answer.
MM-K12
A multimodal K-12 mathematics dataset introduced by the paper. It contains 10,000 seed problems and a separate 500-problem test set.
It supplies MCTS seeds and in-domain evaluation.
Outcome Reward Model
A reward model that assigns feedback only to the final outcome. It cannot directly identify which intermediate step caused a failure.
The paper contrasts ORMs with PRMs.
Open Questions Unanswered questions from this research
- 1 It remains unclear whether Monte Carlo targets measure mathematical validity or mainly the policy model's preferences and sampling competence. Formal verifiers and human audits are needed to separate these effects.
- 2 Stability under non-unique answers, poor images, open-ended proofs, and very long chains is not established. These settings may violate the assumptions behind rollout success estimates.
- 3 The optimal trade-off between more candidates and higher inference cost is unknown. Future work should study adaptive N, confidence calibration, and compute-aware search.
Applications
Immediate Applications
Educational error diagnosis
Tutoring systems can generate several solution attempts and use MM-PRM to identify the most suspicious step, giving students targeted hints. Deployment requires answer verification, image understanding, and extra sampling budget.
Visual mathematics reranking
Exam and chart-solving systems can rerank candidate answers for geometry, diagrams, and visual word problems. The PRM can be added as an external critic, improving an existing multimodal generator without full retraining.
Long-term Vision
Auditable mathematical agents
Combined with theorem provers, calculators, and diagram tools, PRMs could become an audit layer for planning and evidence chains. Major obstacles include tool latency, error repair, calibration, and cross-domain generalization.
Abstract
While Multimodal Large Language Models (MLLMs) have achieved impressive progress in vision-language understanding, they still struggle with complex multi-step reasoning, often producing logically inconsistent or partially correct solutions. A key limitation lies in the lack of fine-grained supervision over intermediate reasoning steps. To address this, we propose MM-PRM, a process reward model trained within a fully automated, scalable framework. We first build MM-Policy, a strong multimodal model trained on diverse mathematical reasoning data. Then, we construct MM-K12, a curated dataset of 10,000 multimodal math problems with verifiable answers, which serves as seed data. Leveraging a Monte Carlo Tree Search (MCTS)-based pipeline, we generate over 700k step-level annotations without human labeling. The resulting PRM is used to score candidate reasoning paths in the Best-of-N inference setup and achieves significant improvements across both in-domain (MM-K12 test set) and out-of-domain (OlympiadBench, MathVista, etc.) benchmarks. Further analysis confirms the effectiveness of soft labels, smaller learning rates, and path diversity in optimizing PRM performance. MM-PRM demonstrates that process supervision is a powerful tool for enhancing the logical robustness of multimodal reasoning systems. We release all our codes and data at https://github.com/ModalMinds/MM-PRM.