MathCoder-VL: Bridging Vision and Code for Enhanced Multimodal Mathematical Reasoning
MathCoder-VL uses executable code for vision–math alignment, reaching 73.6% on MathVista GPS.
Key Findings
Methodology
The framework has two stages: image-to-code mid-training on ImgCode-8.6M, followed by mathematical instruction tuning on MM-MathInstruct-3M. FigCodifier, built on InternVL2-8B, translates mathematical images into TikZ or Python. Executing the code renders ImageC, yielding intrinsically aligned image–code pairs. With temperature 0.7, FigCodifier synthesizes new figures; Qwen2.5-72B-Instruct creates questions, while Qwen2.5-Math-72B-Instruct and Qwen2.5-72B-Instruct independently solve them and retain only consistent answers.
Key Results
- MathCoder-VL-8B reaches 26.1% on MATH-Vision, 46.5% on MathVerse, and 73.6% on MathVista GPS, improving over InternVL2-8B by 6.1, 10.6, and 11.6 percentage points. It also obtains 51.2% on Chinese GAOKAO-MM Math, leading comparable open-source systems.
- On MathVista geometry, it surpasses GPT-4o and Claude 3.5 Sonnet by 8.9 and 9.2 percentage points, respectively. On MATH-Vision plane geometry, its angle, area, and length accuracies are 48.6%, 32.2%, and 32.1%.
- The data engine retains 4.3M TikZ and 4.3M Python pairs. Rendering success improves from 46.5% for initial TikZ outputs to 81.2% for TikZ and 84.5% for Python on the DaTikZ test set; synthetic-answer filtering has a 51% pass rate.
Significance
The work addresses a central weakness of multimodal mathematical reasoning: natural-language captions omit exact points, measurements, topology, and spatial relations, while image–text pairs may be inconsistent. Executable code provides a complete and testable description of how a figure is generated. The resulting data engine makes mathematical visual supervision scalable and supports substantial gains in geometry, functions, and multistep reasoning. It offers an important open-source alternative to relying exclusively on proprietary multimodal models.
Technical Contribution
The main engineering contribution is a model-in-the-loop image-to-code pipeline: FigCodifier proposes code, an executor validates rendering, and successful examples are added for iterative improvement. This expands far beyond the 119K-pair DaTikZ seed set and manually designed figure types. GPT-4o mini converts TikZ to Python, diversifying the program distribution. During alignment training, the vision encoder and MLP projector are trainable while the LLM is frozen; full-model tuning is reserved for the subsequent mathematical instruction stage.
Novelty
The fundamental novelty is generating new mathematical figures together with new questions and verified solutions, rather than synthesizing only textual problems. Compared with MAVIS, whose human-written code covers only three figure types, FigCodifier learns from K12, textbook, arXiv, MathV360K, and Multi-Math images and automatically produces structurally varied executable programs. The paper presents this as the first high-quality multimodal math instruction engine combining novel text and diverse novel figures at this scale.
Limitations
- Successful rendering does not guarantee semantic correctness: a program may omit labels, distort proportions, or encode incorrect geometric relations. Requiring agreement between two Qwen solvers retains only 51% of generated samples and may still preserve shared errors.
- Construction is computationally expensive, requiring large-scale TikZ/Python execution and 32 or 64 NVIDIA A800 80GB GPUs for the 2B and 8B models. Source mixtures and heuristic filters may introduce domain bias.
- Evaluation emphasizes answer accuracy and does not fully measure proof validity, visual hallucination, calibration, robustness, or performance on complex non-Euclidean and 3D diagrams.
Future Work
Future systems should combine code execution with formal geometry solvers, symbolic algebra, and proof checking, distinguishing renderability from mathematical validity. Useful extensions include 3D, animated, handwritten, noisy, and multilingual figures; harder-example mining; stronger semantic deduplication; and evaluations of reasoning traces, calibration, latency, and energy use. A verifier-in-the-loop data engine could substantially reduce shared model errors.
AI Executive Summary
Large multimodal models have improved rapidly on natural images, yet they often fail on elementary geometry. The difficulty lies not only in solving equations but in faithfully reading points, lengths, angles, labels, and spatial relations. Conventional caption datasets, including large web-scale sources such as LAION-5B, favor natural scenes and frequently omit the details that determine a mathematical answer. They also provide no guarantee that a caption is exactly consistent with its image.
MathCoder-VL uses executable code as the bridge between vision and language. Its FigCodifier model converts mathematical figures into TikZ or Python programs; executing those programs produces ImageC, creating intrinsically aligned image–code pairs. The resulting ImgCode-8.6M dataset contains 4.3M TikZ and 4.3M Python pairs. With temperature 0.7, FigCodifier then generates new figures. Qwen2.5-72B-Instruct writes questions, while two Qwen models independently solve them, producing MM-MathInstruct-3M after consistency filtering. MathCoder-VL first performs visual–code alignment training and then full mathematical instruction tuning.
The 8B model scores 26.1% on MATH-Vision, 46.5% on MathVerse, 73.6% on MathVista GPS, and 51.2% on GAOKAO-MM Math, improving over InternVL2-8B by 6.1, 10.6, 11.6, and 18.7 points. It also exceeds GPT-4o and Claude 3.5 Sonnet on the reported MathVista geometry comparison. The work demonstrates that programmable representations can make visual supervision both precise and scalable, while leaving open the harder question of whether a rendered diagram is mathematically correct.
Deep Analysis
Background
LLMs now solve difficult mathematical tasks, but LMMs remain weaker because visual extraction precedes reasoning. Prior systems include InternVL2, Qwen2-VL, Math-LLaVA, Multimath, and MAVIS. MathV360K and related datasets mainly enrich questions or solutions; MAVIS uses code but relies on human-designed programs covering only three figure types. Precise, scalable supervision for mathematical visual structure remains missing.
Core Problem
The task is to align visual geometry with language representations while preserving exact labels, coordinates, measurements, and relations. Captions are lossy and potentially inconsistent; manual diagrams are expensive and narrow in variety. Moreover, multimodal problem synthesis has expanded text much faster than images, limiting visual diversity and encouraging memorization rather than robust geometric reasoning.
Innovation
- �� Replace incomplete captions with executable TikZ/Python supervision.
- �� Use FigCodifier and iterative model-in-the-loop validation to expand 3M collected images into ImgCode-8.6M.
- �� Convert TikZ to Python with GPT-4o mini to diversify code structures and rendered appearances.
- �� Generate new figures at temperature 0.7, then synthesize questions and solutions with Qwen models.
- �� Build MM-MathInstruct-3M, including 1M figure-driven synthetic samples, so both textual and visual problem distributions can scale.
Methodology
- �� Seed training: train an initial image-to-code model on 119K DaTikZ image–TikZ pairs.
- �� Collection: combine 1.57M K12 images, 202K textbook images, arXiv figures, MathV360K, and Multi-Math.
- �� Translation: FigCodifier produces TikZ; GPT-4o mini converts TikZ into Python.
- �� Execution: render each program and retain only successful ImageC–Code pairs.
- �� Cleaning: remove 4.4% duplicates, 3.7% low-quality or irrelevant samples, overly long code, and about 0.5% near-white images.
- �� Training: update the vision encoder and MLP projector while freezing the LLM during alignment; fine-tune the complete model during instruction training.
Experiments
The experiments use InternVL-Chat-2B-V1-5 and InternVL2-8B as base models, training one epoch per stage. Mid-training uses batch size 1024 and learning rate 2e-5; instruction tuning uses batch size 512 and 4e-5. DeepSpeed ZeRO-1 and FlashAttention support training on 32 A800 80GB GPUs for 2B and 64 for 8B. Evaluation covers MATH-Vision, MathVerse, MathVista GPS, GAOKAO-MM, and We-Math, with greedy decoding and fixed random seeds.
Results
MathCoder-VL-8B obtains 26.1%, 46.5%, 73.6%, and 51.2% on MATH-Vision, MathVerse, MathVista GPS, and GAOKAO-MM Math. It beats InternVL2-76B by 2.5, 3.7, 5.8, and 10.0 points on these benchmarks, respectively. On MATH-Vision plane geometry, its 37.6% average exceeds GPT-4o's 25.7%; angle, area, and length scores are 48.6%, 32.2%, and 32.1%.
Applications
The system can support geometry tutoring, diagram-aware question answering, textbook and benchmark generation, and automated educational assessment. A practical deployment needs a vision-language model, a sandboxed TikZ/Python renderer, and answer or proof verification. Human review remains important for high-stakes education because executable output is not identical to mathematically valid output.
Limitations & Outlook
The central assumption—that renderable code is useful supervision—does not ensure semantic correctness. Solver agreement is an imperfect filter and reduces synthetic yield to 51%. Rendering millions of programs and training on 8.6M pairs impose substantial compute and engineering costs. Dataset composition may bias performance toward K12-style diagrams and selected languages. Future work should add formal verification, 3D and noisy-figure data, process-level evaluation, and stronger robustness and calibration tests.
Plain Language Accessible to non-experts
Imagine a factory that teaches a worker to understand technical drawings. A normal caption is like a rushed verbal note: “There is a triangle and a circle.” It may forget a measurement, a letter, or which line touches which point. MathCoder-VL instead asks the worker to create a complete construction recipe. The recipe says where every point goes, which lines connect, how large the circle is, and what labels must appear.
FigCodifier is the factory’s drawing interpreter. It reads existing mathematics images and writes TikZ or Python instructions. A computer follows those instructions and redraws the figure. If the redraw succeeds, the instructions and the new image become a matched training example. This is stronger than a vague description because the recipe contains the details needed to build the object.
The factory then changes some recipes to make fresh figures. Another team writes questions about them, and two independent teams solve the questions. Only cases with matching answers are kept. After learning from millions of such examples, MathCoder-VL becomes better at connecting what it sees with what it must calculate. The process is powerful, but a picture can still be built from a wrong recipe, so a future factory needs a mathematical inspector, not just a working printer.
ELI14 Explained like you're 14
Think of a math picture as a game map. It has points, lines, circles, arrows, and numbers, and the question asks you to find something hidden in the map. If someone only says, “There is a triangle,” you cannot solve much—the important clues are missing!
MathCoder-VL gives the map a precise building recipe. The recipe is computer code: put point A here, connect it to B, draw a circle of this size, and write 30 degrees. The computer redraws the picture from the recipe, so the description and the picture match much better. FigCodifier is the helper that looks at a math image and writes this recipe.
Then the researchers ask the helper to create new maps. Qwen models invent questions and solutions, and examples are kept when two solvers agree. This creates a huge practice set with both new pictures and new problems—not just recycled text. After training, the 8B model scores 73.6% on MathVista’s geometry set, 11.6 points above InternVL2-8B.
But it is not unbeatable. A recipe can run successfully while still putting a point in the wrong place, just as a game map can load with a broken quest. Future versions need a math referee that checks proofs and relationships. That could make the system much more trustworthy for schoolwork!
Glossary
FigCodifier
An image-to-code model that converts mathematical figures into executable TikZ or Python. In plain terms, it writes a reproducible drawing recipe rather than a short caption.
It produces training pairs and synthesizes new mathematical figures.
ImgCode-8.6M
An approximately 8.6-million-pair image–code dataset. It contains about 4.3M TikZ pairs and 4.3M Python pairs whose programs render corresponding images.
It supplies the first visual–code mid-training stage.
MM-MathInstruct-3M
A roughly three-million-sample multimodal mathematical instruction dataset, including one million samples generated from new figures. Samples combine images, questions, and solutions.
It is used for full mathematical instruction fine-tuning.
Model-in-the-loop
A closed data-production loop in which a model generates candidate data, external execution or filters validate it, and accepted examples improve later model versions.
The paper uses this loop to iteratively scale image-to-code training.
Cross-modal alignment
The process of making visual and language or code representations refer to the same information. Better alignment lets a model use exact visual structure during reasoning.
It is the objective of the first training stage.
MathVista GPS
The Geometry Problem Solving subset of MathVista, designed to test reasoning from visual mathematical contexts. MathCoder-VL-8B achieves 73.6% on this subset.
It is the paper’s strongest headline benchmark result.
Open Questions Unanswered questions from this research
- 1 How can a system automatically prove that generated code preserves every geometric relation, scale, and label? Rendering success establishes executability, not mathematical validity.
- 2 Performance on low-resolution, handwritten, occluded, 3D, animated, and heavily noisy diagrams is not established and requires targeted robustness benchmarks.
- 3 Two models agreeing can still share an error. Reliable data construction may require symbolic solvers, theorem provers, or formally verified execution.
Applications
Immediate Applications
Geometry tutoring
Educational platforms could input a diagram and receive relation-aware explanations or solution steps. Deployment requires multimodal inference, a sandboxed renderer, and human or symbolic checking; the model should initially assist teachers rather than replace them.
Question-bank generation
Publishers and assessment providers could transform existing figures into varied TikZ/Python diagrams, then generate new questions with Qwen models. Copyright review, difficulty control, duplicate detection, and answer verification are necessary before release.
Long-term Vision
Verifiable multimodal mathematics agents
A future agent could inspect an image, write executable code, call a geometry theorem prover, and explain a formally checked conclusion. Major obstacles include semantic verification for complex diagrams, inference cost, safety, and educational reliability.
Abstract
Natural language image-caption datasets, widely used for training Large Multimodal Models, mainly focus on natural scenarios and overlook the intricate details of mathematical figures that are critical for problem-solving, hindering the advancement of current LMMs in multimodal mathematical reasoning. To this end, we propose leveraging code as supervision for cross-modal alignment, since code inherently encodes all information needed to generate corresponding figures, establishing a precise connection between the two modalities. Specifically, we co-develop our image-to-code model and dataset with model-in-the-loop approach, resulting in an image-to-code model, FigCodifier and ImgCode-8.6M dataset, the largest image-code dataset to date. Furthermore, we utilize FigCodifier to synthesize novel mathematical figures and then construct MM-MathInstruct-3M, a high-quality multimodal math instruction fine-tuning dataset. Finally, we present MathCoder-VL, trained with ImgCode-8.6M for cross-modal alignment and subsequently fine-tuned on MM-MathInstruct-3M for multimodal math problem solving. Our model achieves a new open-source SOTA across all six metrics. Notably, it surpasses GPT-4o and Claude 3.5 Sonnet in the geometry problem-solving subset of MathVista, achieving improvements of 8.9% and 9.2%. The dataset and models will be released at https://github.com/mathllm/MathCoder.