GeoGPT4V: Towards Geometric Multi-modal Large Language Models with Geometric Image Generation

Recent advances in LLMs like GPT-4 have greatly improved natural language understanding and reasoning. Multimodal models integrating visual data, such as GPT-4V and Gemini, have shown impressive image comprehension. However, geometric reasoning remains a bottleneck due to limited high-quality, aligned datasets. Existing datasets like UniGEO and Geometry3K are either too challenging or lack proper text-image alignment, restricting model learning. Data augmentation efforts using ChatGPT-3.5 to modify numerical values often lead to mismatched text and images, impairing model training. Consequently, developing scalable, high-quality, aligned geometric datasets is critical for advancing multimodal geometric reasoning.

The core challenge is the scarcity of high-quality, aligned geometric question-image pairs suitable for training large multimodal models. Existing datasets are either too complex or poorly aligned, limiting models’ ability to learn geometric concepts effectively. Manual annotation is costly and inefficient, hindering large-scale data creation. This bottleneck prevents models from achieving human-level understanding and reasoning in geometry, especially in complex scenarios requiring visual perception. Addressing this gap requires an automated, reliable pipeline capable of generating diverse, well-aligned geometric data at scale.

The key innovations include:

• �� An automated pipeline combining GPT-4V for question simplification and Wolfram code for diagram generation, ensuring high-quality, diverse geometric images.
• �� A scoring mechanism using GPT-4V to evaluate and select images with the best text-image alignment.
• �� Integration of generated problems with open-source datasets to create a balanced, scalable dataset (GeoGPT4V).
• �� Demonstration of significant performance improvements on benchmarks, validating the effectiveness of data augmentation.

This approach differs from prior manual or semi-automated methods, providing a fully automated, scalable solution for high-quality geometric data creation.

• �� Collect open-source geometric datasets and use GPT-4V to generate simplified question-answer pairs, focusing on core geometric concepts.
• �� Generate Wolfram code from these pairs via GPT-4, explicitly naming variables for clarity.
• �� Execute multiple code variants to produce diverse images, then employ GPT-4V to score each image’s alignment with the question.
• �� Select the highest-scoring image, discard questions with low scores (<0.9), and merge these with original data.
• �� Train models like LLaVA-1.5 and ShareGPT4V on the combined dataset, evaluate performance improvements.
• �� Conduct ablation studies to assess the impact of image generation and filtering steps, ensuring robustness.

The experiments utilize MathVista and MathVision benchmarks, measuring accuracy on geometric tasks before and after training with GeoGPT4V. Models of various sizes (7B, 13B, 40B) are trained with and without the new data. Baselines include models trained solely on open-source datasets. Evaluation metrics focus on task-specific accuracy and image-text alignment scores. Ablation studies compare performance when replacing generated images with original images or omitting the scoring step. Results consistently show that data generated via the pipeline improves geometric reasoning capabilities, validating the approach’s effectiveness across different model architectures.

Models trained on GeoGPT4V datasets show an average accuracy increase of 58.2% on MathVista and 33.8% on MathVision benchmarks. For example, LLaVA-1.5-7B’s geometric task accuracy rose from 20.67% to 32.69%. The image-text matching score improved from 0.6754 (G-LLaVA) to 0.9636, indicating better alignment. Ablation experiments reveal that removing image generation or filtering reduces performance gains, confirming the pipeline’s critical role. These results demonstrate that high-quality, aligned data significantly enhances multimodal geometric reasoning.

This methodology can be applied to AI-driven math education, automated problem generation, and intelligent tutoring systems. It enables scalable creation of diverse geometric datasets, facilitating model training for complex reasoning tasks. In industry, it supports automated test generation, intelligent assessment, and scientific research requiring visual reasoning. Long-term, the approach could enable AI systems to autonomously generate and solve advanced geometric problems, contributing to scientific discovery and personalized learning environments.

The generated problems mainly cover basic geometry, limiting applicability to complex scenarios. Image generation remains unstable, occasionally producing inaccurate diagrams. The pipeline’s effectiveness diminishes with highly intricate problems requiring advanced reasoning. Future work should focus on improving image stability, expanding problem complexity, and integrating reasoning modules to handle challenging geometric tasks.