PRiSM: An Agentic Multimodal Benchmark for Scientific Reasoning via Python-Grounded Evaluation
PRiSM evaluates scientific reasoning via Python code with 24,750 physics and math problems.
Key Findings
Methodology
PRiSM uses PrismAgent to generate multimodal scientific reasoning problems, combining text, images, and Python code. Each problem includes dynamic inputs and structured outputs, supporting fine-grained model evaluation.
Key Results
- PRiSM reveals limitations of existing VLMs in scientific reasoning, especially in robustness to input variations and visual perturbations.
- Through five evaluation tasks, PRiSM demonstrates model performance differences in symbolic program synthesis and reasoning correction.
- Experiments show some models perform poorly under uncertainty and ambiguity, exposing reasoning deficiencies.
Significance
PRiSM provides a dynamic, multimodal benchmark for evaluating scientific reasoning, addressing gaps in intermediate reasoning steps and computational verification, with significant academic and industrial implications.
Technical Contribution
PRiSM automates ground truth generation via Python code, enabling fine-grained experimental auditing and offering new engineering possibilities and theoretical guarantees.
Novelty
PRiSM is the first to combine dynamically generated multimodal inputs with Python code verification, offering a more comprehensive evaluation of scientific reasoning than existing benchmarks.
Limitations
- PRiSM may have limitations in handling real-world complexity and diversity, as its problem set is synthetic.
- Some models perform poorly on highly complex symbolic reasoning tasks.
Future Work
Future work could expand PRiSM's domain coverage and explore ways to enhance model robustness and accuracy in complex scientific problems.
AI Executive Summary
PRiSM is a multimodal benchmark for scientific reasoning, evaluated via Python code. Existing vision-language models face challenges in scientific domains, particularly in mathematics and physics. PRiSM provides a comprehensive evaluation framework through dynamically generated physics and math problems, combining text, images, and Python code. Experimental results reveal limitations of current models in handling input variations and visual perturbations, demonstrating PRiSM's effectiveness in uncovering model reasoning capabilities. Future research could further expand PRiSM's application scope, enhancing model performance in scientific reasoning.
Deep Analysis
Background
Scientific reasoning is crucial in fields like mathematics and physics, yet existing benchmarks often lack intermediate reasoning steps and computational verification. PRiSM fills this gap with dynamically generated problems and Python code verification.
Core Problem
Existing benchmarks often lack dynamism and multimodal inputs, failing to comprehensively assess models' reasoning capabilities.
Innovation
PRiSM combines dynamically generated multimodal inputs with Python code verification, providing a comprehensive framework for scientific reasoning evaluation.
Methodology
- �� Use PrismAgent to generate problems
- �� Dynamic inputs and structured outputs
- �� Python code verification
- �� Five evaluation tasks
Experiments
PRiSM includes 24,750 university-level physics and math problems, using five evaluation tasks to test model reasoning capabilities.
Results
Experiments reveal limitations of current models in handling input variations and visual perturbations, demonstrating PRiSM's effectiveness in evaluating model reasoning capabilities.
Applications
PRiSM can be used to assess the robustness and accuracy of scientific reasoning models, especially in handling complex scientific problems.
Limitations & Outlook
PRiSM may have limitations in handling real-world complexity and diversity, future work could expand its domain coverage.
Plain Language Accessible to non-experts
Imagine a complex math problem as a puzzle. PRiSM acts like a smart assistant, helping you solve the puzzle step by step. It not only gives the final answer but also shows every step of the solution, ensuring each step adheres to scientific principles.
ELI14 Explained like you're 14
Imagine you're playing a game that requires solving puzzles. PRiSM is like a super helper, guiding you through each puzzle step by step. It not only tells you the answer but also shows each step of the solution, just like hints in a game. Isn't that cool?
Glossary
PrismAgent
An automated pipeline for generating multimodal scientific problems.
Used to generate problem instances in the PRiSM dataset.
VLMs (Vision-Language Models)
Models that combine visual and language inputs for reasoning.
Evaluated for scientific reasoning.
Python Code
Executable code used to verify scientific problem solutions.
Used for solution verification in PRiSM.
Multimodal
Combining multiple input forms, such as text and images.
PRiSM problems include text and image inputs.
Benchmark
A standard test set for evaluating model performance.
PRiSM serves as a benchmark for scientific reasoning evaluation.
Open Questions Unanswered questions from this research
- 1 How to apply PRiSM-generated problems in real-world scenarios?
- 2 How to improve model robustness in handling complex scientific problems?
Applications
Immediate Applications
Science Education
PRiSM can be used in science education to help students understand complex scientific concepts.
Long-term Vision
Automated Scientific Research
PRiSM could drive advancements in automated scientific research, enhancing research efficiency.
Abstract
Evaluating vision-language models (VLMs) in scientific domains like mathematics and physics poses unique challenges that go far beyond predicting final answers. These domains demand conceptual understanding, symbolic reasoning, and adherence to formal laws, requirements that most existing benchmarks fail to address. In particular, current datasets tend to be static, lacking intermediate reasoning steps, robustness to variations, or mechanisms for verifying scientific correctness. To address these limitations, we introduce PRiSM, a synthetic, fully dynamic, and multimodal benchmark for evaluating scientific reasoning via grounded Python code. PRiSM includes over 24,750 university-level physics and math problems, and it leverages our scalable agent-based pipeline, PrismAgent, to generate well-structured problem instances. Each problem contains dynamic textual and visual input, a generated figure, alongside rich structured outputs: executable Python code for ground truth generation and verification, and detailed step-by-step reasoning. The dynamic nature and Python-powered automated ground truth generation of our benchmark allow for fine-grained experimental auditing of multimodal VLMs, revealing failure modes, uncertainty behaviors, and limitations in scientific reasoning. To this end, we propose five targeted evaluation tasks covering generalization, symbolic program synthesis, perturbation robustness, reasoning correction, and ambiguity resolution. Through comprehensive evaluation of existing VLMs, we highlight their limitations and showcase how PRiSM enables deeper insights into their scientific reasoning capabilities.