Causal Scaffolding for Physical Reasoning: A Benchmark for Causally-Informed Physical World Understanding in VLMs

Recent advances in physical reasoning benchmarks like CLEVRER, CoPhy, PhysBench, and MVPBench have pushed the boundaries of perception and basic causal understanding in AI. These datasets focus on synthetic primitives or simplified scenes, emphasizing geometric and collision-based reasoning. However, they lack the complexity and variability of real-world physics, limiting their applicability. Multimodal datasets have incorporated linguistic knowledge but often neglect explicit causal structures. Consequently, models trained on these benchmarks tend to rely on superficial correlations rather than genuine causal mechanisms, restricting their ability to generalize to real environments. The gap between perception and causal understanding remains a significant challenge in AI research, especially for applications requiring dynamic, intervention-aware reasoning.

Despite progress, current VLMs struggle with causal inference in complex scenes, especially in understanding how objects and attributes causally influence each other over time. They often fail to recognize causal dependencies, leading to brittle reasoning in tasks like viewpoint shifts, object interactions, and dynamic scene evolution. The core issue is the reliance on statistical associations rather than mechanistic causal models, which hampers generalization and interpretability. Existing benchmarks lack explicit causal annotations, making it difficult to diagnose and improve models’ causal reasoning capabilities. Addressing this requires a dataset with rich causal annotations, mechanism-level evaluation metrics, and training strategies that incorporate causal structure supervision.

This paper introduces CausalPhys, a large-scale benchmark integrating expert-annotated causal graphs with real-world scene data. It formalizes causal relationships via typed DAGs capturing objects, attributes, and events, enabling mechanism-level evaluation. The authors develop new metrics—Entity Faithfulness, Description Correctness, and Relation Awareness—that quantify how well models’ rationales align with causal structures, assessed through a judge LLM. The core innovation is the CRFT method, which explicitly guides models to produce reasoning aligned with causal graphs, significantly improving causal relation recognition and overall accuracy. This approach bridges the gap between pattern recognition and causal understanding, setting a new standard for multimodal physical reasoning.

• �� Data collection: Aggregate real-world images and videos from multiple datasets, with expert-designed questions and causal graph annotations. • Causal graph formalism: Nodes represent objects, attributes, or events; edges denote causal dependencies, forming a typed DAG. • Evaluation metrics: Use Entity Faithfulness to verify entity mention, Description Correctness for reasoning fidelity, and Relation Awareness for causal dependency capture, all assessed via a judge LLM. • Fine-tuning: Implement CRFT, which incorporates causal graphs as supervision signals, aligning model reasoning paths with causal structures. • Model training: Optimize models on the annotated data, enforcing causal path consistency, and evaluate on held-out data to measure generalization.

Experiments involve testing multiple VLMs, including GPT-4o and Qwen-3, on the CausalPhys dataset. The models are evaluated on four task categories—Perception, Anticipation, Intervention, and Goal-Orientation—using the proposed metrics. Baseline models are compared with versions fine-tuned via CRFT. Ablation studies analyze the impact of causal supervision, model size, and task complexity. The experiments also examine the correlation between causal metrics and overall answer accuracy, validating the importance of causal alignment. Results demonstrate that CRFT consistently improves causal relation metrics by over 30%, with a corresponding increase in overall accuracy, confirming the effectiveness of causality-guided training.

Post-CRFT, models achieve over 25% gains in causal relation metrics, with answer accuracy rising from approximately 40% to 65%. Entity Faithfulness reaches 0.75, and Relation Awareness exceeds 0.35. These improvements indicate a stronger understanding of causal dependencies, especially in dynamic scenes. The analysis reveals that models trained with causal supervision are less reliant on superficial cues and better at reasoning through complex causal chains. The results validate the hypothesis that explicit causal structure integration enhances both interpretability and robustness, paving the way for more reliable physical reasoning in real-world applications.

The benchmark and methods can be directly applied to autonomous systems, robotics, and intelligent agents that require causal understanding of physical interactions. By explicitly modeling causal dependencies, these systems can perform intervention-aware reasoning, improve safety, and enhance explainability. The approach also benefits scientific visualization, virtual simulation, and education tools that need mechanistic explanations of physical phenomena. Long-term, integrating causal reasoning into multimodal models could revolutionize AI’s ability to learn from limited data, adapt to unseen scenarios, and develop human-like intuitive understanding of the physical world.

Despite promising results, models still face challenges in multi-step causal reasoning, especially in highly complex or ambiguous scenes. The expert annotation process is resource-intensive, limiting dataset scalability. The current approach relies on static causal graphs, which may not fully capture dynamic or probabilistic causal relationships. Additionally, computational costs increase with model size and graph complexity, necessitating more efficient algorithms and scalable annotation methods. Future work should focus on automating causal graph generation, improving generalization to unseen scenarios, and reducing annotation overhead.