Tri-Bench: Stress-Testing VLM Reliability on Spatial Reasoning under Camera Tilt and Object Interference
Tri-Bench tests VLM spatial reasoning under camera tilt and object interference, with ~69% average accuracy.
Key Findings
Methodology
Tri-Bench is a benchmark of 400 images focusing on planar triangle problems, assessing VLM spatial reasoning under camera pose and scene interference. A single fixed prompt explicitly describes the boundary, allowing correct answers via homography.
Key Results
- VLMs achieve ~69% average accuracy with respect to 3D ground truth and 72% on 2D projections. They fail to recognize equilateral, isosceles, and right-angled triangles, with accuracy dropping to ~0%.
- Overall accuracy degrades by ~4.1% under camera tilt, indicating models fail to utilize the explicit frame-of-reference hint.
- Object interference has no significant effect on VLM accuracy.
Significance
This study reveals robustness issues in VLMs under real-world scene changes, particularly the impact of camera tilt and object interference on spatial reasoning. Tri-Bench provides a new diagnostic tool for evaluating VLM reliability in geometric reasoning.
Technical Contribution
Tri-Bench offers a controlled benchmark revealing key failure modes in VLM spatial reasoning, especially the misinterpretation between 3D real-world and 2D image plane. It highlights limitations in VLM geometric reasoning.
Novelty
Tri-Bench is the first to systematically control and isolate the effects of camera pose and object interference on relative spatial reasoning, filling a gap in existing benchmarks.
Limitations
- Models perform poorly on minority shape classes, with accuracy near 0%.
- Camera tilt leads to accuracy degradation, indicating a lack of pose invariance.
Future Work
Future research could explore multi-view geometric reasoning, improved prompting strategies, and extensions to more complex shapes and scenes.
AI Executive Summary
Tri-Bench benchmark reveals limitations in Vision-Language Models (VLMs) for spatial reasoning under camera tilt and object interference. Despite impressive capabilities in general visual reasoning, VLM robustness under realistic scene changes remains a critical unverified barrier.
Tri-Bench, constructed with 400 images focusing on planar triangle problems, assesses VLM spatial reasoning under camera pose and scene interference. Results show VLMs achieve ~69% accuracy with respect to 3D ground truth and 72% on 2D projections. They fail to recognize equilateral, isosceles, and right-angled triangles, with accuracy dropping to ~0%.
This study provides new insights into VLM reliability in geometric reasoning, highlighting challenges in deploying VLMs in real-world scenarios. Future research could explore multi-view geometric reasoning, improved prompting strategies, and extensions to more complex shapes and scenes.
Deep Analysis
Background
Vision-Language Models (VLMs) are increasingly applied in fields like robotics navigation, augmented reality, and 3D reconstruction. However, robustness in geometric reasoning remains a challenge. Existing benchmarks often focus on absolute distance, angle estimation, or abstract diagram problem-solving, lacking stress tests for critical factors like camera pose and object interference.
Core Problem
VLMs lack sufficient geometric reasoning capabilities under realistic scene changes, particularly with camera tilt and object interference. Improving VLM accuracy and robustness under these conditions is a pressing challenge.
Innovation
Tri-Bench systematically evaluates VLM performance in relative spatial reasoning by controlling camera pose and object interference. Constructed with 400 images focusing on planar triangle problems, it offers a new diagnostic tool.
Methodology
- �� Constructed with 400 images focusing on planar triangle problems.
- �� Controls camera pose (planar vs. tilted) and scene interference (10 everyday objects).
- �� Uses a single fixed prompt explicitly describing the boundary, allowing correct answers via homography.
- �� Evaluates six simple tasks covering binary and continuous targets.
Experiments
Experiments evaluated four recent VLMs: Gemini 2.5 Pro, Gemini 2.5 Flash, GPT-5, and Qwen2.5-VL-32B. A single fixed prompt was used for evaluation, with accuracy reported separately for four capture conditions.
Results
VLMs achieve ~69% accuracy with respect to 3D ground truth and 72% on 2D projections. They fail to recognize minority shape classes, with accuracy dropping to ~0%. Camera tilt leads to ~4.1% accuracy degradation.
Applications
Tri-Bench can be used to evaluate VLM spatial reasoning capabilities in fields like robotics navigation and augmented reality, aiding in improving model robustness.
Limitations & Outlook
Models perform poorly on minority shape classes, with accuracy near 0%. Camera tilt leads to accuracy degradation, indicating a lack of pose invariance. Future research could explore multi-view geometric reasoning and improved prompting strategies.
Plain Language Accessible to non-experts
Imagine you're in a room with a square tape on the floor, containing various triangles. Your task is to identify their shapes and angles just by looking. The camera might be tilted, and other objects in the room might block your view. Tri-Bench is like a test to see how accurately you can identify these triangle features under such conditions.
ELI14 Explained like you're 14
Imagine you're playing a game with a square frame on the floor filled with different triangles. Your job is to figure out their shapes and angles just by looking. The camera might be tilted, and other objects might block your view. Tri-Bench is a test to see how well you can identify these triangle features in such situations.
Glossary
Vision-Language Model
An AI model that combines visual and language information for reasoning.
Used to evaluate spatial reasoning under camera tilt and object interference.
Homography
A mathematical transformation mapping points from one plane to another.
Used to recover triangle geometry from images.
Camera Tilt
The angle at which a camera is tilted relative to the horizontal plane, affecting the captured image perspective.
Evaluates VLM performance under different camera poses.
Object Interference
The impact of other objects in a scene on the observation of target objects.
Tests VLM robustness under scene interference.
Benchmark
A standardized test set used to evaluate model performance.
Tri-Bench is used to assess VLM spatial reasoning capabilities.
Open Questions Unanswered questions from this research
- 1 How to improve VLM accuracy in recognizing minority shape classes? Current methods perform poorly, requiring new strategies.
- 2 What is the mechanism by which camera tilt affects model accuracy? Further research is needed to understand this.
Applications
Immediate Applications
Robotics Navigation
Helps robots perform accurate spatial reasoning in complex environments, enhancing navigation capabilities.
Long-term Vision
Augmented Reality
Improves object recognition and spatial awareness accuracy in AR applications, enhancing user experience.
Abstract
Verifiable geometric reasoning is a critical component for trustworthy and controllable agentic AI. Despite impressive capabilities, Vision-Language Models (VLMs) often fail under realistic scene changes. We present Tri-Bench, a compact benchmark of planar triangle problems that isolates relative geometric reasoning while stressing two deployment-critical factors: camera pose (planar vs. tilted) and scene context via object interference (10 everyday objects). To test verifiability and control, we evaluate four recent VLMs using a single, fixed prompt whose guardrail explicitly describes a surrounding square border, enabling correct answers via homography. We evaluate six simple tasks over binary and continuous targets, and observe that the overall accuracy with respect to 3D ground truth is modest, ~69% on average (best ~75%, worst ~64%). The same responses align even more closely with 2D projections in the image plane, where mean accuracy is ~72%. All four VLMs consistently fail, with accuracy falling to ~0%, on recognizing minority shape classes (equilateral, isosceles, right-angled triangles). Additionally, overall VLM accuracy degrades by ~4.1% under camera tilt. This demonstrates that models fail to correctly utilize the explicit frame-of-reference hint provided in the prompt and default to 2D image plane cues. Finally, we find that object interference has no significant effect on VLM accuracy.