PRISM: A Multi-View Multi-Capability Retail Video Dataset for Embodied Vision-Language Models
PRISM leverages multi-view videos and a 3D knowledge ontology to enhance embodied VLM's spatial, physical, and action understanding in retail environments.
Key Findings
Methodology
PRISM is built on 270K samples of multi-view videos, integrating egocentric, exocentric, and 360° perspectives. It constructs a comprehensive 3D knowledge ontology covering spatial, temporal, and embodied action domains. The dataset employs metadata extraction, large language model (LLM) generation, depth analysis, and self-supervised transformations to annotate diverse tasks. Twenty+ capability probes span four key dimensions—Embodied Reasoning, Common Sense, Spatial Perception, and Physics—supporting chain-of-thought reasoning and multi-task fine-tuning. Fine-tuning utilizes LoRA for parameter efficiency, combining multi-modal supervision to improve physical and spatial reasoning in complex real-world scenes.
Key Results
- Post-fine-tuning, the model reduces error rates across all 20+ probes by an average of 66.6%, with embodied action understanding accuracy improving by 36.4%, demonstrating enhanced scene and physical reasoning capabilities.
- Multi-view data fusion outperforms single-view baselines in tasks like multi-object tracking, spatial layout comprehension, and multi-actor activity analysis, confirming the benefit of cross-view supervision.
- Ablation studies highlight that the structured knowledge ontology and multi-view supervision are critical for performance gains, with models trained without these components showing significant degradation.
Significance
This work addresses the critical gap in embodied AI for retail by integrating multi-dimensional spatial, physical, and action knowledge within a unified dataset. It advances the development of robots capable of understanding complex spatial arrangements, causal physics, and multi-agent interactions, crucial for autonomous navigation, task execution, and social interaction in cluttered, dynamic environments. The dataset and training framework set a new standard for domain-specific embodied VLMs, with broad implications for industry applications such as automated checkout, inventory management, and customer assistance, fostering more intelligent and adaptable retail robots.
Technical Contribution
The paper introduces a novel multi-view video dataset grounded in a three-dimensional knowledge ontology, enabling comprehensive training of embodied VLMs. It innovates by combining multi-view supervision, structured knowledge, and parameter-efficient fine-tuning (LoRA), resulting in models with superior spatial, physical, and action understanding. The integration of chain-of-thought reasoning and diverse supervision formats enhances interpretability and robustness, setting a new benchmark for domain-specific embodied AI.
Novelty
PRISM is the first dataset to unify multi-view retail videos with a structured 3D knowledge ontology, covering spatial, temporal, and embodied action domains simultaneously. Unlike prior datasets focused on general daily activities or isolated tasks, PRISM captures complete retail workflows with multi-view supervision, enabling models to learn long-horizon, goal-directed behaviors in real-world settings. Its combination of multi-view, physics-grounded, and chain-of-thought supervision represents a significant innovation in embodied AI training data.
Limitations
- High hardware costs for multi-view data collection limit scalability and transferability to other domains. The reliance on specialized sensors constrains widespread deployment.
- Model performance declines in highly occluded or cluttered scenarios, indicating the need for more robust perception modules.
- Training and inference require substantial computational resources, hindering real-time deployment in resource-constrained environments.
Future Work
Future efforts will focus on improving data efficiency, reducing hardware dependency, and enhancing model robustness in challenging scenarios. Expanding the dataset to include more diverse retail formats and integrating reinforcement learning for autonomous decision-making are promising directions. Additionally, developing lightweight models for real-time deployment will facilitate broader industry adoption.
AI Executive Summary
In the rapidly evolving field of embodied AI, enabling robots to operate effectively in complex retail environments remains a formidable challenge. Existing vision-language models excel at visual recognition but struggle with understanding the spatial layout, physical laws, and action sequences essential for autonomous task execution. Recognizing this gap, PRISM introduces a comprehensive multi-view video dataset grounded in a novel 3D knowledge ontology, designed specifically for retail scenarios.
PRISM comprises 270,000 samples of synchronized egocentric, exocentric, and 360° panoramic videos captured across five diverse supermarket locations. These videos are annotated through a hybrid pipeline involving metadata extraction, large language model (LLM) generation, depth analysis, and self-supervised transformations, covering over 20 capability probes across four key dimensions: Embodied Reasoning, Common Sense, Spatial Perception, and Physics. This rich annotation enables models to learn complex reasoning chains, from predicting the next subtask to understanding multi-actor interactions and causal physics.
The core innovation lies in integrating multi-view supervision with a structured 3D knowledge ontology, which explicitly encodes spatial geometry, temporal causality, and embodied actions. Fine-tuning models with LoRA on this dataset results in a 66.6% reduction in error rates across all probes, with embodied action accuracy improving by 36.4%. These results demonstrate the effectiveness of structured, domain-specific data in enhancing embodied VLM capabilities.
The broader impact of PRISM extends to advancing autonomous retail robots capable of navigation, inventory management, and customer interaction in cluttered, dynamic environments. It sets a new benchmark for domain-specific embodied AI datasets, emphasizing the importance of multi-view, physics-grounded, and reasoning-rich supervision. Future work will aim to improve data efficiency, extend to other retail formats, and develop lightweight models for real-time deployment, ultimately bringing smarter, more adaptable robots into everyday retail operations.
Deep Dive
Abstract
A critical gap exists between the general-purpose visual understanding of state-of-the-art physical AI models and the specialized perceptual demands of structured real-world deployment environments. We present PRISM, a 270K-sample multi-view video supervised fine-tuning (SFT) corpus for embodied vision-language-models (VLMs) in real-world retail environments. PRISM is motivated by a simple observation - physical AI systems fail not because of poor visual recognition, but because they do not understand space, physical dynamics and embodied action well enough to operate reliably in the world. To this end, PRISM is grounded in a novel three-dimensional knowledge ontology that spans spatial knowledge, temporal and physical knowledge, and embodied action knowledge. It covers 20+ capability probes across four evaluation dimensions - Embodied Reasoning (ER), Common Sense (CS), Spatial Perception (SP), and Intuitive Physics (IP), and to our knowledge, PRISM is the first dataset to instantiate all three knowledge dimensions within a single real-world deployment domain. The corpus captures data from egocentric, exocentric and 360° viewpoints across five supermarket locations and includes open-ended, chain-of-thought, and multiple-choice supervision. At 4 fps, PRISM spans approximately 11.8M video frames and approximately 730M tokens, placing it among the largest domain-specific video SFT corpora. Fine-tuning on PRISM reduces the error rate across all 20+ probes by 66.6% over the pre-trained baseline, with significant gains in embodied action understanding where the accuracy improves by 36.4%. Our results suggest that ontology-structured, domain specific SFT can meaningfully strengthen embodied VLMs for real-world settings. The PRISM dataset and more details are available at https://dreamvu.ai/prism