EmbodimentSemantic: A Spatial Scene-Graph Dataset and Benchmark for Vision-Language Models on Embodied Manipulation Trajectories
EmbodimentSemantic introduces a spatial scene-graph dataset to enhance spatial understanding in robotic vision-language models.
Key Findings
Methodology
The study introduces EmbodimentSemantic, a spatial scene-graph dataset for evaluating relational grounding in robotic manipulation. It represents scenes as directed object-relation-object triplets, specifying spatial relations between objects using a fixed set of relations. The dataset includes real-world manipulation observations collected with the low-cost SO101 robot arm and generated scene graphs for studying spatial grounding.
Key Results
- Experiments show that current models predict plausible relations but struggle with depth-aware and viewpoint-dependent spatial structures.
- In the LIBERO benchmark, over 60K manipulation frames and 120K camera-specific scene graphs are used for evaluation.
- Scene graphs injected into existing VLA policy prompts are tested for downstream control improvements.
Significance
This research provides a unified framework for evaluating spatial grounding in vision-language models for robotic manipulation. By introducing EmbodimentSemantic, researchers can better diagnose deficiencies in spatial perception and test its utility in VLA manipulation.
Technical Contribution
The technical contribution lies in providing a new spatial scene-graph dataset and benchmark for evaluating spatial understanding in vision-language models for robotic manipulation. Using both real-world and simulated data, the study demonstrates how to generate and evaluate scene graphs and test their effectiveness in downstream tasks.
Novelty
EmbodimentSemantic is the first dataset focused on spatial scene-graphs for robotic manipulation trajectories. Unlike previous multimodal benchmarks, it directly evaluates models' ability to generate ordered scene-graph triplets.
Limitations
- Current models struggle with depth-aware and viewpoint-dependent spatial structures.
- The dataset may be limited by specific robot and scene settings.
Future Work
Future research directions include expanding the dataset to cover more robots and scenes and developing more robust models to handle complex spatial relations.
AI Executive Summary
Spatial grounding is a critical challenge for vision-language models in robotic manipulation tasks. While current models can recognize objects and follow language instructions, they lack explicit representation of object arrangements in space. EmbodimentSemantic addresses this issue by introducing a new spatial scene-graph dataset and benchmark.
The dataset uses directed object-relation-object triplets to represent scenes and includes real-world manipulation observations collected with the low-cost SO101 robot arm. A simulator-grounded LIBERO benchmark with over 60K manipulation frames and 120K camera-specific scene graphs is also introduced for controlled validation.
Experiments show that current models predict plausible relations but struggle with depth-aware and viewpoint-dependent spatial structures. EmbodimentSemantic provides a unified framework for diagnosing deficiencies in spatial perception and testing its utility in VLA manipulation.
Deep Analysis
Background
Recent advancements in robotics have been driven by developments in vision-language models. However, these models still struggle with explicit representation of scene geometry, leading to challenges in manipulation tasks.
Core Problem
The core problem for vision-language models in robotic manipulation is the lack of explicit representation of object arrangements in space, leading to difficulties in handling support, containment, ordering, occlusion, and depth-sensitive relations.
Innovation
EmbodimentSemantic addresses this issue by introducing a new spatial scene-graph dataset. It uses directed object-relation-object triplets to represent scenes and includes a simulator-grounded LIBERO benchmark for controlled validation.
Methodology
- �� Collect real-world manipulation observations using the SO101 robot arm.
- �� Generate scene graphs for studying spatial grounding.
- �� Introduce a simulator-grounded LIBERO benchmark with over 60K manipulation frames and 120K camera-specific scene graphs.
- �� Test the improvement of scene graphs in downstream control by injecting them into existing VLA policy prompts.
Experiments
The experimental design includes using both real-world and simulated data to generate and evaluate scene graphs. Scene graphs are injected into existing VLA policy prompts to test their improvement in downstream control.
Results
Experiments show that current models predict plausible relations but struggle with depth-aware and viewpoint-dependent spatial structures. Scene graphs injected into existing VLA policy prompts are tested for downstream control improvements.
Applications
The study's application scenarios include spatial grounding and relational prediction in robotic manipulation tasks. By using EmbodimentSemantic, researchers can better diagnose deficiencies in spatial perception.
Limitations & Outlook
Current models struggle with depth-aware and viewpoint-dependent spatial structures. The dataset may be limited by specific robot and scene settings. Future research directions include expanding the dataset to cover more robots and scenes.
Plain Language Accessible to non-experts
Imagine you're in a kitchen trying to follow a recipe. You need to know where each ingredient is, like the salt is in the cupboard, and the pan is on the stove. EmbodimentSemantic is like a guide helping robots understand these location relationships. It tells the robot what the relationship is between the salt and the pan, like the salt is to the left of the pan. This way, the robot can better complete the task as instructed.
ELI14 Explained like you're 14
Imagine you're playing a game where a robot needs to find a hidden item in a room. EmbodimentSemantic is like a map in the game, helping the robot know the relationships between items, like which item is on top of or next to another. This way, the robot can find the target item faster!
Glossary
Vision-Language Model
A model that combines visual and language information to perform tasks.
Used for spatial understanding in robotic manipulation.
Scene Graph
A structured graph representing objects and their relationships.
Used to evaluate the accuracy of spatial relations.
Manipulation Trajectory
The path taken by a robot while performing a task.
Basis for collecting the dataset.
SO101 Robot Arm
A low-cost robotic arm used for collecting manipulation data.
Used for real-world data collection.
LIBERO Benchmark
A simulator-based benchmark for validating scene graph generation.
Provides controlled validation for benchmark testing.
Open Questions Unanswered questions from this research
- 1 How can EmbodimentSemantic be expanded to more complex scenes?
- 2 How can current models improve depth-aware capabilities?
Applications
Immediate Applications
Robotic Manipulation
By using EmbodimentSemantic, robots can better understand and execute complex manipulation tasks.
Long-term Vision
Smart Homes
In the future, EmbodimentSemantic could help develop smarter home robots, enhancing their navigation and manipulation capabilities in complex environments.
Abstract
Spatial grounding remains a key limitation of vision-language-action (VLA) systems for robotic manipulation. While current models can recognize objects and follow language instructions, they often lack an explicit representation of how objects are arranged in space, including support, containment, ordering, occlusion, and depth-sensitive relations. We introduce EmbodimentSemantic, a spatial scene-graph dataset and benchmark for evaluating relational grounding in embodied manipulation. EmbodimentSemantic represents scenes as directed object-relation-object triplets, where each triplet specifies a spatial relation between an ordered pair of objects using a fixed set of relations. This representation enables direct evaluation of object binding, relation prediction, and spatial consistency. The dataset includes real-world manipulation observations collected with the low-cost SO101 robot arm, together with generated scene graphs for studying spatial grounding in practical robotic settings. To provide controlled validation, we also introduce a simulator-grounded LIBERO benchmark with over 60K manipulation frames and more than 120K camera-specific scene graphs across paired third-person and wrist views, where ground-truth relations are derived automatically from MuJoCo geometry, world coordinates, camera projections, and visibility constraints. We further test whether scene graphs improve downstream control by injecting them into existing VLA policy prompts. Experiments across open-source and commercial VLMs show that current models often predict plausible relations but struggle with exact depth-aware and viewpoint-dependent spatial structure. EmbodimentSemantic provides a unified framework for diagnosing spatial grounding in VLM perception and testing its utility for VLA manipulation.