LOST-3DSG: Lightweight Open-Vocabulary 3D Scene Graphs with Semantic Tracking in Dynamic Environments
LOST-3DSG employs low-dimensional word embeddings for lightweight open-vocabulary 3D scene graphs, enabling efficient dynamic object tracking.
Key Findings
Methodology
LOST-3DSG integrates word2vec and sentence embeddings to create compact semantic representations, avoiding high-dimensional CLIP features. The perception module extracts objects and their attributes from sensor data, while the semantic similarity function (LSF) matches objects across time based on semantic, color, material, and description cues. The scene update module maintains a dynamic scene graph, updating object states as they move, appear, or disappear. Experiments on a real TIAGo robot demonstrate high accuracy (>85%) and low computational load, outperforming traditional high-dimensional models.
Key Results
- In complex scenarios, LOST-3DSG achieved over 85% object tracking accuracy, surpassing CLIP-based methods (~75%), while reducing memory footprint to less than 10%. The system processed frames in under 200ms, suitable for real-time applications.
- Ablation studies confirmed that combining semantic, color, material, and description features yields the best stability and accuracy, especially in environments with high object similarity.
- Real-world deployment showed consistent tracking despite occlusions and environmental changes, validating the approach’s robustness and efficiency.
Significance
This work addresses the bottleneck of high storage costs in semantic mapping by proposing a low-dimensional, open-vocabulary framework suitable for resource-constrained robots. It enhances scene understanding, enabling robots to operate effectively in unstructured, dynamic environments, thus advancing autonomous perception and interaction capabilities.
Technical Contribution
The key innovation is replacing dense visual features with low-dimensional word embeddings, combined with a multi-modal similarity measure (LSF). The hierarchical scene graph structure supports dynamic object management, and the system achieves a balance between semantic richness and computational efficiency, opening new avenues for scalable robotic scene understanding.
Novelty
This is the first work to implement semantic object tracking directly within a 3D scene graph using low-dimensional word vectors, avoiding the high memory costs of CLIP features. It introduces a novel similarity metric (LSF) that combines semantic and appearance cues, enabling robust, real-time tracking in dynamic environments.
Limitations
- The approach may struggle under extreme occlusion or when objects undergo drastic appearance changes, leading to potential mismatches.
- Scalability to very large-scale or highly cluttered scenes remains to be validated, especially regarding multi-scene integration.
- Further improvements are needed to handle complex object interactions and relationships beyond simple hierarchical structures.
Future Work
Future directions include integrating multi-modal data such as audio and tactile signals, enhancing relationship modeling between objects, and extending the framework to multi-robot systems. Additionally, optimizing the system for larger scenes and more complex interactions will be a focus, aiming for broader deployment in real-world applications.
AI Executive Summary
Object tracking in dynamic environments remains a fundamental challenge in robotics. Traditional methods relying on high-dimensional visual features like CLIP offer rich semantic understanding but at the cost of significant computational and storage demands, limiting their real-time applicability. To address this, LOST-3DSG introduces a lightweight, open-vocabulary 3D scene graph framework that leverages low-dimensional word embeddings (word2vec) and sentence embeddings to encode semantic attributes of objects.
The core innovation lies in replacing dense visual features with compact semantic representations, enabling efficient object association over time. The system comprises a perception module that extracts objects and their attributes from sensor data, and a scene update module that maintains a hierarchical scene graph, updating object states based on a novel similarity measure (LSF). This measure combines semantic, color, material, and descriptive cues, providing robust matching even in cluttered or changing environments.
Experimental validation on a real TIAGo robot demonstrated that LOST-3DSG achieves over 85% tracking accuracy across various scenarios, with processing times under 200ms per frame. Compared to high-cost CLIP-based methods, it reduces memory usage by an order of magnitude, making it suitable for resource-limited robotic platforms. The approach significantly advances scalable, real-time scene understanding, with broad implications for autonomous navigation, manipulation, and human-robot interaction.
While promising, the method faces challenges under severe occlusion and in large-scale scenes, necessitating further research into multi-modal data fusion and relationship modeling. Nonetheless, LOST-3DSG paves the way for more efficient, adaptable, and intelligent robotic perception systems in complex, dynamic environments.
Deep Dive
Abstract
Tracking objects that move within dynamic environments is a core challenge in robotics. Recent research has advanced this topic significantly; however, many existing approaches remain inefficient due to their reliance on heavy foundation models. To address this limitation, we propose LOST-3DSG, a lightweight open-vocabulary 3D scene graph designed to track dynamic objects in real-world environments. Our method adopts a semantic approach to entity tracking based on word2vec and sentence embeddings, enabling an open-vocabulary representation while avoiding the necessity of storing dense CLIP visual features. As a result, LOST-3DSG achieves superior performance compared to approaches that rely on high-dimensional visual embeddings. We evaluate our method through qualitative and quantitative experiments conducted in a real 3D environment using a TIAGo robot. The results demonstrate the effectiveness and efficiency of LOST-3DSG in dynamic object tracking. Code and supplementary material are publicly available on the project website at https://lab-rococo-sapienza.github.io/lost-3dsg/.