Towards Terrain-Aware Task-Driven 3D Scene Graph Generation in Outdoor Environments

Recent developments in robotic perception have focused on indoor scene graph generation, leveraging RGB-D and LiDAR sensors. Techniques like Kimera and Hydra have demonstrated effective indoor mapping and semantic understanding. However, outdoor environments pose unique challenges: terrain variability, dynamic elements, and unstructured regions. Existing methods often rely on application-specific assumptions or limited semantic models, restricting their generalization. Deep learning models like CLIP and SAM have advanced semantic segmentation and open-set object recognition, but their integration into outdoor scene graphs remains nascent. Addressing these gaps requires combining sensor fusion, semantic embedding, and spatial modeling to create scalable, robust outdoor scene understanding frameworks.

The core challenge is to develop a comprehensive outdoor scene graph that captures geometric, semantic, and terrain information in unstructured, dynamic environments. Existing indoor methods lack the ability to model terrain variability and large-scale outdoor features. The difficulty lies in achieving real-time performance, semantic robustness, and multi-task generalization. Without a unified framework, autonomous systems struggle with environment interpretation, navigation, and planning in outdoor scenarios, limiting their deployment in applications like autonomous driving, off-road exploration, and environmental monitoring.

This work introduces several key innovations: 1) a task-agnostic, multi-modal metric-semantic mapping pipeline that fuses LiDAR and RGB data using deep learning; 2) a Voronoi diagram-based terrain modeling approach that captures terrain topology and spatial relationships; 3) hierarchical multi-layer scene graph architecture supporting object, terrain, and region nodes; 4) integration of CLIP embeddings for open-set object recognition in outdoor scenes. These innovations enable scalable, real-time, terrain-aware environment understanding, surpassing prior indoor-focused scene graph methods and addressing outdoor-specific challenges.

• �� Input: synchronized LiDAR point clouds, RGB images processed through YOLOv11 for terrain detection, FastSAM for segmentation, and CLIP for semantic embedding. • SLAM: employ LIO-SAM to generate a global point cloud. • Semantic features: extract terrain types and object masks, encode with CLIP, and associate with LiDAR points via projection and KD-tree matching. • GVD construction: generate binary terrain masks, apply dilation/erosion, compute distance maps, and generate Voronoi diagrams identifying junctions and corners as place nodes. • Scene graph: create object nodes from detection, connect to nearest terrain nodes, cluster regions based on task-specific criteria, and assemble multi-layer hierarchical graph supporting multi-task reuse.

The framework was tested in a BYU campus outdoor environment with OAK-D camera and Ouster LiDAR. Quantitative metrics included terrain classification accuracy, object detection recall, and scene graph construction time. Ablation studies examined the impact of cosine similarity thresholds and clustering parameters. Comparisons with baseline methods demonstrated improvements in semantic accuracy and processing speed. The system maintained robustness under varying lighting and dynamic elements, validating its practical viability for outdoor autonomous applications.

Results showed 85% accuracy in terrain classification, 80%+ object detection recall, and scene graph generation within 2 seconds. The semantic embeddings achieved an average cosine similarity of 0.95, indicating high semantic consistency. The multi-layer graph effectively supported multi-task scenarios, with significant improvements over traditional geometric-only maps. These findings confirm the method’s robustness, scalability, and potential for real-world deployment.

This technology can be directly applied to autonomous vehicles, off-road robots, and environmental monitoring systems, providing detailed, semantic-rich scene understanding. Its ability to handle unstructured terrains and dynamic elements makes it suitable for complex outdoor tasks. The framework supports multi-task operations like navigation, obstacle avoidance, and environment mapping, facilitating safer and more efficient autonomous systems in diverse outdoor scenarios. Long-term, it can enable fully autonomous exploration and environmental assessment in uncharted terrains.

Current limitations include sensitivity to extreme weather, dynamic scene complexity, and computational costs for large-scale environments. The terrain classification relies on pre-trained models, which may misclassify unseen terrain types. Real-time performance needs further optimization, especially for high-density point clouds. The framework’s robustness under adverse conditions and in highly dynamic scenes remains to be improved. Future work should focus on adaptive algorithms, sensor fusion enhancements, and dataset expansion for better generalization.