Active Semantic Perception

Robotics research has long focused on environment mapping, starting from classical geometric methods like occupancy grids and distance fields. Recent advances incorporate deep learning for object detection and structural recognition, exemplified by algorithms such as YOLOE and YOSO. Despite these improvements, most methods lack the ability to infer unobserved regions or reason about high-level semantic relationships, limiting exploration efficiency. Neural scene representations like NeRF provide photorealistic views but do not explicitly model object relations. Scene graphs offer a sparse, relational representation but are typically static or limited to known environments. This work aims to integrate large language models to enable probabilistic scene completion and reasoning, addressing the gap between geometric accuracy and semantic inference.

Existing exploration systems predominantly rely on geometric maps, which are insufficient for rapid environment understanding in complex indoor settings. They struggle with unobserved regions, leading to inefficient exploration and incomplete scene understanding. The core challenge is to develop a system that can infer high-level semantic structures—such as room types and object relationships—while maintaining geometric consistency. Achieving this requires combining probabilistic scene completion, semantic reasoning, and path optimization, all within computationally feasible limits. Overcoming these bottlenecks is crucial for deploying autonomous robots in real-world scenarios like household assistance or disaster response.

The paper introduces several key innovations: 1) a hierarchical multi-layer scene graph that encodes semantic, structural, and geometric data, enabling detailed environment modeling; 2) a scene completion pipeline leveraging LLMs to generate plausible unobserved regions, validated through collision checks; 3) an information gain-based viewpoint selection mechanism that supports complex spatial reasoning, such as inferring the purpose of doors; 4) an integrated path planning system that dynamically balances exploration speed and safety. These innovations collectively enable a leap beyond traditional geometric or single-layer semantic mapping, supporting high-level inference and efficient exploration.

• �� Scene mapping: utilizes YOLOE for object detection, YOSO for structural elements, constructing multi-layer scene graphs with nodes for objects, rooms, structures, and free space.
• �� Scene completion: converts partial scene graphs into YAML and cross-sectional images, inputs to LLMs for plausible unobserved region generation, validated via collision detection.
• �� Information gain: samples completed scene graphs, simulates future observations, computes mutual information for candidate viewpoints, and selects the most informative ones.
• �� Path planning: builds occupancy grids with nvBlox, applies A* algorithm, and dynamically replans as new observations arrive.
• �� Sampling and validation: multiple scene completions and confidence filtering improve prediction robustness and exploration efficiency.

The evaluation uses four HM3D environments of varying sizes in the Habitat simulator, comparing against frontier-based and semantic occupancy grid baselines. Metrics include F1 score for object detection, graph edit distance for topological accuracy, exploration time, and path length. Hyperparameters such as sampling iterations, information gain weights, and collision thresholds are tuned through ablation studies. Real-world tests involve a Unitree Go 2 robot exploring a six-room apartment, assessing object detection, scene prediction, and exploration speed. Results demonstrate significant improvements in semantic accuracy and exploration efficiency over baselines.

The proposed method achieved an average F1 score of 0.85 across simulation environments, outperforming baselines by 15%. The scene graph's topological error was reduced by 20%, and exploration time decreased by 25%. In real-world experiments, the system predicted unobserved rooms with over 80% success, enabling earlier discovery of targets. The multi-sample scene completion approach proved effective in handling uncertainty, and the information gain strategy led to more optimal viewpoints, balancing speed and safety. These results confirm the approach’s robustness and scalability.

This framework can be directly applied to household robots for faster object localization and environment mapping, especially in cluttered or unknown spaces. It also benefits search and rescue robots by enabling rapid inference of unobserved areas, improving mission efficiency. Long-term, integrating multi-modal data and on-device LLMs could facilitate deployment in dynamic, real-time scenarios, transforming autonomous indoor navigation and environment understanding in industry and service robotics.

Current scene completion accuracy drops in environments with severe occlusion or ambiguous structures, leading to potential errors. Dependence on cloud-based LLMs introduces latency and cost issues, limiting real-time deployment. Handling highly dynamic scenes remains challenging, as scene graphs may lag behind environmental changes. Further, the validation relies on collision checks that may not capture all structural nuances, especially in cluttered or complex environments. Future work must address these issues to improve robustness and scalability.