Humanoid-OmniOcc: Stereo-Based Full-View Occupancy Dataset for Embodied AI
Humanoid-OmniOcc: A stereo-based panoramic occupancy dataset enhancing robot navigation safety.
Key Findings
Methodology
The study introduces the Humanoid-OmniOcc dataset, constructed using NVIDIA Isaac Sim, employing four synchronized stereo cameras for full panoramic visual coverage. The dataset follows a Real2Sim2Real closed-loop paradigm, generating large-scale annotated data in simulation and evaluating model performance in real environments. The proposed HS2Occ model exploits stereo vision depth priors for precise 2D-to-3D conversion.
Key Results
- The HS2Occ model achieved 29.67% IoU and 11.69% mIoU on the test set, significantly outperforming monocular baselines.
- In real-world scenarios, the HS2Occ model reached 35.45% IoU and 19.26% mIoU, demonstrating strong generalization capabilities.
- Stereo depth priors significantly improved semantic recognition of interaction-relevant categories, such as doors and cabinets.
Significance
This study provides the first panoramic stereo vision occupancy dataset for humanoid robots, addressing the limitations of existing datasets in near-field perception. By employing the Real2Sim2Real closed-loop paradigm, it enhances sim-to-real transferability, advancing safe navigation and interaction in complex indoor environments.
Technical Contribution
The study introduces the Humanoid-OmniOcc dataset and HS2Occ model, leveraging stereo vision depth priors for precise 2D-to-3D conversion. Compared to existing monocular and LiDAR methods, this approach offers advantages in cost and deployment convenience.
Novelty
This is the first panoramic stereo vision occupancy dataset for humanoid robots, employing a Real2Sim2Real closed-loop paradigm to significantly enhance sim-to-real transferability.
Limitations
- Performance may degrade under mismatched lighting conditions and material reflectance.
- Limited perception capabilities for dynamic objects.
Future Work
Future work could explore more complex dynamic scenes and multimodal data fusion to further enhance model robustness and generalization.
AI Executive Summary
Humanoid robots require precise occupancy prediction for safe navigation and interaction in complex environments. However, existing datasets are primarily designed for autonomous driving, lacking support for near-field perception in humanoid robots. To address this, the research team introduced the Humanoid-OmniOcc dataset, constructed using NVIDIA Isaac Sim and employing four synchronized stereo cameras for full panoramic visual coverage. The dataset follows a Real2Sim2Real closed-loop paradigm, generating large-scale annotated data in simulation and evaluating model performance in real environments.
The proposed HS2Occ model exploits stereo vision depth priors for precise 2D-to-3D conversion. Experimental results show that HS2Occ significantly outperforms monocular baselines on both test sets and real-world scenarios, particularly in recognizing interaction-relevant semantic categories such as doors and cabinets.
This study provides the first panoramic stereo vision occupancy dataset for humanoid robots, addressing the limitations of existing datasets in near-field perception. By employing the Real2Sim2Real closed-loop paradigm, it enhances sim-to-real transferability, advancing safe navigation and interaction in complex indoor environments. Future work could explore more complex dynamic scenes and multimodal data fusion to further enhance model robustness and generalization.
Deep Analysis
Background
In the field of robot navigation, precise occupancy prediction is crucial for safety. Existing datasets are primarily designed for autonomous driving, emphasizing long-range geometry and static road scenes, which are inadequate for near-field perception in indoor environments for humanoid robots. Recently, indoor occupancy datasets have emerged, but they often rely on monocular cameras or expensive LiDAR equipment, limiting their feasibility in practical applications.
Core Problem
Humanoid robots require precise 3D occupancy perception to navigate and interact safely in indoor environments. Existing datasets lack near-field perception and panoramic coverage, making it challenging to support robots in complex dynamic environments.
Innovation
The Humanoid-OmniOcc dataset is the first panoramic stereo vision occupancy dataset for humanoid robots, employing a Real2Sim2Real closed-loop paradigm. The HS2Occ model leverages stereo vision depth priors for precise 2D-to-3D conversion, significantly enhancing model performance in complex indoor environments.
Methodology
- �� Dataset Construction: Built using NVIDIA Isaac Sim, employing four synchronized stereo cameras for full panoramic visual coverage.
- �� Real2Sim2Real Paradigm: Real sensor parameters drive simulation, simulation generates large-scale annotated data, and models are evaluated in real environments.
- �� HS2Occ Model: Utilizes stereo vision depth priors for precise 2D-to-3D conversion.
Experiments
Experiments were conducted in 15 simulated scenes and 5 real environments using the Humanoid-OmniOcc dataset. The model significantly outperformed monocular baselines on both test sets and real-world scenarios, particularly in recognizing interaction-relevant semantic categories.
Results
The HS2Occ model achieved 29.67% IoU and 11.69% mIoU on the test set, significantly outperforming monocular baselines. In real-world scenarios, the HS2Occ model reached 35.45% IoU and 19.26% mIoU, demonstrating strong generalization capabilities.
Applications
This study supports safe navigation and interaction of humanoid robots in complex indoor environments, particularly in scenarios requiring precise 3D perception.
Limitations & Outlook
The model's performance may degrade under mismatched lighting conditions and material reflectance, with limited perception capabilities for dynamic objects. Future work could explore more complex dynamic scenes and multimodal data fusion.
Plain Language Accessible to non-experts
Imagine you're in a large shopping mall surrounded by people and objects. A robot is like a blind person who needs to know what's around to walk safely. The Humanoid-OmniOcc dataset is like giving the robot a panoramic pair of glasses, allowing it to see its surroundings. Through these glasses, the robot can know where obstacles are and where it can safely pass. Just like we use our eyes and brain to judge our surroundings, the robot uses stereo cameras and algorithms to make judgments. This way, it can navigate safely and interact with people in complex indoor environments.
ELI14 Explained like you're 14
Imagine you're playing a virtual reality game with VR goggles, and you can see the world around you. Robots need this ability too! The Humanoid-OmniOcc dataset is like giving robots VR goggles, allowing them to see their surroundings. Through these 'goggles,' robots can know where obstacles are and where they can safely pass. Just like you avoid obstacles in a game, robots can avoid obstacles in the real world and move safely. Isn't that cool?
Glossary
Stereo Vision
Calculates depth information by capturing image differences from two cameras.
Used to generate 3D occupancy information, enhancing robot navigation capabilities.
Occupancy Prediction
Predicts the occupancy state (free or occupied) of each voxel in the environment.
Used for safe robot navigation in complex environments.
Real2Sim2Real
A closed-loop design from real sensor data to simulation and back to real environments.
Enhances sim-to-real transferability.
HS2Occ Model
A model utilizing stereo vision depth priors for 2D-to-3D conversion.
Used for occupancy prediction on the Humanoid-OmniOcc dataset.
NVIDIA Isaac Sim
A simulation platform for robot simulation and dataset generation.
Used to construct the Humanoid-OmniOcc dataset.
Open Questions Unanswered questions from this research
- 1 How to enhance model robustness and generalization in dynamic environments?
- 2 How can multimodal data fusion further improve occupancy prediction accuracy?
Applications
Immediate Applications
Indoor Robot Navigation
Enhance robot navigation capabilities in complex indoor environments using the dataset to avoid collisions.
Long-term Vision
Smart Home Robots
Provide precise environmental perception for future smart home robots, enabling more natural human-robot interaction.
Abstract
Occupancy prediction at voxel-level granularity is essential for safe robotic navigation and interaction in complex environments. Existing occupancy datasets, however, are predominantly designed for autonomous driving with vehicle-centric biases -- forward-facing cameras, far-field geometry, and static road priors -- limiting their applicability to embodied humanoid perception. We present Humanoid-OmniOcc, a large-scale panoramic stereo-based occupancy dataset tailored for humanoid robots. The dataset encompasses 15 diverse simulated indoor scenes and 5 real-world environments, yielding over 155K samples with broad scene and style diversity. Importantly, the dataset is designed around a Real2Sim2Real closed-loop paradigm: real sensor specifications drive physically accurate simulation, simulation produces large-scale annotated training data, and models trained in simulation are directly evaluated on real-world captures -- enabling iterative refinement of the sim-to-real pipeline. We further propose \textbf{H}umanoid \textbf{S}urround \textbf{S}tereo-guided \textbf{Occ}upancy model (Humanoid-OmniOcc) that exploits robust depth priors for accurate 2D-to-3D lifting. Extensive experiments show that Humanoid-OmniOcc consistently outperforms monocular baselines and generalizes well to both unseen simulated test scenes and real-world environments, validating the effectiveness of the Real2Sim2Real design. Code and data will be available upon acceptance at https://d-robotics-ai-lab.github.io/humanoid-omniocc.