Monocular 3D Occupancy Perception for Robots on Sidewalks via Hybrid 2D-3D Learning
WalkOCC uses hybrid 2D-3D learning for monocular 3D occupancy prediction, achieving a 15.6% mIoU improvement on Sidewalk3D.
Key Findings
Methodology
WalkOCC combines ray-marching and depth-aware lifting to transform monocular images into 3D occupancy grids, using hybrid training with pseudo 3D supervision and 2D consistency loss.
Key Results
- On Sidewalk3D, WalkOCC achieved an mIoU of 16.46, a 15.6% improvement over FlashOCC, excelling in dynamic categories like pedestrians and vehicles.
- In cross-domain tests, WalkOCC improved nighttime mIoU to 8.61, a 55% gain over baselines.
- Hybrid training enabled WalkOCC to achieve higher 2D-projected mIoU (25.5%) in cross-platform evaluations.
Significance
This work addresses the generalization gap of road datasets in sidewalk scenarios, offering a cost-efficient monocular solution for robot navigation in dynamic, complex environments.
Technical Contribution
Introduced ray-marching-based 2D-3D consistency loss, combined pseudo-label generation, and depth-aware lifting to reduce reliance on expensive 3D annotations.
Novelty
First to apply hybrid 2D-3D learning to monocular sidewalk scenarios, integrating pseudo-supervision and large-scale 2D data for cross-domain generalization and fine-grained predictions.
Limitations
- Relies on accurate camera calibration, which may degrade performance in poorly calibrated setups.
- Pseudo-label generation depends on limited RGB-LiDAR paired data, affecting distant object predictions.
- Thin structures like trees remain challenging in sparse or nighttime scenes.
Future Work
Future work could optimize pseudo-label generation, extend to more scenarios (e.g., nighttime), and explore more efficient online inference methods.
AI Executive Summary
Sidewalk environments are dynamic and complex, making generalization from traditional road datasets challenging. WalkOCC introduces a hybrid 2D-3D learning framework that leverages pseudo-3D supervision and large-scale 2D data to significantly enhance monocular 3D occupancy prediction.
Experiments show that WalkOCC achieves substantial mIoU improvements on the Sidewalk3D dataset, particularly excelling in dynamic categories like pedestrians and vehicles. Its hybrid training strategy demonstrates robustness in cross-domain tests, including nighttime and cross-platform scenarios.
However, the method depends on camera calibration and pseudo-label quality. Future research could refine pseudo-supervision strategies and expand to more challenging scenarios, providing stronger support for robot navigation in dynamic sidewalk environments.
Deep Analysis
Background
3D occupancy prediction has been extensively studied in autonomous driving, but existing methods heavily rely on large-scale LiDAR-RGB paired datasets, which fail to capture the complexity of sidewalks.
Core Problem
Sidewalks are dynamic and cluttered, with dense obstacles and pedestrians. Existing methods lack generalization and are constrained by the high cost of 3D annotations.
Innovation
WalkOCC introduces ray-marching-based 2D-3D consistency loss and depth-aware lifting, combining pseudo-label generation and hybrid training to address annotation cost and generalization challenges.
Methodology
- �� Use ResNet-50 to extract 2D image features.
- �� Transform features into 3D voxels via depth-aware lifting.
- �� Apply ray-marching-based 2D-3D consistency loss, combining pseudo-3D labels and 2D supervision for hybrid training.
Experiments
Trained and tested on Sidewalk3D, evaluated using mIoU and Occ IoU. Cross-domain tests included nighttime and cross-platform scenarios.
Results
WalkOCC achieved an mIoU of 16.46 on Sidewalk3D, 8.61 in nighttime scenarios, and 25.5% 2D-projected mIoU in cross-platform tests.
Applications
Applicable to delivery robots and electric wheelchairs in sidewalk scenarios, excelling in dynamic and complex environments.
Limitations & Outlook
Relies on pseudo-label quality and camera calibration, with challenges in sparse or nighttime scenarios.
Plain Language Accessible to non-experts
Imagine walking through a maze with one eye closed. WalkOCC acts like a smart assistant that turns your flat view into a 3D map, helping you avoid obstacles.
ELI14 Explained like you're 14
Think of playing a game with just a flat map. WalkOCC is like a magic tool that turns that map into a 3D model, showing where obstacles are and where you can go!
Glossary
Ray Marching
A method of sampling 3D points along a viewing ray to map 2D image features into 3D space.
Used for computing 2D-3D consistency loss.
Pseudo Label
Approximate labels generated from limited annotated data for weak supervision.
Used to train the initial 3D prediction model.
Depth-aware Lifting
Transforms 2D features into 3D voxels using depth information.
Used to generate 3D occupancy grids.
Sidewalk3D
A dataset with LiDAR-RGB paired data for evaluating 3D occupancy prediction.
Serves as the primary training and testing dataset.
mIoU
Mean Intersection over Union, a metric for evaluating semantic segmentation performance.
Used to assess 3D occupancy prediction accuracy.
Open Questions Unanswered questions from this research
- 1 How can pseudo-label generation be improved for unannotated data?
- 2 How can thin structures like trees be better predicted in sparse or nighttime scenarios?
Applications
Immediate Applications
Delivery Robot Navigation
Helps robots navigate complex sidewalks safely, improving delivery efficiency.
Electric Wheelchair Assistance
Provides real-time environmental awareness for wheelchair users, avoiding collisions.
Long-term Vision
Smart City Infrastructure
Supports city-scale dynamic environment monitoring and navigation optimization.
Abstract
Sidewalks in the real world are crowded, cluttered, and less structured than roads, making 3D occupancy prediction a key ingredient for the safe navigation of mobile robots such as delivery bots and electric wheelchairs. Existing occupancy learning pipelines are largely designed for on-road autonomous driving and often train on large-scale paired LiDAR-RGB datasets with dense 3D supervision and multiple camera inputs, which are costly to collect and do not adequately capture sidewalk-specific characteristics. We propose WalkOCC, a hybrid Ray-marching monocular 3D occupancy perception framework for robots operating on sidewalks. WalkOCC explicitly couples geometric grounding from LiDAR-RGB paired data with scalable learning from large-scale unpaired monocular images. It bootstraps pseudo occupancy supervision from paired sequences and jointly learns image-level representations on additional 2D-only data. It yields stable optimization and improved generalization without requiring costly 3D occupancy annotations. Extensive experiments demonstrate consistent gains in prediction accuracy, fine-grained segmentation of subtle urban structures such as curbs and gutters, and robustness to environmental and cross-embodiment shifts compared with self-supervised image-based baselines. To facilitate evaluation and benchmarking, we also introduce Sidewalk3D, a large-scale sidewalk perception dataset with LiDAR-camera paired sequences collected across multiple locations and time periods, along with 3D semantic occupancy annotations for evaluation. Code and data will be made available.