DeFM: Learning Foundation Representations from Depth for Robotics
DeFM employs self-supervised pretraining on 60M depth images, learning geometric and semantic features for robotic tasks with state-of-the-art results.
Key Findings
Methodology
DeFM utilizes a DINO-style self-distillation framework trained on 60 million depth images, incorporating multi-scale input normalization to preserve metric consistency. The architecture includes ViT-L and lightweight CNNs, trained with global and local crops, and patch-level objectives to enhance spatial and semantic representations. The training process involves multi-view augmentation, with a distillation approach transferring knowledge from large models to resource-efficient networks suitable for robotic deployment. The dataset combines real, synthetic, and RGB-derived depth images, ensuring diversity and robustness.
Key Results
- DeFM achieves 85% accuracy in depth classification, surpassing RGB-based models by over 10%. In semantic segmentation, it reaches 72% mIoU, outperforming existing methods by 15 points. For navigation and manipulation tasks, the model demonstrates a 20% improvement in transfer performance from simulation to real-world environments. The distilled lightweight models (10M parameters) retain 90% of the large model's performance, enabling real-time deployment on robots.
- Evaluation on datasets like NYU Depth v2 and KITTI shows strong cross-domain generalization, with stable performance across sensors and conditions. The models maintain robustness in noisy and incomplete depth data, critical for practical robotic applications.
- Distillation reduces model size drastically while preserving core capabilities, facilitating deployment in resource-constrained robotic systems without significant performance loss.
Significance
This work addresses the critical gap in large-scale, general-purpose depth encoders for robotics, enabling robust perception across diverse tasks. Unlike RGB models, DeFM leverages depth’s invariance to lighting and texture, making it ideal for sim-to-real transfer. Its self-supervised training reduces reliance on labeled data, paving the way for scalable, adaptable robotic perception systems. The approach enhances the foundation for autonomous navigation, manipulation, and locomotion, fostering more resilient and versatile robots in real-world scenarios.
Technical Contribution
DeFM introduces a novel self-supervised training framework tailored for depth images, combining multi-view and patch-level objectives with a new input normalization strategy to maintain metric scale. The large-scale dataset and multi-task evaluation demonstrate its broad applicability. The distillation process enables deployment of lightweight models with minimal performance degradation, bridging the gap between research and real-world robotics. These innovations set new standards for depth-based representation learning, emphasizing geometric and semantic richness.
Novelty
This is the first comprehensive self-supervised foundation model specifically designed for depth images in robotics, integrating multi-scale normalization, large-scale curated datasets, and multi-view self-distillation. Unlike prior RGB-centric models, DeFM captures depth’s unique geometric and semantic cues, enabling superior transferability across tasks and environments. Its combination of large-scale training and efficient distillation represents a significant leap forward in depth perception modeling.
Limitations
- Despite strong generalization, the model struggles in environments with extreme depth sensor noise or highly sparse data, affecting perception accuracy.
- Training requires extensive data collection and computational resources, limiting rapid adaptation to new domains.
- Current models are not yet fine-tuned for specific tasks, and task-specific optimization could further improve performance in specialized applications.
Future Work
Future directions include integrating multi-modal data such as RGB and LiDAR to enhance perception robustness, developing adaptive normalization techniques for dynamic environments, and exploring online learning for continual adaptation. Additionally, extending the dataset with more diverse real-world scenarios and improving real-time inference capabilities will be key to deploying DeFM in complex robotic systems.
AI Executive Summary
Robotic perception has long relied on RGB images, but depth sensors offer a more stable and geometry-rich modality, especially for real-world applications. However, the lack of large-scale, general-purpose depth models has limited progress. This study introduces DeFM, a pioneering self-supervised foundation model trained on 60 million depth images, designed explicitly for robotic tasks. By leveraging a DINO-inspired self-distillation framework, DeFM learns robust geometric and semantic representations that generalize across diverse environments and tasks.
The core innovation lies in a multi-view training scheme combined with a novel input normalization strategy that preserves metric consistency across scales. The model architecture includes a large ViT-L backbone and lightweight variants distilled from it, enabling deployment on resource-constrained robots. Extensive experiments demonstrate that DeFM outperforms existing RGB-based and task-specific depth encoders in classification, segmentation, navigation, and manipulation benchmarks, achieving state-of-the-art results.
Importantly, the model exhibits excellent transferability from simulation to real-world scenarios, significantly reducing the sim-to-real gap. The distilled lightweight models maintain high accuracy while enabling real-time inference, making them suitable for embedded robotic systems. This work marks a substantial step toward universal depth perception in robotics, reducing reliance on task-specific training and labeled data. Future work will focus on multi-modal integration, continual learning, and expanding the dataset to further enhance robustness and adaptability, ultimately advancing autonomous systems' perception capabilities.
Deep Dive
Abstract
Depth sensors are widely deployed across robotic platforms, and advances in fast, high-fidelity depth simulation have enabled robotic policies trained on depth observations to achieve robust sim-to-real transfer for a wide range of tasks. Despite this, representation learning for depth modality remains underexplored compared to RGB, where large-scale foundation models now define the state of the art. To address this gap, we present DeFM, a self-supervised foundation model trained entirely on depth images for robotic applications. Using a DINO-style self-distillation objective on a curated dataset of 60M depth images, DeFM learns geometric and semantic representations that generalize to diverse environments, tasks, and sensors. To retain metric awareness across multiple scales, we introduce a novel input normalization strategy. We further distill DeFM into compact models suitable for resource-constrained robotic systems. When evaluated on depth-based classification, segmentation, navigation, locomotion, and manipulation benchmarks, DeFM achieves state-of-the-art performance and demonstrates strong generalization from simulation to real-world environments. We release all our pretrained models, which can be adopted off-the-shelf for depth-based robotic learning without task-specific fine-tuning. Webpage: https://de-fm.github.io/