DINOcular: Self-Supervised Visuospatial Representations

The evolution of RGB-D perception has seen significant progress with the advent of deep learning, especially in robotics and autonomous driving. Early methods relied on supervised learning for scene reconstruction and semantic segmentation, exemplified by architectures like DFormerv2 and multi-view stereo systems. Despite these advances, models predominantly trained on RGB data lack explicit spatial reasoning, limiting their ability to understand 3D structures. Recent transformer-based models such as DINO and iBOT have demonstrated remarkable semantic feature extraction but remain weak in geometric consistency. The integration of depth information remains a challenge due to heterogeneity, noise, and the difficulty of encoding 3D relationships directly into feature spaces. This gap motivates the development of methods that can learn joint visual-spatial representations in a scalable, self-supervised manner.

The core challenge is enabling vision models to learn comprehensive 3D spatial representations directly from RGB-D data without extensive supervision. Existing models either treat depth as a prediction target or rely on separate geometric modules, which limits end-to-end learning and generalization. The difficulty lies in effectively encoding spatial relationships within the attention mechanisms of transformers, managing noisy and heterogeneous depth data, and ensuring that learned features transfer well across tasks like semantic segmentation, correspondence estimation, and pose recognition. Addressing these issues is crucial for deploying autonomous systems capable of robust spatial reasoning in real-world environments.

This work introduces three key innovations: 1) the extension of rotary positional encoding into three dimensions (3D RoPE), embedding spatial relationships directly into attention weights; 2) a lightweight intra-patch depth feature fusion mechanism, incorporating local geometric cues such as surface normals; 3) a multi-view self-supervised training paradigm combining semantic and geometric objectives, including contrastive and correspondence losses. These innovations enable the model to encode 3D geometry explicitly, improve spatial consistency, and generalize across diverse tasks, surpassing prior RGB-only and RGB-D methods in both geometric and semantic benchmarks.

• �� Generate dense RGB-D data via monocular depth estimation and multi-view triangulation, providing input for training. • Embed patch coordinates (u, v) and mean depth (z) into attention via 3D RoPE, encoding spatial relationships directly into the transformer’s positional encodings. • Fuse local depth features, such as surface normals, with RGB patch embeddings through a linear projection, enriching intra-patch geometric information. • Employ multi-view pretraining with different viewpoints, applying self-supervised objectives (DINO, iBOT) at patch and image levels, with masking strategies to enhance spatial and semantic learning. • Incorporate multi-view correspondence and contrastive losses to enforce spatial consistency across views, with teacher-student models updated via exponential moving average. • Optimize the entire architecture end-to-end, ensuring the learned features encode both appearance and 3D structure, validated through downstream tasks.

The model was trained on dense RGB-D datasets generated from monocular depth estimation (MapAnything) on ImageNet-1k and multi-view triangulation (MVImgNet2.0). Baselines include DFormerv2 and other self-supervised RGB models like DINO. Evaluation metrics encompass geometric consistency (Probe3D), semantic segmentation (NYU DepthV2, SUN RGB-D), and correspondence estimation. Ablation studies tested different depth encoding strategies, multi-view loss formulations, and architectural modifications. The training involved approximately 94 million parameters, with hyperparameters tuned for multi-view and intra-patch dropout to prevent overfitting. The experiments demonstrated the effectiveness of the proposed methods across multiple tasks, with significant improvements over prior approaches.

The proposed model achieved a Probe3D geometric consistency score of 40.92, outperforming previous methods by about 4 points. In semantic segmentation, linear probing on NYU DepthV2 and SUN RGB-D reached 47.46% and 42.25%, respectively, surpassing RGB-only models. Multi-view consistency training improved spatial matching and pose estimation accuracy by 5-10%. Ablation results confirmed that 3D RoPE and intra-patch depth fusion significantly contributed to performance gains. The model also demonstrated strong generalization in correspondence and pose estimation tasks, validating the effectiveness of explicit geometric encoding within the transformer architecture.

This approach can be directly applied to robotic navigation, autonomous vehicles, and AR/VR systems, where accurate spatial understanding is critical. Its self-supervised nature reduces reliance on labeled data, making it suitable for large-scale deployment. The model’s ability to encode 3D geometry enables better scene reconstruction, object recognition, and spatial reasoning, facilitating more autonomous and intelligent systems. Long-term, integrating additional modalities like LiDAR or IMU could further enhance spatial perception, pushing towards fully autonomous spatial reasoning in complex environments.

Dependence on large-scale dense RGB-D datasets increases data collection costs. Depth noise and heterogeneity from different sensors can impair performance, especially in cluttered or dynamic scenes. The model’s computational complexity (~94M parameters) may hinder real-time deployment on resource-constrained devices. Its robustness in extreme scale variations and highly dynamic environments remains to be validated, necessitating further optimization and adaptation for practical applications.