Depth Anything in $360^\circ$: Towards Scale Invariance in the Wild
DA360 leverages ViT-based shift learning for scale-invariant panoramic depth estimation, producing high-quality point clouds with seamless boundaries.
Key Findings
Methodology
The approach introduces a learnable shift parameter within the ViT backbone, transforming the model’s disparity output from affine-invariant to scale-invariant. This enables direct generation of well-structured 3D point clouds. Circular padding is integrated into the DPT decoder to eliminate seam artifacts, ensuring spherical continuity. Training on synthetic datasets with a scale-invariant disparity loss enhances zero-shot generalization across indoor and outdoor scenes, outperforming existing methods like PanDA and DreamCube.
Key Results
- On indoor benchmarks Matterport3D and Stanford2D3D, depth errors are reduced by over 50% and 10%, respectively, demonstrating significant accuracy improvements. In outdoor scenes, errors decrease by approximately 30%, establishing new state-of-the-art. Compared to PanDA, the model achieves about 30% relative error reduction across three datasets. Ablation studies confirm the importance of shift learning and circular padding, especially in complex outdoor environments, leading to more accurate and continuous point clouds.
Significance
This work addresses the longstanding challenge of achieving scale-invariant and spatially continuous depth estimation in panoramic images, crucial for applications like autonomous navigation and AR/VR. By enabling zero-shot transfer from perspective models, it reduces reliance on large labeled datasets, broadening deployment potential. The method’s robustness across diverse environments marks a significant step forward in 3D scene understanding, facilitating real-world applications with minimal supervision.
Technical Contribution
The key technical innovation is embedding a learnable shift parameter into the ViT-based depth estimation pipeline, transforming disparity outputs into scale-invariant estimates. Circular padding in the DPT decoder preserves spherical continuity, eliminating boundary artifacts. The combination of these mechanisms, supervised via a scale-invariant loss, allows direct point cloud generation. This approach fundamentally differs from prior affine-invariant methods, offering improved robustness and applicability in open-world scenarios.
Novelty
This is the first work to incorporate a learnable shift parameter within a Transformer-based panoramic depth model to achieve true scale invariance. The integration of circular padding into the dense prediction decoder to ensure seamless spherical boundaries is also novel. These innovations collectively enable direct, accurate 3D point cloud generation in diverse outdoor environments, surpassing existing zero-shot methods like PanDA and Marigold.
Limitations
- Despite improvements, the model still struggles with extremely complex scenes or distant objects under severe occlusion, mainly due to synthetic training data limitations. The reliance on synthetic datasets may limit real-world adaptation. Additionally, Transformer-based architectures impose computational costs that hinder real-time deployment, especially on resource-constrained devices.
Future Work
Future research will focus on integrating multi-modal data such as LiDAR and semantic cues to further improve accuracy. Developing lightweight Transformer variants for real-time inference and expanding outdoor datasets will enhance robustness. Exploring unsupervised or semi-supervised training strategies could reduce dependency on synthetic data, facilitating broader real-world deployment.
AI Executive Summary
Panoramic depth estimation is vital for comprehensive 3D scene understanding, with applications spanning robotics, AR/VR, and autonomous navigation. However, existing methods often rely heavily on large labeled datasets and struggle with generalization across diverse environments. This paper introduces DA360, a novel model that leverages a Transformer backbone with a learnable shift parameter to achieve true scale invariance in panoramic depth estimation. By transforming disparity outputs from affine-invariant to scale-invariant, DA360 can directly generate accurate 3D point clouds that respect spherical continuity.
The core innovation involves integrating a circular padding mechanism into the Dense Prediction Transformer (DPT) decoder, which effectively eliminates seam artifacts at the ERP boundaries, ensuring spatial coherence. The model is trained on synthetic datasets with a scale-invariant disparity loss, enabling it to generalize zero-shot to real indoor and outdoor scenes. Extensive evaluations on standard benchmarks like Matterport3D, Stanford2D3D, and a newly curated outdoor dataset Metropolis demonstrate that DA360 reduces depth errors by over 50% indoors and about 30% outdoors compared to baseline models, outperforming state-of-the-art methods such as PanDA.
This work significantly advances panoramic depth estimation by addressing the critical issues of scale ambiguity and boundary discontinuity. Its ability to produce high-quality, continuous point clouds in diverse environments opens new possibilities for real-world applications, including autonomous navigation, virtual reality, and 3D reconstruction. The approach’s robustness and efficiency suggest promising directions for future research, such as multi-modal fusion, lightweight architectures, and large-scale outdoor data collection, to further push the boundaries of panoramic scene understanding.
Deep Analysis
Background
Panoramic depth estimation作为空间理解的重要方向，经历了从传统几何方法到深度学习的演变。早期研究如OmniDepth探索全景几何特性，后续引入多视角融合和畸变抵抗架构（如PanoFormer、Elite360D），提升了模型的适应性。然而，受限于数据稀缺和模型泛化能力，现有方法在开源环境表现有限。近年来，基于Transformer的深度模型（如DPT、MiDaS）在单视角深度估计中取得突破，但在全景场景中的尺度不变性和连续性仍是难点。大型预训练模型如Depth Anything、Marigold通过迁移学习实现零样本泛化，但在全景应用中仍面临尺度模糊和边界缝隙问题。缺乏户外大规模标注数据，限制了模型的实际应用推广。
Core Problem
核心问题在于全景深度估计中的尺度不变性和空间连续性难以兼顾。现有模型多输出尺度和偏移不确定的深度或视差，导致点云结构不完整或边界缝隙明显，影响实际应用效果。尤其在开源环境中，缺乏统一的尺度不变监督机制，限制了模型的泛化能力。如何在保证空间连续性的同时实现尺度不变的深度估计，是当前技术的瓶颈。
Innovation
本研究提出三项核心创新：1）引入可学习的偏移参数，将深度从尺度和偏移不变转变为纯尺度不变，增强点云的结构完整性；2）结合Transformer中的全局特征，利用类Token回归偏移参数，提升尺度鲁棒性；3）在DPT解码器中引入圆形填充技术，确保全景图边界的空间连续性。这些创新有效解决了边界缝隙和尺度模糊问题，显著提升模型在开源场景中的表现。
Plain Language Accessible to non-experts
想象你在用一台全景相机拍摄一座城市的景色。传统方法就像用普通相机拍一张照片，虽然可以看到很多东西，但无法知道距离有多远，也不能把这些信息变成三维模型。而这项研究就像给相机装上了智能眼睛，能自动判断每个建筑物的距离，并且知道整个城市的空间布局。它还学会了在不同距离和角度下都能准确测量，不管是在室内还是户外。通过这项技术，我们可以用一张全景图直接生成城市的三维模型，帮助机器人导航、虚拟现实等应用变得更智能、更真实。
ELI14 Explained like you're 14
想象你用一个超级厉害的全景相机拍摄你的房间。以前的技术就像用普通相机拍照，只能看到房间的一部分，不能知道每个家具离你有多远，也不能拼出整个房间的三维图。而这次的研究就像给相机装上了魔法，让它可以自动判断每个家具的距离，并且知道整个房间的空间布局。它还能在照片的边界处无缝拼接，不会出现缝隙或错位。这样，你只用一张全景照片，就能得到一个完整的房间三维模型，非常酷！未来，这项技术可以帮机器人更聪明地在房间里走动，或者让虚拟现实变得更真实。
Glossary
Transformer (变换器)
一种基于自注意力机制的深度学习架构，擅长捕捉全局信息。在本文中用于提取全景图像的全局特征。
作为模型的主干网络，用于学习全景图像的全局表示。
尺度不变 (Scale Invariance)
模型输出对输入深度的缩放不敏感，能在不同距离范围内保持一致的估计效果。
核心目标，确保深度估计在不同场景中具有鲁棒性。
圆形填充 (Circular Padding)
一种边界处理技术，使全景图边缘像环一样连续，避免缝隙。
用于解码器中，确保空间连续性。
偏移参数 (Shift Parameter)
可学习的参数，用于将深度视差从尺度和偏移不变转变为纯尺度不变。
通过ViT中的类Token回归实现。
点云 (Point Cloud)
由大量空间点组成的三维空间表示，用于场景重建。
模型输出的深度信息可直接转化为点云。
Open Questions Unanswered questions from this research
- 1 如何进一步提升模型在极端复杂环境中的鲁棒性，尤其是在极远距离或极端光照条件下的表现。
- 2 缺乏多模态融合机制，结合LiDAR或语义信息以增强深度估计的准确性仍是未来方向。
Abstract
Panoramic depth estimation provides a comprehensive solution for capturing complete $360^\circ$ environmental structural information, offering significant benefits for robotics and AR/VR applications. However, while extensively studied in indoor settings, its zero-shot generalization to open-world domains lags far behind perspective images, which benefit from abundant training data. This disparity makes transferring capabilities from the perspective domain an attractive solution. To bridge this gap, we present Depth Anything in $360^\circ$ (DA360), a panoramic-adapted version of Depth Anything V2. Our key innovation involves learning a shift parameter from the ViT backbone, transforming the model's scale- and shift-invariant output into a scale-invariant estimate that directly yields well-formed 3D point clouds. This is complemented by integrating circular padding into the DPT decoder to eliminate seam artifacts, ensuring spatially coherent depth maps that respect spherical continuity. Evaluated on standard indoor benchmarks and our newly curated outdoor dataset, Metropolis, DA360 shows substantial gains over its base model, achieving over 50\% and 10\% relative depth error reduction on indoor and outdoor benchmarks, respectively. Furthermore, DA360 significantly outperforms robust panoramic depth estimation methods, achieving about 30\% relative error improvement compared to PanDA across all three test datasets and establishing new state-of-the-art performance for zero-shot panoramic depth estimation.