TR2M: Transferring Monocular Relative Depth to Metric Depth with Language Descriptions and Dual-Level Scale-Oriented Contrast
TR2M leverages image and text to predict pixel-wise scale maps, converting relative to absolute depth with high cross-domain generalization.
Key Findings
Methodology
TR2M integrates features from frozen image (DINOv2) and text (CLIP) encoders via cross-modality attention. It predicts scale and shift maps at pixel level to transform relative depth (from a pretrained RDE model) into metric depth. Pseudo depth generation and filtering refine supervision signals. Dual-level scale-oriented contrastive loss enforces feature consistency aligned with depth distribution, enhancing scale perception. The lightweight architecture (19M parameters) demonstrates strong zero-shot cross-domain performance.
Key Results
- On NYUv2, TR2M achieves an AbsRel of 0.082, surpassing many state-of-the-art methods, with fewer parameters and no post-processing. It outperforms existing language-based transfer methods and approaches the accuracy of supervised models.
- On KITTI, it reaches an AbsRel of 0.066, competitive with top end-to-end models, confirming robustness in outdoor scenes. Ablation studies show that multimodal fusion, pseudo depth filtering, and contrastive loss significantly boost performance.
- Zero-shot evaluation on five unseen datasets confirms the model’s excellent generalization, maintaining low errors and high scale consistency across diverse environments.
Significance
This work addresses the longstanding challenge of scale ambiguity in monocular depth estimation, especially in cross-domain scenarios. By integrating language descriptions with visual cues, TR2M offers a low-cost, scalable solution that can be deployed in autonomous vehicles, robotics, and AR/VR applications. Its ability to perform pixel-wise scale transfer with minimal parameters marks a significant step toward universal, sensor-free depth perception, reducing reliance on expensive sensors and calibration. The approach opens new avenues for multimodal scene understanding, fostering more adaptable and intelligent perception systems.
Technical Contribution
The paper introduces a novel multimodal pixel-wise scale prediction framework, combining pre-trained encoders, cross-modality attention, pseudo depth filtering, and dual-level contrastive learning. These innovations enable precise scale transfer with minimal parameters, outperforming traditional methods that rely on large datasets or sensor calibration. The model’s design emphasizes efficiency, interpretability, and robustness, setting a new benchmark for cross-domain monocular depth estimation.
Novelty
This is the first work to utilize textual descriptions for pixel-level transfer from relative to absolute depth, integrating multimodal feature fusion with pseudo depth filtering and scale contrast. Unlike prior methods that depend solely on geometric cues or large-scale supervision, TR2M leverages language as an explicit, flexible guide, enabling zero-shot generalization across diverse scenes. Its lightweight design and pixel-wise transformation mechanism represent a pioneering advance in depth estimation.
Limitations
- The approach depends on the quality of pre-trained encoders; ambiguous or inaccurate descriptions can degrade performance. It may struggle in highly dynamic scenes or with complex motion where static cues are insufficient.
- Pseudo depth filtering relies on hyperparameters sensitive to noise, which may introduce errors in noisy environments. The method’s effectiveness diminishes if the pseudo depth quality is compromised.
- Current model primarily targets static scenes; extending to real-time dynamic environments requires further optimization for speed and robustness.
Future Work
Future directions include integrating temporal information for video-based depth estimation, exploring unsupervised or self-supervised training to reduce reliance on pre-trained encoders, and enhancing robustness in dynamic or low-light conditions. Additionally, expanding the framework to multi-task learning with semantic and scene understanding could further improve practical deployment.
AI Executive Summary
Monocular depth estimation remains a fundamental yet challenging task in computer vision, vital for scene understanding in robotics, autonomous driving, and AR/VR. Traditional methods either rely on metric depth models, which require extensive calibration and domain-specific data, or on relative depth models that generalize well but lack absolute scale information. This dichotomy limits the applicability of monocular depth estimation in real-world, cross-domain scenarios. Addressing this, the paper introduces TR2M, a novel framework that leverages both image and language descriptions to predict pixel-wise scale maps, enabling the transformation of relative depth into metric depth with high accuracy and generalization.
TR2M employs pre-trained multimodal encoders—DINOv2 for images and CLIP for text—to extract features, which are fused via cross-modality attention. The core innovation lies in predicting scale and shift maps at pixel level, allowing fine-grained depth conversion. To supervise this process, the authors generate pseudo depth maps by aligning relative and ground-truth metric depths through least-squares fitting, filtering high-confidence pseudo labels to improve supervision quality. Additionally, a dual-level scale-oriented contrastive loss enforces feature consistency aligned with depth distributions, further enhancing scale perception.
Extensive experiments on NYUv2 and KITTI datasets demonstrate that TR2M outperforms existing methods, achieving an AbsRel of 0.082 and 0.066 respectively, with only 19 million parameters. Its zero-shot performance on five unseen datasets confirms robust cross-domain generalization. The lightweight design and reliance on accessible data (images and descriptions) make TR2M highly practical for deployment in resource-constrained environments.
This work significantly advances monocular depth estimation by integrating language as an explicit guide for pixel-level scale transfer, reducing dependence on sensors and large-scale annotations. It opens new avenues for multimodal scene understanding, with promising applications in autonomous systems and virtual environments. Future research will focus on dynamic scenes, real-time processing, and multi-task integration, further broadening its impact.
Deep Analysis
Background
深度估计是计算机视觉中的核心任务之一，早期依赖激光扫描或结构光等昂贵传感器。随着深度学习的发展，出现了端到端的单目深度估计模型，如Eigen et al.（2014）提出的多尺度卷积网络，以及ResNet、DPT等架构。这些模型在特定场景中取得了显著性能，但在跨域应用中表现不足，主要因尺度不一致和环境变化。相对深度估计（MRDE）通过学习场景中的相对关系，增强了泛化能力，但缺乏绝对尺度信息，限制了实际应用。近年来，多模态信息（如文本描述）被引入深度估计，旨在结合视觉和语义信息提升模型的鲁棒性和尺度感知能力。
Core Problem
现有深度估计方法在尺度一致性和跨域泛化方面存在瓶颈。绝对深度模型依赖大量标注数据和特定传感器，难以迁移到新场景；相对深度模型虽泛化好，但无法提供绝对尺度，限制了导航、机器人等应用。引入文本描述作为辅助信息，虽然提升了某些场景的性能，但多模态融合的像素级尺度转化仍未充分解决。如何在保持模型参数少、泛化强的同时，实现像素级的尺度转移，是当前的核心难题。
Innovation
本研究提出TR2M，结合图像和文本信息，利用预训练的多模态编码器提取特征，通过交叉模态注意力融合信息，预测像素级的尺度图和偏移图，实现相对深度到绝对深度的转化。引入伪深度筛选策略，利用最小二乘法对伪深度进行线性校准，筛选高置信度样本作为监督信号。采用双层尺度对比损失，强化尺度一致性和特征分布匹配。模型参数极少，具备强大的跨域迁移能力。这些创新点共同推动了深度估计的理论和实践发展。
Methodology
- �� 输入：RGB图像I和文本描述L。
- �� 特征提取：用预训练的DINOv2（图像）和CLIP（文本）提取FI和FL。
- �� 特征融合：通过交叉模态注意力机制（Eq.1-2）融合特征，得到Ff。
- �� 变换图预测：用两个轻量级解码器预测尺度图A和偏移图B。
- �� 深度转化：利用公式（Eq.3）将相对深度Dr通过像素线性变换转为绝对深度。
- �� 伪深度筛选：通过最小二乘法（Eq.5）校准伪深度，筛选可信样本。
- �� 损失函数：结合尺度不变的对数损失（Eq.4）、伪深度对比（Eq.6）和尺度对比（Eq.7-9）优化模型。
- �� 训练：端到端，加入边缘平滑损失（Les）确保平滑性。
Experiments
在NYUv2和KITTI上训练评估，指标包括AbsRel、RMSE等。模型参数少，采用AdamW，批次8，学习率1e-5，训练20轮。对比不同模态融合、伪深度筛选和尺度对比的效果。在五个未见数据集上验证零样本泛化能力。结果显示，TR2M在绝对误差和尺度一致性方面优于多项基线，验证其跨域适应性。
Results
NYUv2上，AbsRel为0.082，优于大部分迁移模型；KITTI上，AbsRel为0.066，接近最先进的端到端模型。参数仅19M，模型轻量，适合部署。消融实验表明，融合文本信息、伪深度筛选和尺度对比是性能提升关键。零样本测试中，表现优异，展现强大泛化能力。
Applications
该技术适用于无人驾驶、机器人导航、虚拟现实等场景，尤其在缺乏多传感器时，通过文本描述实现高精度深度估计。参数少，易于在边缘设备部署，成本低。未来结合动态场景、多任务学习，提升实时性和鲁棒性。
Limitations & Outlook
依赖预训练编码器，描述模糊或语义不清时性能下降。伪深度筛选对超参数敏感，噪声多时可能引入误差。主要针对静态场景，动态变化和运动信息未充分考虑。未来需优化鲁棒性和实时性。
Plain Language Accessible to non-experts
想象你在一个工厂里，工人们需要知道每个物品的具体位置和大小，但没有尺子或传感器。于是他们用相机拍照，再用描述告诉别人：‘这个箱子大约有两米长，放在左边的架子上’。工人们结合图片和描述，利用经验和规则，推算出每个物品的准确位置和大小。TR2M就像这些工人，利用图片和文字信息，拼出场景的深度，让机器人更聪明、更懂环境。它通过学习不同场景的规律，能在没有专门传感器的情况下，准确判断距离和深度，帮助机器人更安全、更智能地工作。
ELI14 Explained like you're 14
想象你在玩一个游戏，你需要知道每个物体离你有多远，但没有尺子。你只能看图片和听描述，比如‘前面有辆车，距离大约20米’。你根据线索猜出距离。这就像TR2M，用图片和文字帮机器人判断距离，不用激光或特殊传感器。它学会用这些线索，像我们用眼睛和话语一样，理解场景中的距离关系。这样，机器人在街道、家里或手术室都能准确知道距离。这让机器人变得更聪明，也更能帮上忙。
Abstract
This work presents a generalizable framework to transfer relative depth to metric depth. Current monocular depth estimation methods are mainly divided into metric depth estimation (MMDE) and relative depth estimation (MRDE). MMDEs estimate depth in metric scale but are often limited to a specific domain. MRDEs generalize well across different domains, but with uncertain scales which hinders downstream applications. To this end, we aim to build up a framework to solve scale uncertainty and transfer relative depth to metric depth. Previous methods used language as input and estimated two factors for conducting rescaling. Our approach, TR2M, utilizes both text description and image as inputs and estimates two rescale maps to transfer relative depth to metric depth at pixel level. Features from two modalities are fused with a cross-modality attention module to better capture scale information. A strategy is designed to construct and filter confident pseudo metric depth for more comprehensive supervision. We also develop scale-oriented contrastive learning to utilize depth distribution as guidance to enforce the model learning about intrinsic knowledge aligning with the scale distribution. TR2M only exploits a small number of trainable parameters to train on datasets in various domains and experiments not only demonstrate TR2M's great performance in seen datasets but also reveal superior zero-shot capabilities on five unseen datasets. We show the huge potential in pixel-wise transferring relative depth to metric depth with language assistance. (Code is available at: https://github.com/BeileiCui/TR2M)