Self-supervised Learning for Dense Depth Estimation in Monocular Endoscopy
Proposes a self-supervised monocular endoscopy depth estimation method using SfM-generated sparse supervision, achieving submillimeter accuracy.
Key Findings
Methodology
This paper introduces a dual-branch Siamese network trained with self-supervision from SfM-derived sparse depth points and camera poses. The network includes a depth scale adjustment layer and a depth warping layer, enabling consistency enforcement across views. Loss functions comprise a scale-invariant weighted loss and a depth consistency loss, which together guide the network to predict dense, accurate depth maps from monocular endoscopic videos without manual annotations or CT data. The training leverages the sparse 3D points projected onto image planes, with confidence weights, to supervise the network. This approach allows fully automatic training on existing videos, adaptable to different patients and devices.
Key Results
- On sinus endoscopy data from two patients, the method achieved mean residual errors of 0.84mm and 0.63mm, outperforming traditional sparse SfM reconstructions. The dense depth maps closely matched CT ground truth, with errors below 1mm, suitable for clinical navigation. Ablation studies confirmed the importance of the scale-invariant and consistency losses. The model demonstrated robustness across different anatomical regions and patient data, indicating strong generalization.
- Quantitative comparisons showed significant improvements over unsupervised single-view methods, with error reductions of over 50%. The incorporation of multi-view geometric constraints was key to achieving high accuracy. The results suggest that leveraging sparse multi-view reconstructions effectively addresses the scale ambiguity and data sparsity issues inherent in monocular endoscopy.
- The approach's ability to produce dense, sub-millimeter accurate depth maps from standard videos highlights its potential for real-time surgical guidance, reducing reliance on manual annotations and expensive hardware.
Significance
This work addresses a critical bottleneck in endoscopic navigation: obtaining dense, accurate depth maps without additional hardware or manual labeling. By harnessing multi-view geometry and self-supervision, the method enables scalable, cost-effective 3D reconstruction directly from routine videos. Its high accuracy and robustness open new avenues for real-time, patient-specific surgical navigation, potentially improving outcomes in minimally invasive procedures. The framework also paves the way for broader applications in medical imaging where dense ground truth is unavailable, offering a paradigm shift in how deep learning models are trained in clinical settings.
Technical Contribution
The paper introduces a novel framework combining sparse multi-view geometric supervision with deep neural networks, incorporating a scale-invariant loss to resolve the inherent ambiguity in monocular depth. The network architecture is flexible, with custom layers for depth scaling and warping, enabling end-to-end training. This approach bridges the gap between traditional geometric methods and modern deep learning, providing a scalable, annotation-free solution for high-precision dense depth estimation in challenging endoscopic environments. It also demonstrates how to effectively utilize sparse reconstructions for supervising dense predictions, a significant step forward in medical image analysis.
Novelty
This is the first work to integrate sparse SfM-based multi-view supervision into a deep learning framework for dense depth estimation in monocular endoscopy. The key innovation lies in the scale-invariant loss and the depth consistency enforcement, which together enable accurate, dense reconstruction without manual labels or CT data. Unlike prior methods relying on synthetic data or dense annotations, this approach leverages existing geometric reconstructions, making it practical and scalable for clinical use. It represents a significant advancement over previous unsupervised or semi-supervised techniques, especially in the context of complex, texture-scarce endoscopic scenes.
Limitations
- The method's success heavily depends on the quality of SfM or SLAM reconstructions; failure in feature-poor regions or large systematic errors in pose estimation can impair performance. Extreme reflections, shadows, or lack of features in certain areas may cause reconstruction failures. The current approach also requires accurate camera intrinsic parameters, limiting its immediate deployment without calibration. Moreover, the model's generalization across different endoscopes and anatomical regions needs further validation.
- Computational costs during training are significant due to the geometric computations and network complexity. Real-time inference is feasible but not yet optimized for clinical workflows. Future work should focus on automatic failure detection, robustness to diverse lighting conditions, and multi-frame extensions for dynamic scenes.
Future Work
Future directions include developing multi-frame architectures to handle dynamic scenes, integrating real-time SLAM for continuous depth updates, and expanding datasets across diverse patients and devices. Automating failure detection and robustness to lighting variations will enhance clinical reliability. Additionally, combining this approach with other modalities like ultrasound or fluorescence imaging could further improve accuracy. Long-term, the goal is to embed this technology into real-time surgical navigation systems, enabling safer, more precise minimally invasive procedures.
AI Executive Summary
In minimally invasive surgery, precise navigation relies heavily on accurate 3D reconstructions of internal anatomy. Traditional methods depend on manual annotations, expensive hardware, or preoperative CT scans, which are costly and time-consuming. This paper introduces a groundbreaking deep learning framework that leverages self-supervision from existing endoscopic videos, eliminating the need for manual labels or CT data. The core idea is to utilize sparse 3D points reconstructed via Structure from Motion (SfM) as supervisory signals, guiding a neural network to predict dense depth maps from monocular images.
The framework employs a dual-branch Siamese network architecture, incorporating a scale-invariant loss to address the inherent ambiguity in monocular depth estimation. It also enforces depth consistency across views using relative camera poses, ensuring spatial coherence. These innovations enable the model to learn high-precision depth maps with residual errors below 1mm, validated on sinus endoscopy data from two patients. The results demonstrate that the method surpasses traditional sparse reconstructions, providing dense, accurate 3D models suitable for clinical navigation.
This approach offers a scalable, fully automatic solution that can be trained on routine videos without manual intervention, significantly lowering deployment barriers. Its robustness across different anatomical regions and patients indicates strong potential for widespread clinical adoption. While current limitations include dependence on SfM reconstruction quality and feature scarcity in some regions, ongoing improvements in multi-view geometry algorithms and deep learning will likely address these issues. Ultimately, this work marks a significant step toward real-time, patient-specific endoscopic navigation, promising safer and more effective minimally invasive procedures in the future.
Deep Analysis
Background
内窥镜在微创手术中的应用不断扩大，但其二维图像缺乏深度信息，限制了导航的精度。传统深度估计方法依赖硬件设备如立体相机或手动标注，成本高且操作繁琐。近年来，深度学习在场景深度估计中取得突破，但在内窥镜场景中面临数据稀缺、非Lambertian反射等挑战。多视几何方法如SfM和SLAM能提供稀疏的三维重建，但精度不足，难以满足临床需求。本文试图利用多视几何信息实现无标注的高精度深度重建，推动微创手术导航的发展。
Core Problem
核心问题是如何在没有人工标注和CT配准的情况下，从单目内窥镜视频中获得高精度的深度信息。现有方法多依赖稠密标注或硬件设备，成本高且不便于临床应用。多视几何重建虽能提供稀疏深度，但缺乏密集性和鲁棒性，难以实现精确导航。如何利用稀疏的多视几何信息训练深度网络，解决尺度模糊和数据稀疏问题，是当前的技术难点。
Innovation
本研究的创新点包括：1）结合SfM生成的稀疏深度和相机姿态作为自监督信号，用于训练深度网络；2）引入尺度不变损失，有效解决尺度模糊问题，使模型学习深度比例关系；3）设计深度一致性损失，确保不同视角间深度的空间一致性。这些创新使得无需人工标注即可实现亚毫米级深度重建，为内窥镜深度学习提供新思路。
Methodology
- �� 利用SfM或SLAM从内窥镜视频中自动生成稀疏三维点云和相对相机姿态。• 将稀疏点云投影到图像平面，生成稀疏深度图和软掩码，作为训练目标。• 构建双支路Siamese网络，包含深度尺度调整层和深度变换层，用于视角间深度一致性约束。• 设计尺度不变加权损失，忽略无深度值区域，解决尺度模糊问题。• 引入深度一致性损失，确保不同视角深度空间一致。• 训练过程中，结合稀疏深度和相机运动信息，优化网络参数，获得高精度深度图。
Experiments
使用两名患者的鼻窦内窥视频，采集22段短视频，分为训练和验证集。模型在不同场景下进行测试，比较稀疏SfM重建与深度网络输出的点云，利用CT进行配准，计算残差误差。超参数包括学习率1e-4，损失权重2e-4。采用多场景验证，确保模型鲁棒性。结果显示，平均残差低于1mm，满足临床导航需求。
Results
模型在两名患者的测试场景中，平均残差分别为0.84mm和0.63mm，优于传统稀疏重建。深度估计的误差显著低于单纯无监督方法，验证多视几何信息的有效性。消融实验表明，尺度不变损失和深度一致性损失对性能提升起关键作用。模型表现出良好的泛化能力，适应不同解剖区域和患者，显示出临床应用潜力。
Applications
该方法可应用于微创手术中的导航系统，利用现有内窥镜视频自动生成高精度三维模型，无需额外硬件或手动标注。适合在复杂解剖结构中实现精准导航，减少手术风险。未来可结合实时SLAM实现动态场景的深度估计，为机器人辅助手术提供支持。
Limitations & Outlook
依赖SfM或SLAM的重建质量，若重建失败或误差较大，模型性能受限。极端反射或阴影区域可能导致重建失效。模型对光照变化和非Lambertian表面表现有限，需结合多模态信息提升鲁棒性。未来需解决重建失败自动检测和多源数据融合问题，扩大适用范围。
Plain Language Accessible to non-experts
想象你在厨房做饭，食材、调料和厨具就像内窥镜中的图像和深度信息。传统方法就像用手工记录每样食材的具体位置，非常繁琐且容易出错。而这项新技术像是用一台智能相机，通过观察厨房中的多个角度，自动学习每样食材的准确位置和深度，不需要你手动标记。它利用厨房中不同角度的照片，结合厨师的动作，训练出一台“聪明的厨师助手”，可以在你还没告诉它具体位置时，就帮你准确找到每样食材。这样一来，厨房变得更智能，做饭也更快更准。这个比喻说明了如何用多视角信息训练深度模型，达到无需人工干预的高精度效果。
ELI14 Explained like you're 14
想象你在玩一个超级酷的游戏，你的任务是找到隐藏在房间里的宝藏，但你只能用一台相机看。每次你转头看不同的角度，你都能看到一些线索，但没有直接告诉你宝藏在哪里。现在，这个新方法就像是让游戏中的智能机器人，通过观察房间的多个角度，自己学习宝藏的藏法。它不用你告诉它宝藏在哪里，只靠观察和一些线索，就能学会判断宝藏的位置。它还会记住每次观察的角度和距离，确保每次都能找到宝藏，甚至比人类更快更准。这个机器人就像是内窥镜中的深度“侦探”，用聪明的算法帮医生在手术中找到关键的结构，避免误伤。是不是很酷？它让复杂的手术变得更安全、更精准，就像你在游戏中变成了超级侦探一样！
Abstract
We present a self-supervised approach to training convolutional neural networks for dense depth estimation from monocular endoscopy data without a priori modeling of anatomy or shading. Our method only requires sequential data from monocular endoscopic videos and a multi-view stereo reconstruction method, e.g. structure from motion, that supervises learning in a sparse but accurate manner. Consequently, our method requires neither manual interaction, such as scaling or labeling, nor patient CT in the training and application phases. We demonstrate the performance of our method on sinus endoscopy data from two patients and validate depth prediction quantitatively using corresponding patient CT scans where we found submillimeter residual errors.