Dense Depth Estimation in Monocular Endoscopy with Self-supervised Learning Methods
Proposed a self-supervised monocular endoscopy depth estimation method leveraging SfM sparse supervision, achieving sub-millimeter accuracy without manual labels.
Key Findings
Methodology
This paper introduces an end-to-end deep learning framework that combines multi-view geometric information with neural networks for dense depth estimation from monocular endoscopic videos. The core innovation lies in utilizing sparse point clouds and camera poses obtained via Structure from Motion (SfM) as self-supervisory signals. The architecture employs a dual-branch Siamese network, where each branch processes a frame, predicts depth, and incorporates geometric constraints through custom loss functions. These include Sparse Flow Loss (SFL) and Depth Consistency Loss (DCL), which enforce the geometric relationships between frames without relying on photometric constancy. A Depth Scaling Layer aligns the predicted depth scale with SfM outputs, addressing the scale ambiguity inherent in monocular setups. The training pipeline automates data preprocessing, including undistortion, sparse reconstruction, and sparse flow computation, enabling scalable training on unlabeled videos. The model learns to predict dense depth maps that are consistent with sparse geometric cues, resulting in high-precision reconstructions suitable for clinical applications.
Key Results
- In cross-patient experiments using CT scans as ground truth, the method achieved an average residual error below 0.5 mm, outperforming previous self-supervised methods like Zhou et al. (2017) and Yin et al. (2018), which reported errors around 1.2 mm.
- On in vivo sinus endoscopy data, the proposed approach significantly outperformed recent natural video-based self-supervised methods, reducing depth estimation errors by over 40%, demonstrating robustness across different anatomical regions and imaging conditions.
- The model generalizes well across different patients and endoscope types, maintaining sub-millimeter accuracy without retraining, validated through point cloud registration with CT surface models and sparse depth comparisons.
Significance
This work addresses a critical challenge in medical imaging: achieving accurate, dense 3D reconstructions from monocular videos without extensive manual annotation or expensive hardware. By leveraging geometric cues inherent in endoscopic videos, the method offers a scalable, cost-effective solution for intraoperative navigation, surgical planning, and robotic assistance. Its ability to produce high-fidelity depth maps in real-time can transform minimally invasive procedures, reducing risks and improving outcomes. Moreover, the framework sets a precedent for applying self-supervised learning in other medical imaging modalities where dense ground truth is scarce, opening new avenues for research and clinical deployment.
Technical Contribution
The main technical innovations include: • Integration of SfM-derived sparse point clouds and camera poses into a deep learning training pipeline, enabling self-supervision without manual labels; • Development of novel loss functions—Sparse Flow Loss and Depth Consistency Loss—that enforce geometric constraints and long-range correlations between frames; • Introduction of a Depth Scaling Layer to address the scale ambiguity problem, facilitating cross-patient and cross-device generalization; • Custom differentiable layers for depth warping and flow estimation, allowing end-to-end optimization of the network with geometric supervision. These contributions collectively advance the state-of-the-art in monocular depth estimation within challenging medical environments.
Novelty
This research is pioneering in its exclusive reliance on monocular endoscopic videos and sparse geometric cues for dense depth estimation, eliminating the need for manual annotations, CT scans, or stereo hardware. Unlike prior works that depend on synthetic data or hardware modifications, this approach harnesses multi-view geometry directly from videos, integrating it into a deep learning framework. The combination of geometric loss functions, scale-aware layers, and end-to-end training represents a significant leap forward, enabling sub-millimeter accuracy in complex biological tissues—a feat not achieved by previous methods in this domain.
Limitations
- The method's performance depends heavily on the quality of SfM reconstructions; scenes with poor feature matching or occlusions can lead to degraded depth estimates.
- The approach estimates relative depth maps, lacking absolute scale; additional calibration or external references are needed for real-world measurements.
- In scenes with extreme lighting variations or rapid tissue deformation, geometric constraints may be insufficient, necessitating further robustness enhancements.
- Computational complexity and processing time may limit real-time deployment without hardware acceleration; future work should optimize inference speed.
Future Work
Future research will focus on integrating additional modalities such as ultrasound or optical coherence tomography to recover absolute scale, improving robustness to tissue deformation and lighting changes. Developing real-time capable architectures and exploring unsupervised domain adaptation techniques will facilitate clinical translation. Additionally, extending the framework to dynamic scenes with tissue motion and deformation will be crucial for intraoperative applications. Collaborations with clinical partners are planned to validate the system in live surgical settings, aiming for seamless integration into surgical workflows.
AI Executive Summary
Minimally invasive surgeries, particularly in delicate regions like the sinuses, demand precise spatial awareness to avoid critical structures such as the brain, eyes, and major arteries. Traditional navigation systems rely heavily on preoperative imaging like CT scans, which require complex registration procedures and are not always feasible intraoperatively. Moreover, existing real-time depth estimation methods often depend on stereo cameras or extensive manual annotations, limiting their practicality and scalability.
This paper introduces a novel self-supervised deep learning framework for dense depth estimation from monocular endoscopic videos. The core idea is to leverage the inherent geometric information captured by structure-from-motion (SfM) algorithms applied to unlabeled videos. By extracting sparse 3D points and camera poses, the authors design a training pipeline that enforces geometric consistency through custom loss functions, including Sparse Flow Loss and Depth Consistency Loss. These losses guide the neural network to predict dense depth maps that align with the sparse geometric cues, without requiring manual labels or additional imaging modalities.
The network architecture builds upon a modified DenseNet backbone, optimized for depth prediction, and incorporates several novel layers—such as the Depth Scaling Layer—to address the scale ambiguity problem common in monocular setups. During training, pairs of frames are processed through a Siamese network, with the geometric constraints ensuring that the predicted depths are consistent across different viewpoints. The training pipeline automates data preprocessing, including undistortion, sparse reconstruction, and flow computation, making it scalable to large datasets.
Experimental results demonstrate that this approach achieves sub-millimeter accuracy in cross-patient experiments, with errors below 0.5 mm when compared to CT-derived ground truth. The model generalizes well across different patients, devices, and anatomical regions, outperforming existing self-supervised methods designed for natural scenes. These findings highlight the potential of integrating multi-view geometry with deep learning to overcome traditional limitations in medical image analysis.
Overall, this work paves the way for practical, cost-effective, and highly accurate intraoperative navigation systems based solely on monocular videos. Its ability to produce dense, reliable depth maps without extensive annotations or specialized hardware could revolutionize minimally invasive surgery, making procedures safer and more precise. Future directions include incorporating additional modalities for absolute scale recovery, optimizing for real-time deployment, and validating in clinical settings to fully realize its transformative potential.
Deep Dive
Abstract
We present a self-supervised approach to training convolutional neural networks for dense depth estimation from monocular endoscopy data without a priori modeling of anatomy or shading. Our method only requires monocular endoscopic videos and a multi-view stereo method, e.g., structure from motion, to supervise learning in a sparse manner. Consequently, our method requires neither manual labeling nor patient computed tomography (CT) scan in the training and application phases. In a cross-patient experiment using CT scans as groundtruth, the proposed method achieved submillimeter mean residual error. In a comparison study to recent self-supervised depth estimation methods designed for natural video on in vivo sinus endoscopy data, we demonstrate that the proposed approach outperforms the previous methods by a large margin. The source code for this work is publicly available online at https://github.com/lppllppl920/EndoscopyDepthEstimation-Pytorch.
References (20)
Visual SLAM for Handheld Monocular Endoscope
Oscar G. Grasa, E. Bernal, Santiago Casado et al.
Self-supervised Learning for Dense Depth Estimation in Monocular Endoscopy
Xingtong Liu, Ayushi Sinha, M. Unberath et al.
Evaluation and Stability Analysis of Video-Based Navigation System for Functional Endoscopic Sinus Surgery on In Vivo Clinical Data
S. Léonard, Ayushi Sinha, A. Reiter et al.
Deconvolution and Checkerboard Artifacts
Augustus Odena, Vincent Dumoulin, Christopher Olah
Learning a similarity metric discriminatively, with application to face verification
S. Chopra, R. Hadsell, Yann LeCun
Magnetically Activated Stereoscopic Vision System for Laparoendoscopic Single-Site Surgery
Massimiliano Simi, M. Silvestri, C. Cavallotti et al.
Screened poisson surface reconstruction
M. Kazhdan, Hugues Hoppe
Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
D. Eigen, Christian Puhrsch, R. Fergus
Iterative Most Likely Oriented Point Registration
Seth D. Billings, R. Taylor
Vision-based endoscope tracking for 3D ultrasound image-guided surgical navigation
Liangjing Yang, Junchen Wang, Takehiro Ando et al.
U-Net: Convolutional Networks for Biomedical Image Segmentation
O. Ronneberger, P. Fischer, T. Brox
Spatial Transformer Networks
Max Jaderberg, K. Simonyan, Andrew Zisserman et al.
Vision-based markerless registration using stereo vision and an augmented reality surgical navigation system: a pilot study
H. Suenaga, H. Tran, H. Liao et al.
Compact stereo endoscopic camera using microprism arrays.
Sung-Pyo Yang, Jae-jun Kim, Kyung-Won Jang et al.
Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue
Ravi Garg, B. V. Kumar, G. Carneiro et al.
Deeper Depth Prediction with Fully Convolutional Residual Networks
Iro Laina, C. Rupprecht, Vasileios Belagiannis et al.
Unsupervised Scale-consistent Depth and Ego-motion Learning from Monocular Video
Jiawang Bian, Zhichao Li, Naiyan Wang et al.
The One Hundred Layers Tiramisu: Fully Convolutional DenseNets for Semantic Segmentation
S. Jégou, M. Drozdzal, David Vázquez et al.
Simultaneous segmentation and correspondence improvement using statistical modes
Ayushi Sinha, A. Reiter, S. Léonard et al.
Cited By (20)
EndoPBR: Material and Lighting Estimation for Photorealistic Surgical Simulations via Physically-based Rendering
Improved Self-supervised Monocular Endoscopic Depth Estimation based on Pose Alignment-friendly Dynamic View Selection
EndoDepth: A Benchmark for Assessing Robustness in Endoscopic Depth Prediction
Benchmarking Robustness of Endoscopic Depth Estimation with Synthetically Corrupted Data
EndoOmni: Zero-Shot Cross-Dataset Depth Estimation in Endoscopy by Robust Self-Learning from Noisy Labels
Self-supervised endoscopy depth estimation framework with CLIP-guidance segmentation
Depth estimation from monocular endoscopy using simulation and image transfer approach
EndoPerfect: High-Accuracy Monocular Depth Estimation and 3D Reconstruction for Endoscopic Surgery via NeRF-Stereo Fusion
Direct Camera-Only Bundle Adjustment for 3-D Textured Colon Surface Reconstruction Based on Pre-Operative Model
Self-Supervised Feature Detection and 3D Reconstruction for Real-Time Neuroendoscopic Guidance
High-Fidelity 3D Reconstruction for Accurate Anatomical Measurements in Endoscopic Sinus Surgery
High-Quality Novel View Synthesis of Robotic Surgical Scenes using Gaussian Splatting with Depth Prior
Towards Dynamic 3D Reconstruction of Hand-Instrument Interaction in Ophthalmic Surgery
TR2M: Transferring Monocular Relative Depth to Metric Depth with Language Descriptions and Scale-Oriented Contrast
EndoMUST: Monocular Depth Estimation for Robotic Endoscopy via End-to-end Multi-step Self-supervised Training
Artificial Intelligence in Rhinology.
Geo-RepNet: Geometry-Aware Representation Learning for Surgical Phase Recognition in Endoscopic Submucosal Dissection
A review on computer-aided diagnostic system to classify the disorders of the gastrointestinal tract
Endoscopic Depth Estimation Based on Deep Learning: A Survey
M$^{3}$-DEGREES Net: Monocular-Guided Metric Marching Depth Estimation With Graph-Based Relevance Ensemble for Endoluminal Surgery