AI-based single-shot structured-light depth reconstruction for real-time laparoscopic surgical guidance
Single-shot structured light depth reconstruction using VQ-VAE and U-Net achieves 3.70mm MAE at 26Hz, enabling real-time laparoscopic guidance.
Key Findings
Methodology
This paper introduces a passive LED-illuminated binary mask combined with a compact projection module integrated into a dual-channel laparoscope. The system captures a single frame of the scene with a passive binary pattern projected onto the target surface, eliminating the need for active DMD projection and synchronization. During training, a Zivid structured-light 3D camera provides reference depth maps, which are reprojected into the endoscopic image frame. The core of the deep learning approach employs a vector-quantized variational autoencoder (VQ-VAE) to encode the input fringe image into a discrete latent space, which is then processed by a custom multi-resolution U-Net to directly regress the dense depth map. This end-to-end pipeline operates in real-time, achieving high accuracy without explicit segmentation or multi-shot acquisition, and is optimized for clinical deployment in minimally invasive surgery.
Key Results
- On a dataset of 722 paired phantom images, the proposed VQ-VAE + U-Net model achieved an MAE of 3.70mm, outperforming the baseline dual U-Net MaskNet + DepthNet, which had an MAE of 4.12mm. The model also achieved an absolute relative error (AbsRel) of 0.0326, with δ=1.1 accuracy of 0.962 and δ=1.12 accuracy of 0.970. These results demonstrate a significant improvement over off-the-shelf monocular depth models, which showed higher errors and lower accuracy metrics. The system maintained a stable inference rate of 26Hz over 301 consecutive frames on an NVIDIA A100 GPU, confirming its suitability for real-time surgical guidance.
- Furthermore, the approach effectively decouples depth estimation from explicit segmentation, leveraging the learned prior in the latent space to produce smooth and consistent depth maps. The experimental validation confirms that the passive binary pattern remains resolvable at typical laparoscopic working distances (7-10cm), ensuring practical applicability. The results highlight the advantages of task-specific training and the robustness of the latent-space regularization, which collectively improve depth accuracy and inference stability.
Significance
This work addresses a critical bottleneck in minimally invasive surgery: real-time, accurate depth perception without complex hardware synchronization or multi-frame acquisition. By replacing active projectors with passive binary masks and employing deep generative priors, the system simplifies hardware design, reduces size and thermal load, and enhances robustness against physiological motion. The approach paves the way for deploying high-precision, video-rate depth sensing in clinical laparoscopic platforms, significantly advancing the capabilities of robotic-assisted surgery. It also opens new avenues for integrating deep learning-based structured light into compact, low-cost surgical tools, potentially transforming intraoperative navigation and autonomous robotic interventions.
Technical Contribution
The key technical innovations include the integration of a passive LED-illuminated binary mask with a miniaturized projection module, eliminating the need for active DMD-based projection and synchronization. The deep learning framework leverages a VQ-VAE to encode fringe images into a discrete latent space, which acts as a structured prior, constraining the depth prediction to plausible solutions. The custom U-Net operates directly in this latent space, removing the need for explicit mask segmentation and enabling stable, fast inference. The training protocol uses independent reference depth maps obtained from a Zivid structured-light camera, decoupling the model from traditional multi-shot calibration pipelines. The entire system operates at 26Hz, demonstrating a practical, scalable solution for real-time endoscopic depth sensing.
Novelty
This study is the first to combine passive LED binary pattern projection with a VQ-VAE-based deep regression framework for single-shot endoscopic depth reconstruction. Unlike prior approaches relying on active DMD projection and multi-frame acquisition, this work achieves synchronization-free, high-accuracy depth estimation from a single image. The use of a learned discrete latent space as a structural prior introduces a new paradigm in endoscopic structured light, significantly simplifying hardware and improving robustness. The approach also decouples depth supervision from traditional multi-shot algorithms, enabling more flexible and scalable system design, which is a notable departure from existing methods in the field.
Limitations
- The system's performance depends heavily on precise calibration between the structured-light camera and the endoscope; calibration errors can introduce depth inaccuracies. The binary pattern visibility may degrade under extreme lighting conditions or highly reflective surfaces, limiting robustness in certain surgical environments. The current hardware design, while compact, still faces challenges related to thermal management and optical alignment in clinical settings. Additionally, the model's generalization to different tissue types and geometries needs further validation, especially in vivo. Future work should focus on improving calibration robustness, adaptive pattern visibility, and broader clinical testing.
Future Work
Future directions include integrating multi-modal imaging modalities such as OCT or ultrasound to enhance depth accuracy and robustness. Developing adaptive calibration algorithms that can self-correct in real-time will further improve system reliability. Hardware improvements aimed at miniaturization, thermal management, and robustness in complex surgical environments are also planned. Moreover, expanding the dataset to include in vivo tissues and diverse surgical scenarios will be essential for clinical translation. Ultimately, the goal is to create a fully autonomous, real-time depth sensing platform that seamlessly integrates into robotic surgical workflows, enabling safer and more precise minimally invasive procedures.
AI Executive Summary
Accurate depth perception is a cornerstone of effective minimally invasive surgery, yet traditional structured light systems face significant hurdles in clinical deployment. These systems often rely on multi-shot acquisition, active projectors like digital micromirror devices (DMDs), and precise hardware synchronization, all of which complicate miniaturization and robustness in dynamic surgical environments. Recognizing these limitations, the present study introduces an innovative approach that leverages passive LED illumination paired with a binary mask to achieve single-shot depth reconstruction, suitable for real-time laparoscopic guidance.
The core of this system is a compact projection module integrated into a dual-channel laparoscope. The passive binary mask, illuminated by a simple LED, projects a pattern onto the tissue surface. The same endoscope captures the pattern-distorted image, which is then processed by a deep neural network. Unlike conventional methods, this approach does not require synchronization between projection and imaging, significantly simplifying hardware design. During training, a high-precision Zivid structured-light 3D camera provides reference depth maps, which are reprojected into the endoscopic image frame to supervise the neural network.
The deep learning component employs a vector-quantized variational autoencoder (VQ-VAE) to encode the fringe pattern into a discrete latent space. A custom U-Net operating in this latent space predicts the dense depth map directly, bypassing the need for explicit segmentation or multi-stage processing. This end-to-end pipeline is optimized for speed and accuracy, achieving 26 frames per second on an NVIDIA A100 GPU, making it suitable for real-time surgical applications.
Experimental validation on a dataset of 722 paired phantom images demonstrates the system’s high accuracy, with a mean absolute error of only 3.70mm and an absolute relative error of 0.0326. The system outperforms existing baseline models, including dual U-Net architectures and off-the-shelf monocular depth estimation networks. Its robustness across typical laparoscopic working distances (7-10cm) and stable inference rate highlight its potential for clinical translation.
This work marks a significant advance in intraoperative depth sensing by combining hardware simplicity with deep learning sophistication. The passive binary pattern projection reduces system complexity, while the latent-space regularization ensures stable, high-fidelity depth maps. The approach addresses key challenges in robotic-assisted surgery, such as miniaturization, robustness, and real-time performance, paving the way for smarter, safer minimally invasive procedures. Future efforts will focus on integrating additional modalities, refining calibration procedures, and expanding clinical validation, ultimately aiming to embed this technology into next-generation surgical robots and endoscopic systems.
Deep Dive
Abstract
Significance. Accurate intraoperative depth perception is important for autonomous and semi-autonomous robotic laparoscopic surgery. Conventional fringe projection profilometry can achieve millimeter-scale accuracy but often requires multi-shot acquisition, digital-micromirror-device projection, and projector-camera synchronization, complicating integration into compact laparoscopic systems. Aim. To develop a synchronization-free, single-shot depth-sensing platform using a passive LED-illuminated binary mask and a VQ-VAE prior with a custom U-Net depth head. Approach. A compact projection module was coupled to one channel of a dual-channel laparoscope, while the second channel imaged the fringe-illuminated target. A Zivid 3D camera acquired reference depth for 722 paired phantom images. Zivid depth maps were reprojected into the SSLE image frame for supervised training and evaluation. The VQ-VAE encoded each input into a discrete latent representation, and a latent-space U-Net predicted depth without a separate mask-prediction branch. Results. Using a fixed train/validation/test split, the proposed model achieved an MAE of 3.70 mm, AbsRel of 0.0326, delta=1.1 accuracy of 0.962, and delta=1.1^2 accuracy of 0.970. It achieved lower MAE than the dual U-Net MaskNet + DepthNet baseline and outperformed off-the-shelf monocular depth models in MAE, AbsRel, and threshold accuracy. The pipeline operated at 26.0 Hz over 301 consecutive frames on an NVIDIA A100 GPU. Conclusions. The LED-illuminated binary-pattern platform with latent-space depth reconstruction enables synchronization-free, video-rate endoscopic depth estimation. Results demonstrate Zivid-referenced phantom reconstruction without an explicit segmentation stage, while emphasizing the importance of dataset size and SSLE-Zivid calibration accuracy.
References (20)
Neural Discrete Representation Learning
Aäron van den Oord, O. Vinyals, K. Kavukcuoglu
Deep-learning-based endoscopic single-shot fringe projection profilometry
R. Zuo, Shuwen Wei, Yaning Wang et al.
Recent progresses on real-time 3D shape measurement using digital fringe projection techniques
Song Zhang
Assessment and Calibration of a RGB-D Camera (Kinect v2 Sensor) Towards a Potential Use for Close-Range 3D Modeling
E. Lachat, H. Macher, T. Landes et al.
Demonstration of a laparoscopic structured-illumination three-dimensional imaging system for guiding reconstructive bowel anastomosis
Hanh N. D. Le, Hieu Nguyen, Zhaoyang Wang et al.
Deep point cloud landmark localization for fringe projection profilometry.
Shuwen Wei, M. Kam, Yaning Wang et al.
Image quality assessment: from error visibility to structural similarity
Zhou Wang, A. Bovik, H. Sheikh et al.
Semi-autonomous Robotic Anastomoses of Vaginal Cuffs Using Marker Enhanced 3D Imaging and Path Planning
M. Kam, H. Saeidi, Shuwen Wei et al.
Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
D. Eigen, Christian Puhrsch, R. Fergus
Structured-light 3D surface imaging: a tutorial
J. Geng
Single-shot fringe projection profilometry based on Deep Learning and Computer Graphics
Fanzhou Wang, Chenxing Wang, Qingze Guan
Phase shifting algorithms for fringe projection profilometry: A review
Chao Zuo, Shijie Feng, Lei Huang et al.
Toward real-time remote processing of laparoscopic video
Z. Ronaghi, Edward B. Duffy, D. Kwartowitz
Automatic and real-time tissue sensing for autonomous intestinal anastomosis using hybrid MLP-DC-CNN classifier-based optical coherence tomography
Yaning Wang, Shuwen Wei, R. Zuo et al.
Accurate 3D Shape Reconstruction from Single Structured-Light Image via Fringe-to-Fringe Network
Hieu Nguyen, Zhaoyang Wang
Dense Depth Estimation in Monocular Endoscopy with Self-supervised Learning Methods
Xingtong Liu, Ayushi Sinha, M. Ishii et al.
Adam: A Method for Stochastic Optimization
Diederik P. Kingma, Jimmy Ba
Advanced Geometric Camera Calibration for Machine Vision
Minh Vo, Zhaoyang Wang, Long Luu et al.
Surface reconstruction of abdominal organs using laparoscopic structured light for augmented reality
J. Ackerman, K. Keller, H. Fuchs
Hyper-accurate flexible calibration technique for fringe-projection-based three-dimensional imaging
Minh Vo, Zhaoyang Wang, B. Pan et al.