EndoPBR: Material and Lighting Estimation for Photorealistic Surgical Simulations via Physically-based Rendering
EndoPBR uses physically-based differentiable rendering to estimate materials and lighting, enabling photorealistic novel view synthesis in surgical scenes.
Key Findings
Methodology
This approach employs a differentiable rendering framework that explicitly disentangles scene materials and lighting from endoscopic images with known geometry. It models lighting as a simple spotlight with learnable parameters and predicts material BRDFs via neural networks. By grounding predictions in the rendering equation, it synthesizes photorealistic images from arbitrary viewpoints. The model incorporates domain-specific priors, such as the absence of ambient light and co-located light source and camera, to stabilize training. Evaluation on C3VD shows competitive novel view synthesis performance compared to NeRF and 3D Gaussian Splatting, with PSNR of 30.39 and SSIM of 0.86.
Key Results
- In novel view synthesis, EndoPBR outperforms NeRF with PSNR of 30.39 versus 18.93, and SSIM of 0.86 versus 0.67, demonstrating high fidelity. It also produces stable results across different sequences. Synthetic data generated by EndoPBR, when used to fine-tune depth models like Depth Anything V2, achieves similar accuracy to models fine-tuned with real images, with less than 5% difference in error metrics.
- The approach effectively handles non-Lambertian surfaces and non-stationary lighting, addressing key challenges in surgical scene reconstruction. The explicit physical modeling enhances robustness and generalization, enabling realistic virtual scene generation for training and navigation.
- The ability to generate large-scale photorealistic datasets and improve downstream tasks like depth estimation highlights the method’s potential for advancing medical AI, virtual training, and robotic surgery.
Significance
This work addresses longstanding issues in surgical 3D reconstruction caused by complex lighting and surface properties, providing a physics-informed, learnable framework for material and light estimation. The generated photorealistic views facilitate better surgical navigation, robot perception, and training, reducing reliance on scarce labeled data. By enabling synthetic data to improve depth and scene understanding, it paves the way for more robust, scalable medical AI solutions. The explicit modeling approach offers a new paradigm for scene understanding in challenging environments, with broad implications for virtual surgery, diagnostics, and simulation.
Technical Contribution
The paper introduces a novel combination of simplified spotlight models and neural BRDF prediction within a differentiable rendering pipeline, explicitly decoupling scene material and lighting parameters. This approach overcomes NeRF’s limitations under non-stationary lighting, providing stable training and high-quality synthesis. It leverages physical rendering principles, domain-specific priors, and neural network optimization to produce realistic images, enabling large-scale synthetic dataset generation for downstream tasks. The method’s modular design allows extension to complex lighting and dynamic scenes, representing a significant advance in physics-based neural rendering for medical applications.
Novelty
This is the first work to explicitly model and disentangle scene materials and lighting in surgical endoscopy using neural BRDFs and a simplified spotlight model. Unlike prior NeRF-based methods assuming static lighting, it incorporates domain-specific priors to handle non-stationary illumination. The explicit physical modeling and scene priors enable robust, photorealistic view synthesis in challenging environments, marking a new direction in medical scene reconstruction.
Limitations
- The model assumes a single, co-located moving spotlight, limiting its applicability in scenes with multiple or complex light sources. Handling dynamic or multi-source lighting remains a challenge.
- High specular reflections and complex surface interactions are not fully captured, leading to artifacts like black edges in some renderings. Improving reflectance models is necessary.
- Training requires accurate geometry and camera pose information, which may not always be available or easy to obtain in real-world scenarios, restricting practical deployment.
Future Work
Future efforts will focus on extending the model to handle multiple, dynamic light sources, and complex scene geometries. Incorporating unsupervised or self-supervised learning strategies could reduce dependence on known geometry. Additionally, integrating temporal consistency for video sequences and exploring real-time applications in surgical navigation are promising directions.
AI Executive Summary
Reconstructing 3D scenes in surgical environments remains a formidable challenge due to complex, non-stationary lighting and non-Lambertian surfaces. Traditional neural rendering methods like NeRF excel in natural scenes but struggle under these conditions, limiting their utility in medical applications. To address this, the authors propose EndoPBR, a physics-informed, differentiable rendering framework designed specifically for endoscopic scenes. The core innovation lies in explicitly modeling scene lighting as a simple, learnable spotlight and materials via neural BRDFs, grounded in the rendering equation. This explicit disentangling allows the system to generate highly photorealistic images from arbitrary viewpoints, even in challenging lighting conditions.
The approach leverages domain-specific priors—such as the co-location of light source and camera, and the absence of ambient light—to stabilize training and improve robustness. Extensive evaluation on the Colonoscopy 3D Video Dataset demonstrates that EndoPBR achieves competitive results in novel view synthesis, with PSNR and SSIM surpassing many existing methods. Importantly, the authors show that synthetic images generated by their model can be used to fine-tune depth estimation models, achieving performance comparable to training on real images. This validates the potential of physics-based synthetic data for medical AI, especially in scenarios where annotated datasets are scarce.
Overall, EndoPBR offers a significant step forward in realistic scene synthesis for surgical applications. Its explicit physical modeling enhances robustness and generalization, paving the way for improved surgical navigation, training, and robotic assistance. While current limitations include assumptions of single light sources and simplified reflectance models, future work aims to incorporate more complex lighting and dynamic scenes. This research opens new avenues for scalable, physics-based virtual environments in medicine, promising safer, more effective minimally invasive surgeries and training tools in the near future.
Deep Analysis
Background
医学影像中的三维重建技术经历了从传统几何匹配到深度学习的快速发展。NeRF等神经辐射场算法在自然场景中表现出色，但在手术环境中受限于非定常光照和非朗伯表面特性。合成数据和自监督方法虽取得一定进展，但缺乏逼真度和一致性。近年来，结合物理渲染与神经网络的研究逐渐兴起，旨在解决手术场景中的特殊挑战，为虚拟仿真和导航提供更真实的场景模型。
Core Problem
手术场景中存在复杂的非定常光照和非朗伯反射，导致传统3D重建方法难以实现逼真、稳定的虚拟视图。缺乏大规模标注数据限制了深度学习模型的泛化能力。如何在复杂光照环境中准确估算材料和光照参数，成为当前亟待突破的核心问题。这不仅影响虚拟仿真效果，也制约了手术导航和机器人感知的精度。
Innovation
本文提出EndoPBR，结合简化的聚光灯模型和神经网络预测的BRDF参数，显式解耦材料与光照。引入手术场景特有的先验约束（如光源与相机共位、无环境光），增强模型稳定性。利用可微渲染技术，生成逼真图像，突破NeRF在非定常光照环境下的限制。创新点在于结合物理模型与深度学习，提供一种可扩展、鲁棒的虚拟场景重建方案，极大提升了手术场景的逼真度和实用性。
Methodology
- �� 输入端镜图像和已知几何信息，反投影计算每个像素对应的3D点。
- �� 利用神经网络（MLP）预测每个点的BRDF参数（基色、粗糙度、金属度）和光照参数（聚光灯强度、方向）。
- �� 采用简化的渲染方程，将材料和光照参数结合，计算每个像素的颜色值。
- �� 通过最小化预测图像与真实图像的L1损失，优化模型参数。
- �� 引入手术场景特有的先验（光源移动、无环境光），确保训练稳定。
- �� 利用合成数据进行深度模型微调，验证虚拟数据在实际中的应用潜力。
Experiments
在C3VD数据集上，进行新视角合成任务，比较PSNR、SSIM和LPIPS指标，验证模型逼真度。通过不同光照、材质和视角变换，测试模型鲁棒性。用不同方法（NeRF、REIM-NeRF）进行对比，分析性能差异。微调深度模型（Depth Anything V2）以验证合成数据的实用性。设置合理超参数（如学习率1e-4、批量5万像素），训练1500轮，确保模型收敛。
Results
EndoPBR在新视角合成中，PSNR达30.39，优于NeRF的18.93，SSIM为0.86，优于NeRF的0.67，LPIPS为0.30，表现出良好的逼真度。模型在不同序列中表现稳定，黑色边界问题较少。利用合成图像微调深度模型，误差指标与用真实图像微调相差无几，验证虚拟数据的有效性。这表明物理基础的渲染能在复杂场景中实现高质量合成。
Applications
该技术可广泛应用于手术导航、机器人感知、虚拟培训等场景，为手术模拟提供逼真数据。模型对几何和光照的显式解耦，为个性化手术方案和数据增强提供技术支撑。未来可扩展多光源、多场景，推动医学AI的智能化发展，提升手术安全性和效率。
Limitations & Outlook
模型假设单一移动聚光灯，未考虑多光源或复杂照明环境，可能在极端光照条件下表现不足。对高光反射和非朗伯表面处理仍有限，黑色边界现象提示模型在复杂反射条件下存在不足。训练依赖已知几何和相机姿态，实际应用中需精确校准，限制了模型的普适性。未来需增强模型的多光源适应性和无监督学习能力。
Plain Language Accessible to non-experts
想象你在厨房做饭，灯光是厨房的灯泡，食材代表材料。每次你看菜的颜色和光线反射，都受到灯光亮度和菜品材质的影响。如果你想让别人也觉得菜色一样漂亮，就得知道灯光有多亮，菜的反光有多强。EndoPBR就像个聪明的厨师，能根据菜的照片，推算出灯光和菜的反光特性，然后用这些信息，从不同角度重新展示菜肴，让人觉得是真的在厨房里看菜一样。它用科学模拟光和材料的互动，确保每个角度都逼真，帮医生更好地了解手术场景。
ELI14 Explained like you're 14
想象你在玩一个超级逼真的3D游戏，但游戏里的灯光和物体反光都很真实。开发者用了一种特别的技术，让虚拟世界的光线和材质可以像真实一样变化。比如，灯可以移动，反光可以变强或变弱。这个技术叫EndoPBR，它能根据手术视频里的图片，猜出手术器械和组织的材质，还能知道灯光的方向和强度。这样，医生在虚拟环境里可以从不同角度看到手术场景，像真的一样。这不仅让虚拟手术更逼真，还能帮医生练习，甚至训练机器人做手术。就像给虚拟世界装上了“眼睛”和“感官”，让它变得更聪明、更逼真。
Glossary
Neural Radiance Fields (NeRF) (神经辐射场)
一种用神经网络表示3D场景的方法，通过体积渲染实现新视角合成。
本文用以对比传统神经场景重建技术的局限性。
Bidirectional Reflectance Distribution Function (BRDF) (双向反射分布函数)
描述材料如何反射光线的函数，决定材质的光学特性。
用神经网络预测手术场景中的材料反射特性。
Physically-based Rendering (PBR) (物理基础渲染)
基于光学和物理原理模拟光线与材质交互的渲染技术。
实现逼真图像生成的核心技术。
Colonoscopy 3D Video Dataset (C3VD) (结肠镜3D视频数据集)
包含手术视频、深度和相机参数的公开数据集，用于评估虚拟重建方法。
模型在该数据集上进行性能验证。
Differentiable Rendering (可微渲染)
允许通过梯度优化调整场景参数的渲染技术。
实现材料和光照参数的端到端学习。
Open Questions Unanswered questions from this research
- 1 如何在多光源、多光照变化环境中保持模型鲁棒性仍待研究，现有模型多假设单一光源，限制了实际应用的复杂性。
- 2 模型对高光反射和非朗伯表面处理能力有限，未来需改进反射模型以应对更复杂的材质。
Abstract
The lack of labeled datasets in 3D vision for surgical scenes inhibits the development of robust 3D reconstruction algorithms in the medical domain. Despite the popularity of Neural Radiance Fields and 3D Gaussian Splatting in the general computer vision community, these systems have yet to find consistent success in surgical scenes due to challenges such as non-stationary lighting and non-Lambertian surfaces. As a result, the need for labeled surgical datasets continues to grow. In this work, we introduce a differentiable rendering framework for material and lighting estimation from endoscopic images and known geometry. Compared to previous approaches that model lighting and material jointly as radiance, we explicitly disentangle these scene properties for robust and photorealistic novel view synthesis. To disambiguate the training process, we formulate domain-specific properties inherent in surgical scenes. Specifically, we model the scene lighting as a simple spotlight and material properties as a bidirectional reflectance distribution function, parameterized by a neural network. By grounding color predictions in the rendering equation, we can generate photorealistic images at arbitrary camera poses. We evaluate our method with various sequences from the Colonoscopy 3D Video Dataset and show that our method produces competitive novel view synthesis results compared with other approaches. Furthermore, we demonstrate that synthetic data can be used to develop 3D vision algorithms by finetuning a depth estimation model with our rendered outputs. Overall, we see that the depth estimation performance is on par with fine-tuning with the original real images.