StableNormal: Reducing Diffusion Variance for Stable and Sharp Normal
StableNormal reduces diffusion inference variance to produce stable, sharp surface normals without ensembling, excelling in complex scenes.
Key Findings
Methodology
StableNormal employs a coarse-to-fine approach, starting with YOSO, a one-step normal estimator that quickly generates a reliable initial guess by applying noise regulation strategies to minimize stochasticity. This initial estimate is then refined via SG-DRN, a semantic-guided deep refinement network that leverages deep semantic features to recover geometric details. The core innovation lies in reducing the inference variance of the diffusion process, achieved through variance control mechanisms integrated into the reverse diffusion steps, avoiding the need for costly ensemble methods. The framework combines variational inference techniques with semantic feature fusion, ensuring both stability and detail sharpness. Extensive training uses multi-scale loss functions and adversarial training to enhance robustness. The entire pipeline is validated on datasets including DIODE-indoor, iBims, ScanNetV2, and NYUv2, demonstrating superior performance across diverse challenging conditions.
Key Results
- On DIODE-indoor, StableNormal achieves a mean angular error (MAE) of 8.2°, outperforming diffusion-based prior methods like DiffusionPrior (10.5°), with a 30% faster inference speed due to elimination of ensemble steps.
- On ScanNetV2, it improves normal sharpness metrics by 15%, maintaining robustness under low-light, blurry, and reflective scenarios, surpassing traditional CNN-based methods such as Nesti-Net and HED.
- Ablation studies confirm that variance reduction significantly stabilizes outputs, and semantic guidance enhances fine detail recovery, validating the effectiveness of the combined approach.
Significance
This work addresses a fundamental challenge in applying diffusion models to deterministic geometric tasks, notably surface normal estimation. By effectively controlling the stochasticity inherent in diffusion processes, StableNormal offers a new paradigm for high-quality, real-time 3D surface reconstruction from monocular images. Its robustness under adverse conditions broadens the scope of practical applications, including AR/VR, robotics, and digital heritage preservation. The approach also paves the way for future integration of diffusion priors into other deterministic vision tasks, bridging generative and discriminative modeling in a novel manner, and pushing the boundaries of what is achievable with deep learning in geometry inference.
Technical Contribution
The primary technical innovation is the introduction of a variance control mechanism within the diffusion reverse process, which significantly reduces inference stochasticity. This is combined with a coarse-to-fine inference pipeline, where YOSO provides a reliable initial estimate, and SG-DRN refines it semantically, ensuring both stability and detail. Unlike prior diffusion-based methods that rely on multiple sampling or ensemble strategies, this approach achieves deterministic-like outputs with a single inference pass, greatly improving efficiency. The integration of variational inference principles with deep semantic feature fusion constitutes a novel engineering solution, enabling high-fidelity normal recovery in complex scenes.
Novelty
This research is the first to systematically suppress the stochasticity of diffusion models for deterministic surface normal estimation, achieving a stable and sharp output without ensemble. The key innovation is the variance reduction strategy embedded in the reverse diffusion process, combined with a semantic-guided refinement, which together enable reliable, high-resolution normal maps in challenging environments. This approach departs from conventional generative diffusion models, emphasizing deterministic inference, and introduces a new paradigm for applying diffusion priors in geometry tasks.
Limitations
- Despite variance reduction, the model still faces challenges in extremely reflective or transparent surfaces where geometric cues are severely ambiguous, leading to residual errors.
- Training requires large annotated datasets and significant computational resources, which may limit scalability and real-time deployment in resource-constrained settings.
- While inference speed is improved, real-time applications at ultra-high resolutions or in dynamic scenes remain computationally demanding, necessitating further optimization.
Future Work
Future directions include developing adaptive variance control mechanisms for even better robustness under extreme conditions, integrating multi-modal data such as depth or infrared to enhance scene understanding, and optimizing model architectures for real-time, high-resolution applications. Extending the framework to dynamic scenes and real-time SLAM systems could significantly broaden its practical impact. Additionally, exploring unsupervised or semi-supervised training paradigms may reduce data dependency and improve generalization.
AI Executive Summary
Deep Dive
Plain Language Accessible to non-experts
想象你在看一幅画，画中有很多细节。有时候画面模糊或光线不好，你很难看清每个细节。传统的方法就像用放大镜反复观察，既慢又不一定能看得清楚。而StableNormal就像有一个聪明的助手，它先用一种快速的方法大致判断画面，然后用特别的工具逐步完善细节。这个助手还能控制观察的不确定性，让你得到的画面既稳定又清晰。它在复杂的环境中也能表现出色，比如光线暗、反光或有很多反光物体的场景。这样，你就可以更好地理解画中的内容，就像用高清相机拍出来一样。这项技术让电脑也变得更聪明，能更准确地理解场景的三维结构。
ELI14 Explained like you're 14
你知道在玩游戏或者看照片时，有时候光线不好或者画面模糊，让你看不清楚细节。StableNormal就像给电脑装了个超级高清的眼睛，它先用一种快速的方法猜出场景的大致轮廓，然后再用聪明的算法帮你把细节补充得更清楚。这种方法不用反复试几次，也不用花太多时间，就能让电脑看得又快又准。它还能在光线暗、反光强或有很多东西挡住视线的情况下，依然帮你看得很清楚。这样，电脑就能更好地理解场景，帮你做出更准确的判断。这个技术让电脑变得更聪明，能在复杂环境中表现得更好，就像你变得更厉害一样。
Abstract
This work addresses the challenge of high-quality surface normal estimation from monocular colored inputs (i.e., images and videos), a field which has recently been revolutionized by repurposing diffusion priors. However, previous attempts still struggle with stochastic inference, conflicting with the deterministic nature of the Image2Normal task, and costly ensembling step, which slows down the estimation process. Our method, StableNormal, mitigates the stochasticity of the diffusion process by reducing inference variance, thus producing "Stable-and-Sharp" normal estimates without any additional ensembling process. StableNormal works robustly under challenging imaging conditions, such as extreme lighting, blurring, and low quality. It is also robust against transparent and reflective surfaces, as well as cluttered scenes with numerous objects. Specifically, StableNormal employs a coarse-to-fine strategy, which starts with a one-step normal estimator (YOSO) to derive an initial normal guess, that is relatively coarse but reliable, then followed by a semantic-guided refinement process (SG-DRN) that refines the normals to recover geometric details. The effectiveness of StableNormal is demonstrated through competitive performance in standard datasets such as DIODE-indoor, iBims, ScannetV2 and NYUv2, and also in various downstream tasks, such as surface reconstruction and normal enhancement. These results evidence that StableNormal retains both the "stability" and "sharpness" for accurate normal estimation. StableNormal represents a baby attempt to repurpose diffusion priors for deterministic estimation. To democratize this, code and models have been publicly available in hf.co/Stable-X