Decoupled Residual Denoising Diffusion Models for Unified and Data Efficient Image-to-Image Translation

Over the past few years, diffusion models such as DDPM and DDIM have established themselves as powerful generative frameworks, surpassing GANs in image quality and diversity. These models operate by gradually corrupting data with Gaussian noise and then learning to reverse this process, effectively modeling the data distribution. Their success has led to widespread adoption in image synthesis, restoration, and translation tasks. Notable works like SR3, WeatherDiff, and IR-SDE have extended diffusion frameworks to specific applications like super-resolution, weather simulation, and denoising. However, most existing methods focus on single-task scenarios, with limited exploration of multi-task or multi-domain translation. Challenges such as domain gaps, limited paired data, and the erosion of domain-specific features during reverse diffusion remain unresolved. Researchers have attempted to incorporate conditioning, residual prediction, and multi-modal inputs, but these solutions often involve complex training and limited generalization. The need for a unified, data-efficient approach that can handle diverse tasks across multiple domains has driven recent innovations, including the current work’s focus on decoupling diffusion processes.

Despite the impressive progress, current diffusion-based I2I translation methods face significant challenges. The coupled reverse process tends to diminish domain-specific features as noise and residuals are removed simultaneously, leading to poor generalization across tasks and domains. Additionally, reliance on large-scale paired datasets limits applicability in real-world scenarios where such data are scarce. The difficulty of maintaining semantic consistency while achieving domain alignment further complicates model design. Existing methods struggle to balance the preservation of structural details with the flexibility to adapt to multiple tasks, resulting in suboptimal performance, especially under limited data conditions. Addressing these issues requires a paradigm shift that decouples domain alignment from semantic mapping, enabling more robust and data-efficient translation.

The paper’s primary innovation is the explicit decoupling of the diffusion process into two stages: a stochastic noise diffusion that acts as a domain harmonizer, and a deterministic residual diffusion that models semantic mapping within a fixed-noise domain. This approach ensures that the domain alignment effects induced by noise are preserved during the core mapping process, which is not possible in traditional coupled models. The authors provide a rigorous mathematical proof that Gaussian noise reduces the KL divergence between feature distributions across domains, formalizing the domain harmonization effect. During training, the noise diffusion stage leverages abundant unpaired target images, while the residual diffusion network learns residual mappings from limited paired data. The reverse process is similarly decoupled, first removing residuals within the noisy domain, then denoising to produce high-fidelity images. This design simplifies learning, enhances robustness, and broadens applicability across various diffusion algorithms.

• �� The forward process begins with a target image, which undergoes stochastic Gaussian noise injection at each step, controlled by a schedule βt, producing a noise-carrying target image. • The noise diffusion stage is trained on large unpaired target images, optimizing a residual network to predict residuals and a denoising network to estimate noise, using mean squared error losses. • The output of the noise diffusion stage serves as input to the residual diffusion stage, which models the target-to-source mapping by injecting residuals scaled by αt, learned from limited paired data. • During training, residual networks are trained to predict residuals Ires = Iin - I0, minimizing the residual prediction loss, while the denoising network minimizes the noise prediction loss. • The reverse process involves first removing residuals within the fixed-noise domain, then denoising to generate the final image, with each step guided by trained networks. • The entire pipeline is optimized via a combination of residual and noise loss functions, enabling efficient learning with limited paired data and abundant unpaired target images.

The experimental setup includes multiple datasets such as All-in-One-5, CDD-11, and MNMD, covering tasks like image restoration, super-resolution, and multi-domain denoising. The models are trained using Adam optimizer with specific learning rates, and evaluated with metrics including SSIM, LPIPS, and FID. Baselines include DDPM, DDIM, IR-SDE, and recent diffusion models. Ablation studies assess the impact of noise levels, decoupling strategies, and network architectures. The experiments demonstrate that DRDD outperforms existing methods in both quantitative metrics and qualitative visual quality, especially under limited data conditions. Cross-task and cross-domain evaluations confirm its robustness and generalization. Additional tests verify the compatibility of DRDD with various diffusion algorithms and the effect of different noise schedules, providing comprehensive insights into its operational dynamics.

Quantitative results show that DRDD achieves an average SSIM of 0.916 across multiple restoration tasks, surpassing previous methods by approximately 0.02. LPIPS scores are reduced to 0.073, indicating better perceptual quality, while FID scores are consistently lower, reflecting improved realism. In scenarios with limited paired data, DRDD maintains high performance, demonstrating superior data efficiency. Ablation experiments confirm that the noise as a domain harmonizer significantly improves feature alignment, with optimal noise levels identified through grid search. Compatibility tests reveal that DRDD can be integrated with various diffusion frameworks without performance loss. Visual comparisons illustrate clearer details, fewer artifacts, and more consistent domain translation, validating the effectiveness of the decoupling strategy.

The proposed DRDD framework is highly applicable to real-world tasks such as medical image enhancement, remote sensing data correction, and autonomous vehicle perception, where collecting large paired datasets is challenging. Its ability to perform multiple restoration and translation tasks within a single model reduces deployment complexity and resource requirements. The model’s data efficiency makes it suitable for scenarios with scarce annotations, enabling broader adoption in industry. Additionally, the framework can be extended to multi-modal data, including videos and 3D models, with further research. Its robustness and flexibility open avenues for developing versatile AI systems capable of handling diverse image processing challenges in healthcare, security, and environmental monitoring.

Despite its advantages, DRDD faces limitations such as sensitivity to noise level tuning, which affects domain harmonization effectiveness. High noise levels may diminish feature alignment, leading to degraded translation quality. The computational cost remains significant due to multiple diffusion steps, posing challenges for real-time applications. The current focus on static images limits its immediate applicability to dynamic data like videos, where temporal consistency must be addressed. Moreover, the model's performance under extremely large domain gaps or highly complex scenes requires further validation. Future work should aim to optimize efficiency, extend to multi-modal data, and improve robustness under diverse conditions.