Multimodal Image Colorization: Quantifying the Impact of Text-Conditioned Guidance on Grayscale-to-Color Translation
Quantifies text-conditioned impact on grayscale-to-color models (U-Net, Stable Diffusion 1.5); PSNR +5.6%-5.8%, LPIPS -7.6%-11.3%.
Key Findings
Methodology
This study evaluates U-Net and Stable Diffusion 1.5 architectures with and without CLIP-based text conditioning for grayscale-to-color image translation. Using the Open Image Preferences v1 dataset, the models were trained and tested with identical configurations, and performance was measured using PSNR, SSIM, LPIPS, and colorfulness metrics. Experiments were conducted on an NVIDIA DGX-1 cluster.
Key Results
- In U-Net, text conditioning improved PSNR by 5.6% (19.13→20.21), SSIM by 1.2%, LPIPS by -7.6% (0.376→0.348), and colorfulness by 36.6%.
- In Stable Diffusion, text conditioning improved PSNR by 5.8% (19.65→20.79), SSIM by 1.5%, LPIPS by -11.3% (0.236→0.209), but colorfulness increased only by 0.6%.
- Text conditioning had a more pronounced impact on U-Net, especially in colorfulness, highlighting its utility for models without pretrained priors.
Significance
This study provides the first quantitative evaluation of text-conditioned guidance in grayscale image colorization. Results show that text conditioning improves pixel-level metrics (e.g., PSNR, SSIM), perceptual quality (LPIPS), and color vividness, addressing the multimodal nature of the task. The findings are significant for advancing multimodal generative tasks, particularly for models trained from scratch without pretrained priors.
Technical Contribution
The study isolates the effect of text conditioning through controlled ablation experiments, avoiding confounding factors from architectural changes. It demonstrates the utility of CLIP-based cross-attention mechanisms in enhancing colorization quality and validates the approach across two architectures of vastly different scales (U-Net with ~13-16M parameters and Stable Diffusion with ~859M parameters).
Novelty
Unlike prior works that combine text conditioning with other architectural changes, this study isolates text's contribution through controlled experiments. This is the first work to quantify text-conditioned improvements in grayscale-to-color translation across different model scales.
Limitations
- The study uses a single dataset (Open Image Preferences v1), limiting generalizability to other domains.
- U-Net outputs are upscaled from 256×256 to 512×512 for evaluation, potentially affecting results.
- No user studies were conducted; perceptual quality was assessed solely through automated metrics.
Future Work
Future work could explore higher CFG guidance scales to better balance text influence and grayscale fidelity. Evaluating on larger, diverse datasets and conducting user studies would provide stronger evidence of generalizability and perceptual improvements.
AI Executive Summary
Grayscale image colorization is a challenging task in computer vision, with applications in historical photo restoration, medical imaging, and digital art. The task is inherently ill-posed, as grayscale intensity does not uniquely determine color. Traditional methods struggle to generate vivid, semantically accurate colors.
This study investigates the impact of text-conditioned guidance on grayscale-to-color translation using two architectures: U-Net and Stable Diffusion 1.5. By incorporating CLIP-based text embeddings through cross-attention mechanisms, the researchers conducted controlled experiments to isolate the effect of text conditioning. Results show consistent improvements across pixel-level and perceptual metrics, with U-Net showing a 5.6% PSNR increase and a 36.6% boost in colorfulness, while Stable Diffusion achieved an 11.3% LPIPS reduction.
Despite its success, the study acknowledges limitations, including reliance on a single dataset and the absence of user studies. Future research could explore the impact of higher guidance scales, larger datasets, and human evaluations. This work highlights the potential of text-conditioned guidance to address the multimodal challenges of image colorization, particularly for models without pretrained priors.
Deep Analysis
Background
Grayscale image colorization is widely used in historical restoration, medical imaging, and art. Early methods relied on CNNs to predict chrominance channels in CIELAB space, but struggled with multimodality, often producing desaturated or semantically inconsistent results. Recent advances in text-guided models like TIC and DiffColor show promise but lack controlled studies isolating text's impact.
Core Problem
The core challenge is multimodality: a single grayscale image can correspond to many plausible colorizations. Existing methods fail to balance structural fidelity, semantic accuracy, and perceptual quality, especially for ambiguous scenes.
Innovation
This study isolates the effect of text conditioning by holding all other variables constant. It uses CLIP-based cross-attention to integrate text embeddings into U-Net and Stable Diffusion, enabling models to leverage textual cues for improved colorization.
Methodology
- �� Dataset: Open Image Preferences v1 with 7,459 grayscale-color pairs and captions.
- �� Models: U-Net (13.4M params) and Stable Diffusion 1.5 (859M params) with/without text conditioning.
- �� Training: U-Net uses MSE loss; Stable Diffusion employs 𝜖-MSE with Min-SNR weighting.
- �� Evaluation: Metrics include PSNR, SSIM, LPIPS, and colorfulness.
Experiments
Experiments were conducted on an NVIDIA DGX-1 cluster. U-Net models trained at 256×256 resolution, while Stable Diffusion operated at 512×512. Metrics were computed at 512×512 for cross-tier comparison, with U-Net outputs upscaled.
Results
Text conditioning improved all metrics across both architectures. U-Net saw a 5.6% PSNR increase and 36.6% colorfulness boost, while Stable Diffusion achieved an 11.3% LPIPS reduction but limited colorfulness improvement.
Applications
Applications include historical photo restoration, medical imaging, and digital art. The method is particularly useful for tasks requiring semantic consistency and vivid colors.
Limitations & Outlook
The study relies on a single dataset, and U-Net's lower resolution may impact cross-tier comparisons. Additionally, perceptual quality was not validated through user studies.
Plain Language Accessible to non-experts
Imagine you have an old black-and-white photo. Traditional colorization methods are like using a fixed set of rules: trees are green, the sky is blue. But what if you can't tell the color of a car? This study adds a text description, like 'a red car,' to guide the model. With this hint, the model can color the car red instead of guessing randomly.
ELI14 Explained like you're 14
Think of a black-and-white coloring book. You want to color it, but some objects, like a car, could be any color. Now imagine someone whispers, 'This car is red!' That’s what this study does for computers — it teaches them to use text hints to color pictures better. Cool, right? It’s like giving computers a little extra help to make smarter guesses!
Glossary
PSNR (Peak Signal-to-Noise Ratio)
Measures image reconstruction quality; higher values indicate better fidelity.
Used to evaluate pixel-level similarity between generated and ground-truth images.
SSIM (Structural Similarity)
Quantifies structural similarity between images; higher values indicate better structural preservation.
Assesses how well the model preserves image details.
LPIPS (Learned Perceptual Image Patch Similarity)
A deep-learning-based metric for perceptual quality; lower values indicate better quality.
Measures perceptual similarity between generated and reference images.
CLIP
A model trained on image-text pairs to produce aligned visual and textual embeddings.
Used as the text encoder for generating embeddings in this study.
Stable Diffusion
A high-resolution image synthesis framework based on diffusion models.
Serves as the backbone for the SD-tier experiments in this paper.
Open Questions Unanswered questions from this research
- 1 How does text conditioning generalize to larger, more diverse datasets?
- 2 What is the optimal CFG scale for balancing text influence and grayscale fidelity?
Applications
Immediate Applications
Historical Photo Restoration
Use text descriptions to restore realistic colors to historical photos, enhancing their visual and cultural value.
Medical Imaging Enhancement
Add color to grayscale medical scans to improve diagnostic clarity for healthcare professionals.
Long-term Vision
Multimodal Art Generation
Combine text and image generation to drive innovation in digital art and virtual reality applications.
Abstract
Grayscale images are commonly found in historical photography restoration, medical imaging, and artistic media. However, automatically applying color to these images remains a significant challenge in computer vision because many plausible colorizations can correspond to the same grayscale input. In this work, we quantify the effect of text conditioning on pixel-level and perceptual metrics for grayscale-to-color image models. Specifically, we compare two architectures, a U-Net and Stable Diffusion 1.5, each tested with and without CLIP text conditioning while holding all other variables constant. Our results show that text conditioning improves PSNR by 5.6%, SSIM by 1.2%, and colorfulness by 36.6%, while reducing LPIPS by 7.6% in the U-Net tier. In the Stable Diffusion tier, text conditioning improves PSNR by 5.8%, SSIM by 1.5%, and colorfulness by 0.6%, while reducing LPIPS by 11.3%. These results indicate that text conditioning provides consistent, measurable improvements to colorization quality across both architecture scales.