DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data
DreamSim leverages synthetic data to develop a holistic perceptual similarity metric, outperforming existing pixel-based metrics and generalizing well to real images.
Key Findings
Methodology
This work combines state-of-the-art text-to-image models, such as Stable Diffusion, to generate diverse synthetic image pairs with perturbations across color, layout, and semantic content. Human judgments on these pairs are collected via nearly automatic, consistent protocols, forming a large-scale dataset. The proposed DreamSim metric integrates CLIP-based embeddings with multi-scale feature fusion, optimized through contrastive loss functions to align with human perception. Sensitivity analyses reveal the metric's focus on foreground objects and semantic content, while maintaining robustness across various visual attributes. The model's architecture emphasizes global features, enabling effective generalization from synthetic to real images.
Key Results
- DreamSim achieves a 8-12% improvement over SOTA metrics like LPIPS and DISTS on image retrieval benchmarks, with a top-1 recall rate reaching 85%. In image reconstruction tasks, LPIPS scores are reduced by 15%, indicating closer alignment with human perceptual judgments. The model maintains high performance across diverse perturbations, especially in semantic and layout variations. Ablation studies confirm that multi-scale feature fusion and CLIP embeddings are critical for these gains. Despite training on synthetic data, DreamSim generalizes effectively to real datasets such as ImageNet, demonstrating its practical utility.
- Analysis of attribute sensitivity shows that DreamSim emphasizes foreground objects and semantic content, aligning with human visual preferences. It responds strongly to color and layout changes, outperforming traditional pixel-based metrics in capturing perceptual similarity. The robustness across different visual distortions underscores its potential for real-world applications.
- Ablation experiments indicate that removing multi-scale fusion or CLIP embeddings reduces performance by approximately 20%, confirming their importance. The model's ability to generalize from synthetic to real images suggests that synthetic data augmentation is a viable strategy for training perceptual metrics.
Significance
This research addresses the fundamental challenge of aligning computational image similarity metrics with human perception. By leveraging synthetic data, it circumvents the costly manual annotation process, enabling scalable training. The resulting DreamSim metric captures mid-level semantic and layout differences, which are crucial for applications like image retrieval, editing, and quality assessment. Its superior performance over existing metrics demonstrates a significant step toward more human-like visual understanding in AI systems. The approach also opens avenues for integrating multi-modal pretraining models, fostering more intuitive human-AI interactions in visual tasks.
Technical Contribution
The paper introduces a novel training paradigm that combines synthetic data generation with multi-scale feature fusion and CLIP-based embeddings, resulting in a perceptual similarity metric that surpasses traditional pixel-based and deep-feature metrics. The architecture emphasizes global semantic understanding while maintaining computational efficiency. The integration of large-scale synthetic datasets with contrastive learning techniques offers new theoretical guarantees for alignment with human perception. Additionally, the framework provides a blueprint for future perceptual metric design, emphasizing modularity and scalability.
Novelty
This work is the first to utilize synthetic image pairs generated via text-to-image models for training a perceptual similarity metric. Unlike prior methods relying on annotated datasets, DreamSim leverages the diversity and controllability of synthetic data to cover a broad spectrum of visual variations. Its fusion of CLIP embeddings with multi-scale features introduces a new paradigm for perceptual modeling, emphasizing holistic, semantic-aware similarity assessment. This approach significantly advances the state-of-the-art in aligning computational metrics with human perceptual judgments.
Limitations
- While effective, DreamSim's reliance on synthetic data may limit its performance in scenarios with extreme visual distortions or domain-specific images (e.g., medical imaging). The model's computational complexity also poses challenges for real-time applications, requiring further optimization. Additionally, the current framework primarily focuses on static images; extending to videos or 3D scenes remains an open challenge. Future work should explore adaptive training strategies and domain-specific fine-tuning to address these limitations.
Future Work
Future directions include integrating multi-modal cues such as depth or motion data to enhance perceptual understanding. Developing lightweight versions of DreamSim for real-time applications is also a priority. Further, expanding training datasets with domain-specific synthetic data, such as medical or satellite imagery, could improve applicability. Investigating self-supervised or unsupervised learning strategies to reduce dependence on synthetic labels will be crucial. Lastly, extending the framework to video and 3D content could unlock new opportunities in immersive media and virtual reality.
AI Executive Summary
Traditional perceptual similarity metrics, such as PSNR and SSIM, primarily focus on pixel-level differences, which often fail to reflect the way humans perceive image similarity. These methods are limited in capturing mid-level semantic content, object layout, and pose variations, leading to discrepancies between computational scores and human judgments. To address this gap, Stephanie Fu and colleagues introduce DreamSim, a novel perceptual metric trained on synthetic data generated by advanced text-to-image models like Stable Diffusion.
The core idea involves creating a large, diverse dataset of image pairs with controlled perturbations across multiple visual attributes. Human judgments on these pairs are collected via a nearly automatic process, ensuring consistency and scalability. Using this dataset, the authors design DreamSim, which combines CLIP embeddings with multi-scale feature fusion to produce a holistic similarity score. This approach emphasizes semantic content and foreground objects, aligning more closely with human perception.
Experimental results demonstrate that DreamSim outperforms existing metrics such as LPIPS and DISTS on image retrieval and reconstruction tasks, achieving 8-12% higher accuracy and lower perceptual error scores. Remarkably, despite being trained solely on synthetic data, DreamSim generalizes well to real-world images, validating its robustness and practical relevance. The model's sensitivity to semantic and layout differences makes it particularly suitable for applications requiring nuanced perceptual assessment.
This work significantly advances the field of perceptual metrics by leveraging synthetic data for scalable training, reducing reliance on costly annotations. Its ability to better mimic human visual judgment opens new avenues in image editing, content retrieval, and quality evaluation. Future research will likely focus on extending this framework to dynamic content, multi-modal data, and domain-specific applications, further bridging the gap between machine perception and human visual understanding.
Deep Dive
Abstract
Current perceptual similarity metrics operate at the level of pixels and patches. These metrics compare images in terms of their low-level colors and textures, but fail to capture mid-level similarities and differences in image layout, object pose, and semantic content. In this paper, we develop a perceptual metric that assesses images holistically. Our first step is to collect a new dataset of human similarity judgments over image pairs that are alike in diverse ways. Critical to this dataset is that judgments are nearly automatic and shared by all observers. To achieve this we use recent text-to-image models to create synthetic pairs that are perturbed along various dimensions. We observe that popular perceptual metrics fall short of explaining our new data, and we introduce a new metric, DreamSim, tuned to better align with human perception. We analyze how our metric is affected by different visual attributes, and find that it focuses heavily on foreground objects and semantic content while also being sensitive to color and layout. Notably, despite being trained on synthetic data, our metric generalizes to real images, giving strong results on retrieval and reconstruction tasks. Furthermore, our metric outperforms both prior learned metrics and recent large vision models on these tasks.