A Study on the Evaluation of Generative Models
This study uses a synthetic dataset to compare FID, IS, and classical f-divergences, revealing that divergence-based metrics are more stable for model evaluation.
Key Findings
Methodology
Using a high-quality synthetic dataset generated by Image-GPT trained on ImageNet, the authors compute true likelihoods and estimate KL and reverse KL divergences. These are then compared with empirical metrics like FID and IS across multiple models. The study analyzes correlations and ranking stability, especially focusing on the suitability of features used in FID. Multiple model training runs and metric extensions ensure robustness. The evaluation highlights the volatility of heuristic metrics versus the stability of probabilistic divergences, especially in fine-grained model comparisons.
Key Results
- FID and IS, while correlated with various f-divergences, show significant ranking fluctuations during model training, indicating high volatility. KL and reverse KL exhibit high agreement with a Kendall’s τ of 0.89, outperforming FID and IS. Extended metrics like FID∞ and KID demonstrate higher stability and correlation with divergences. Features from CLIP outperform Inception features on non-ImageNet datasets, confirming their broader applicability.
- Likelihood-based models such as PixelSnail and VD-VAE show strong correlation between their likelihood estimates and classical metrics, validating these as indicators of distribution matching. The instability of FID and IS makes fine-grained model distinctions challenging, emphasizing the need for multiple metrics. The study also finds that FID∞ correlates best with KL and RKL, making it a more reliable evaluation tool.
- Feature analysis reveals that CLIP features are better suited than Inception features for non-ImageNet datasets, especially when assessing models trained on diverse domains. Gaussianity tests show that CLIP features are closer to normal distribution, whereas Inception features are less so, especially on synthetic data. This supports using CLIP for more robust cross-domain evaluation.
Significance
This work advances the understanding of evaluation metrics for generative models, highlighting the limitations of heuristic scores like FID and IS. By establishing a synthetic benchmark with known likelihoods, it provides a rigorous basis for comparing classical probabilistic divergences with empirical metrics. The findings advocate for a multi-metric approach, integrating divergence measures and feature-based scores, to improve model selection and development. The demonstrated superiority of CLIP features on diverse datasets opens new avenues for cross-domain evaluation, crucial for real-world applications in image synthesis, audio, and beyond. Overall, this research addresses a long-standing challenge in generative modeling—reliable, interpretable, and fine-grained evaluation—paving the way for more robust and meaningful progress.
Technical Contribution
The paper introduces a synthetic dataset derived from Image-GPT, enabling exact likelihood computation for model evaluation. It systematically compares classical divergences (KL, RKL) with heuristic metrics (FID, IS), revealing their correlation and volatility. The authors propose using extended metrics like FID∞ and KID, which reduce bias and variance. They also incorporate CLIP features, trained on diverse data, demonstrating their effectiveness over Inception features. The study provides a comprehensive analysis of feature distribution assumptions, validating the Gaussianity hypothesis and proposing alternative feature representations. These contributions enhance the interpretability and robustness of generative model evaluation, bridging the gap between probabilistic and empirical metrics.
Novelty
This is the first comprehensive study leveraging a synthetic dataset with known likelihoods to evaluate the correlation and stability of various generative model metrics. It systematically compares classical divergences with heuristic scores, introduces extended unbiased metrics, and emphasizes the importance of feature choice, especially advocating CLIP over Inception features for cross-domain evaluation. The integration of likelihood-based and feature-based metrics in a unified framework represents a significant step forward, addressing the limitations of prior work that relied solely on heuristic scores or limited datasets.
Limitations
- The evaluation relies on synthetic data generated by Image-GPT, which, despite high quality, may not fully capture the complexity of real-world data distributions. The approach is limited to likelihood-based models, excluding implicit models like GANs. The stability of metrics can vary depending on model architecture and dataset domain, requiring further validation across diverse scenarios. Computational costs are high due to multiple training and feature-fitting procedures. Future work should explore more efficient, domain-agnostic evaluation strategies and extend to implicit models.
- The Gaussianity assumption for features, especially in the context of FID, may not hold universally, impacting the reliability of certain metrics. The dependence on pre-trained features like CLIP and Inception introduces biases related to their training data, which may limit generalization. Further research is needed to develop domain-agnostic, scalable evaluation metrics that can handle complex, multimodal data beyond images.
Future Work
Future directions include integrating multi-modal features (text, audio, video) for comprehensive evaluation, developing unsupervised or semi-supervised metrics to reduce reliance on synthetic data, and exploring feature distribution models beyond Gaussian assumptions. Extending the framework to implicit models like GANs, with new divergence estimation techniques, is also promising. Additionally, automating multi-metric fusion and real-time evaluation could facilitate rapid model development cycles. These efforts aim to establish universally applicable, interpretable, and computationally efficient evaluation standards for the next generation of generative models.
AI Executive Summary
The rapid advancement of deep generative models, such as GANs and diffusion techniques, has revolutionized image synthesis and related fields. However, evaluating their performance remains a persistent challenge. Traditional heuristic metrics like Inception Score (IS) and Fréchet Inception Distance (FID) are widely used but lack clear theoretical grounding and are sensitive to implementation details. This paper addresses this gap by constructing a synthetic dataset, NotImageNet32, generated via Image-GPT trained on ImageNet, which allows for exact likelihood computation. Using this benchmark, the authors compare classical probabilistic divergences—KL and reverse KL—with empirical metrics, revealing that while FID and IS correlate with these divergences, they exhibit high volatility and inconsistent rankings, especially among high-quality models. The study further demonstrates that extended metrics like FID∞ and KID offer more stable and reliable model comparisons.
A key contribution is the evaluation of features used in FID. The authors find that CLIP features, trained on diverse datasets, outperform Inception features, especially on non-ImageNet data, and better satisfy Gaussian assumptions. This insight suggests that feature choice critically impacts evaluation robustness. The findings advocate for a multi-metric evaluation strategy, combining divergence measures with feature-based scores, to improve model selection and development.
Overall, this work provides a rigorous, data-driven framework for assessing generative models, emphasizing the importance of stability, interpretability, and cross-domain applicability. While limitations exist—such as reliance on synthetic data and computational costs—the proposed approach marks a significant step toward more meaningful and reliable evaluation standards, fostering progress in generative modeling research and applications.
Deep Dive
Abstract
Implicit generative models, which do not return likelihood values, such as generative adversarial networks and diffusion models, have become prevalent in recent years. While it is true that these models have shown remarkable results, evaluating their performance is challenging. This issue is of vital importance to push research forward and identify meaningful gains from random noise. Currently, heuristic metrics such as the Inception score (IS) and Frechet Inception Distance (FID) are the most common evaluation metrics, but what they measure is not entirely clear. Additionally, there are questions regarding how meaningful their score actually is. In this work, we study the evaluation metrics of generative models by generating a high-quality synthetic dataset on which we can estimate classical metrics for comparison. Our study shows that while FID and IS do correlate to several f-divergences, their ranking of close models can vary considerably making them problematic when used for fain-grained comparison. We further used this experimental setting to study which evaluation metric best correlates with our probabilistic metrics. Lastly, we look into the base features used for metrics such as FID.