Assessing Generative Models via Precision and Recall

Generative models like GANs and VAEs have advanced rapidly, enabling realistic image and text synthesis. Evaluation metrics such as FID and Inception Score are widely used but have limitations: they produce a single score, cannot distinguish between quality and diversity issues, and are sensitive to mode collapse. Recent efforts aim to develop more nuanced metrics, but most lack a theoretical foundation or practical scalability. Accurate, multi-dimensional evaluation remains a key challenge for advancing generative modeling research and applications, especially in real-world scenarios where understanding failure modes is critical.

Current evaluation metrics like FID and IS provide a single scalar value, which cannot disentangle whether poor scores are due to low sample quality or limited diversity. They are insensitive to mode dropping or mode inventing, making it difficult to diagnose specific model failures. This hampers targeted improvements and fair comparisons across models. The core problem is to develop an evaluation framework that can separately quantify the quality of generated samples and their coverage of the target distribution, providing a more detailed and actionable assessment.

The paper introduces a distribution decomposition based on support set intersection, defining separate measures of precision and recall. It employs a parametric trade-off curve (PRD) that captures the entire spectrum of model performance, unlike single-score metrics. The algorithm leverages a λ-parameter and min functions to efficiently compute the PRD set, which relates to total variation distance. This approach is novel in its ability to visualize and quantify the trade-offs between sample fidelity and diversity, applicable to unlabeled, multi-modal data, and scalable to large datasets.

• �� Define support sets of target and generated distributions, decompose into mixture models. • Use support set intersection to guide the separation of distribution components. • Introduce a λ-parameter and compute α(λ), β(λ) via min functions over the distributions. • Construct the PRD curve by plotting (α(λ), β(λ)) for a range of λ values. • Cluster samples in feature space (e.g., using k-means) to handle high-dimensional data. • Calculate the PRD set as the union of segments parameterized by λ, capturing the entire trade-off spectrum. • Relate the metrics to total variation, ensuring theoretical soundness. • Extend to multi-class, multi-modal, and unlabeled scenarios for broad applicability.

The evaluation involved multiple GAN and VAE variants trained on datasets like MNIST, CelebA, and CIFAR-10. Features were extracted using a pre-trained Inception network, then clustered with mini-batch k-means. PRD curves were generated by varying λ, and results were compared with FID and IS. Experiments included mode dropping, mode inventing, class imbalance, and high-dimensional data. Multiple runs ensured robustness, with hyperparameters tuned for stability. The evaluation demonstrated PRD's ability to distinguish different failure modes and quantify trade-offs between quality and coverage.

PRD successfully identified high-quality, low-diversity models (high precision, low recall) and vice versa, aligning with visual assessments. For example, on CIFAR-10, certain GANs showed similar FID scores (~60) but vastly different PRD curves, revealing differences in mode coverage. The metrics correlated weakly with existing scores, confirming their complementary nature. The results also confirmed industry folklore: VAEs tend to produce diverse but blurry samples (high recall, low precision), while GANs produce sharper but mode-collapsed outputs. PRD provides a nuanced, actionable evaluation.

The method can be used in model selection, hyperparameter tuning, and diagnosing failure modes in image, text, and audio generation. It is particularly useful in multi-modal tasks, class imbalance scenarios, and real-world applications requiring detailed performance understanding. The scalable clustering-based implementation allows deployment in large-scale training pipelines, guiding improvements in sample fidelity and diversity. It also facilitates fair benchmarking across different architectures and datasets, promoting standardized evaluation practices.

Dependence on clustering quality and parameter λ introduces subjective bias. In high-dimensional spaces, finite sample effects may bias support set estimates. Computational complexity increases with sample size, limiting scalability without optimization. The method's effectiveness relies on meaningful feature embeddings; poor embeddings can impair accuracy. Future work should address adaptive clustering, theoretical bounds, and real-time evaluation integration.