Optimal Transport Audio Distance with Learned Riemannian Ground Metrics

The evolution of deep generative models for audio, including speech synthesis and music generation, has driven the development of evaluation metrics like Inception Score, FID, and FAD. While these metrics measure distributional similarity, they often lack sensitivity to micro-level artifacts. FAD, specifically designed for audio, combines embedding-based distances with Gaussian assumptions, but suffers from invariance issues and spectrum-dependent attenuation, limiting its ability to detect rare defects. Recent work introduced kernel-based distances like KAD to relax Gaussian assumptions, yet they still struggle with fine-grained detection. The need for metrics that can precisely identify subtle quality issues and provide sample-level diagnostics remains unmet. This paper builds on the 2-Wasserstein framework, analyzing the limitations of existing metrics and proposing geometric and algorithmic corrections to enhance sensitivity and interpretability.

Current audio evaluation metrics, especially FAD, are hindered by invariance properties of fixed embeddings and the Gaussian coupling assumption, which dilute the sensitivity to rare but impactful artifacts. These limitations prevent effective detection of mode collapse, subtle distortions, or anomalies critical in high-fidelity audio synthesis. The core challenge is to develop a metric that maintains distributional comparison while being sensitive to micro-level defects, capable of providing interpretable, sample-specific diagnostics. Overcoming the invariance and spectrum-dependent attenuation inherent in existing methods is essential for reliable quality assessment in real-world applications, such as speech synthesis, music production, and multimedia generation.

The paper introduces a residual Riemannian ground-metric adapter learned via deep neural networks, which dynamically reshapes the embedding space to expose subtle artifacts. This correction addresses the invariance problem by adjusting local geometry, making the cost function sensitive to micro-level differences. Simultaneously, replacing Gaussian coupling with entropy-regularized Sinkhorn OT enhances the detection of rank-1 contamination, which models mode collapse or extreme distortions. The combined approach results in OTAD, a metric that is both geometrically adaptive and computationally efficient, providing detailed, sample-level diagnostics. This systematic correction surpasses prior fixed-cost and Gaussian assumptions, representing a significant leap in perceptual audio evaluation.

• �� Establish the 2-Wasserstein distance framework, identifying the limitations of FAD’s cost and coupling primitives.
• �� Develop a residual deep neural network gθ(z)=z+fθ(z) to learn a local Riemannian metric Mθ(z)=Jθ(z)⊤Jθ(z), where Jθ(z) is the Jacobian of gθ.
• �� Use the Jacobian determinant |det Jθ(z)|−1 for local density reweighting, compensating for geometric distortions.
• �� Train the adapter with a contrastive triplet loss (Lctr) to ensure cost correction without coupling bias, and fine-tune with Sinkhorn divergence (Lnative) for perceptual alignment.
• �� Replace Gaussian coupling with entropy-regularized Sinkhorn OT, computing transport plans T_{ij} for each sample pair.
• �� Derive per-sample transport costs c_j, enabling anomaly detection and interpretability.
• �� Validate the approach on FSD50K and ESC-50 datasets, across multiple encoders, with contamination experiments and human correlation analysis.

• �� Training adapters on FSD50K, evaluating on ESC-50 with five encoders (d=128, 512, 768, 2048), testing various contamination levels (ε=0.005–0.2).
• �� Comparing FAD, KAD, Sinkhorn, and exact OT in detecting rank-1 contamination, focusing on sensitivity and spectrum effects.
• �� Using multiple evaluation axes—recall, semantic, precision, structure—to assess micro-level detection.
• �� Analyzing sample transport costs (c_j) for anomaly detection, validated via AUROC and separation ratios.
• �� Conducting factorial experiments to isolate effects of cost correction, measure correction, and their interaction, across different encoder dimensions.

• �� OTAD exhibits 1.9 to 3.6 times higher sensitivity to rank-1 contamination than FAD at ε=0.05, confirming its improved defect detection.
• �� On DCASE 2023, OTAD’s correlation with human MOS surpasses FAD and KAD, with mean Spearman coefficients increased by 0.1-0.2.
• �� Sample-level diagnostics achieve AUROC ≥ 0.86, effectively identifying contaminated samples across various artifact types, enabling precise defect localization.
• �� Factor analysis shows cost correction dominates at low dimensions; spectrum-dependent attenuation explains FAD’s insensitivity to micro artifacts; OTAD’s geometric adaptation overcomes this.

• �� Supports automatic, fine-grained quality assessment in speech synthesis, music generation, and multimedia content creation.
• �� Enables model developers to perform detailed diagnostics, identify failure modes, and improve training strategies.
• �� Long-term, facilitates real-time monitoring, multi-modal evaluation, and large-scale quality control in AI-generated content pipelines.

• �� Dependence on large annotated datasets for training adapters; generalization to unseen domains remains challenging.
• �� High computational complexity limits real-time deployment, especially in large-scale or low-resource environments.
• �� The learned geometric corrections may overfit in complex, high-dimensional scenarios, requiring further regularization or unsupervised approaches.