The Cramer Distance as a Solution to Biased Wasserstein Gradients

The evolution of probability distances in machine learning reflects a shift from KL divergence, favored for its simplicity, to Wasserstein distance, appreciated for its geometric interpretability. Early models relied on KL due to ease of optimization via maximum likelihood, but faced issues in capturing perceptual similarities. Wasserstein distance addressed this by considering the underlying geometry, improving sample quality in generative models like WGAN. However, practical deployment revealed a critical flaw: biased gradient estimates from finite samples, leading to unstable training and convergence issues. Recent research has sought to reconcile geometric fidelity with unbiased estimation, but no comprehensive solution existed until now.

The main challenge is that Wasserstein distance, despite its theoretical appeal, produces biased gradients when estimated from finite samples, impairing stochastic gradient descent optimization. This bias causes models to converge to incorrect minima or fail to converge, especially problematic in low-sample or low-entropy distributions common in real-world data. Existing methods either resort to heuristic fixes or compromise on geometric sensitivity, limiting their effectiveness. Addressing this bias without sacrificing geometric properties is crucial for advancing generative modeling and distribution estimation.

This paper introduces the Cramér distance, a probability metric that combines the geometric sensitivity of Wasserstein with the unbiased gradient property of KL divergence. Key innovations include: 1) Theoretical proof that Cramér distance satisfies scale sensitivity, sum invariance, and unbiased gradients. 2) Embedding the distance into a GAN framework with a learned transformation function h, enabling stable training and diverse outputs. 3) Demonstrating through rigorous analysis and experiments that the Cramér distance overcomes Wasserstein's bias issues, leading to more reliable and accurate models. This represents a significant step forward in probability distance design for deep learning.

• �� Analyze the bias in sample Wasserstein gradients, establishing theoretical foundations. • Define the Cramér distance as the L2 norm of the difference between distribution CDFs. • Prove that the distance satisfies key properties: scale sensitivity, sum invariance, and unbiased gradients. • Design a GAN architecture (Cramér GAN) that minimizes the energy distance between transformed samples, using a learned function h. • Incorporate gradient penalties to enforce Lipschitz constraints, ensuring training stability. • Use dual forms of the distance to facilitate efficient optimization. • Validate the approach through theoretical proofs and extensive experiments across tasks, confirming the unbiasedness and performance gains.

• �� Conducted on CelebA dataset, comparing the visual quality, diversity, and training stability of Cramér GAN versus WGAN-GP. • Evaluated image completion, regression (Year Prediction MSD), and distribution estimation metrics. • Varied critic update frequency and batch sizes to analyze robustness. • Measured convergence speed, sample diversity, and quantitative scores such as FID and RMSE. • Performed ablation studies on the transformation function h and regularization techniques to optimize performance. The experiments demonstrate that Cramér GAN achieves higher quality, more diverse samples, and more stable training dynamics.

• �� Cramér GAN outperforms WGAN-GP in image diversity, producing varied face completions with different hairstyles and backgrounds, unlike WGAN-GP’s near-deterministic outputs. • Quantitative metrics show a 15% improvement in FID scores and over 10% reduction in RMSE on regression tasks. • Training curves reveal faster convergence and fewer mode collapse issues. The unbiased gradient property ensures models do not get trapped in poor minima, especially with limited samples. These results confirm the theoretical advantages of the Cramér distance in practical scenarios.

• �� The method is suitable for high-fidelity image synthesis, distributional reinforcement learning, and robust density estimation. • It benefits scenarios with limited data or low-entropy distributions, where traditional Wasserstein-based methods struggle. • Long-term, the approach could enable more reliable generative models in medical imaging, scientific simulations, and AI-driven design, by providing stable, unbiased training signals.

• �� Computational cost of the Cramér distance increases with data dimensionality, requiring further optimization for large-scale applications. • Theoretical guarantees are primarily established for univariate cases; multivariate extensions need more research. • In extremely low-sample regimes, bias may still affect the results, necessitating adaptive algorithms. Future work should focus on scalable algorithms, high-dimensional analysis, and integration with other deep learning frameworks.