Policy Optimization as Wasserstein Gradient Flows

The evolution of RL has seen significant milestones, from early value-based methods to policy gradient techniques like REINFORCE, TRPO, and PPO. Deep RL further advanced the field with neural network function approximators, enabling complex task solving such as Atari games and robotic control. However, these methods primarily optimize parameters directly, lacking a geometric understanding of the policy distribution's evolution. Recent developments in optimal transport and Wasserstein geometry have provided new insights into probability measure spaces, inspiring approaches that model distributional changes more naturally. Despite these advances, integrating Wasserstein geometry into RL remains underexplored, especially in a way that guarantees convergence and scalability. This paper leverages the mathematical framework of Wasserstein gradient flows to address these gaps, proposing a novel, principled approach to policy optimization grounded in measure-theoretic geometry.

Existing RL algorithms often suffer from slow convergence, limited exploration, and poor scalability in high-dimensional spaces due to their reliance on parameter-space optimization. They lack a unified geometric perspective that captures the evolution of entire policy distributions, which could improve exploration and stability. Moreover, the absence of theoretical guarantees for convergence in complex environments hampers their reliability. The core challenge is to formulate a scalable, convergent algorithm that models the policy as a distribution evolving along a gradient flow in the Wasserstein space, providing both interpretability and robustness. Achieving this requires overcoming computational hurdles associated with infinite-dimensional measure spaces and designing effective discretization schemes.

The paper's key innovation is modeling RL policy optimization as a Wasserstein gradient flow, enabling the use of optimal transport geometry to guide policy evolution. It introduces a particle approximation scheme combined with the JKO method, transforming the infinite-dimensional distribution optimization into a sequence of finite-dimensional problems. This approach unifies and generalizes existing RL algorithms within a geometric framework, providing convergence guarantees under convex energy functionals. It also extends the scope of policy representation to complex, multimodal distributions via energy-based models, surpassing traditional parametric methods. The framework's flexibility allows for both indirect (parameter distribution) and direct (policy distribution) optimization, broadening its applicability across RL tasks.

• �� Define the policy as a probability distribution over actions conditioned on states, with an associated energy functional representing the expected reward.
• �� Construct the Wasserstein metric space on probability measures, establishing a geometric structure.
• �� Formulate the policy update as a gradient flow driven by the energy functional, following the PDE \(\partial_ au \mu_ au =

abla \cdot (\mu_ au

abla \delta F / \delta \mu_ au)\).

• �� Discretize the flow using the JKO scheme: \(\mu_{k+1} = \arg\min_\mu F(\mu) + rac{1}{2h} W_2^2(\mu, \mu_k)\), where \(W_2\) is the Wasserstein distance.
• �� Approximate distributions with particles, updating their positions via gradient descent that involves KL divergence and Wasserstein distance gradients.
• �� Implement neural networks to approximate the energy functional and policy, enabling end-to-end training in deep RL settings.

Experiments involved continuous control tasks in MuJoCo (Walker2d, Humanoid) and Atari environments, comparing the proposed WGF algorithms with TRPO and PPO. Performance metrics included average reward, convergence speed, and policy diversity. Hyperparameters such as particle number, step size, and energy functional parameters were tuned systematically. Ablation studies examined the impact of particle count and functional choices. Results showed consistent improvements in reward (up to 12%), faster convergence (30% reduction in training steps), and more diverse policies, validating the theoretical advantages of the geometric approach.

The particle-based WGF method achieved an average reward increase of 12% over TRPO in MuJoCo tasks, with convergence 30% faster. In Atari games, it outperformed DQN by about 15% in sample efficiency. Ablation studies confirmed that increasing particles improved performance but raised computational costs. Proper energy functional design significantly affected convergence speed and policy quality, with convex functionals ensuring stability and optimality.

This framework is applicable to robotics, autonomous driving, game AI, and other complex decision-making tasks involving high-dimensional continuous actions. Its ability to model rich, multimodal policy distributions makes it suitable for real-world scenarios requiring exploration and robustness. The particle approximation approach also facilitates scalable implementation with neural networks, enabling deployment in large-scale, real-time systems.

High computational cost due to particle management in high-dimensional spaces limits real-time applications. The choice of energy functional heavily influences convergence; poor design can lead to suboptimal policies. Non-convex landscapes may cause local minima, reducing optimality. Further research is needed to develop adaptive schemes, reduce complexity, and extend theoretical guarantees to broader classes of problems.