A Closer Look at Deep Policy Gradients

Deep RL has experienced rapid development, with algorithms like DQN, TRPO, and PPO pushing the boundaries of what agents can learn. Early successes in Atari games and robotic tasks demonstrated the potential of policy gradient methods. However, these methods often rely on assumptions such as accurate gradient estimates, well-behaved reward landscapes, and effective value functions. Despite theoretical guarantees, practical training frequently encounters instability, high variance, and poor reproducibility. Recent studies highlight issues like hyperparameter sensitivity and sample inefficiency, prompting researchers to question whether the foundational assumptions hold in complex, high-dimensional environments. This background underscores the importance of understanding the core primitives—gradient estimation, value prediction, and reward landscape behavior—to improve algorithm robustness and theoretical fidelity.

Current deep policy gradient methods depend heavily on noisy gradient estimates, imperfect value functions, and surrogate objectives that may not reflect the true reward landscape. These issues lead to unstable training, suboptimal policies, and poor reproducibility. The core challenge is that the theoretical assumptions—such as accurate gradient estimation and landscape alignment—are often violated in practice, especially in high-dimensional, sample-limited regimes. This disconnect hampers progress toward reliable, scalable deep RL algorithms. Addressing these gaps requires a detailed empirical analysis of the estimation quality, landscape structure, and the interplay between value functions and policy updates, which remains largely unexplored.

The paper introduces a comprehensive evaluation framework that combines gradient correlation metrics, landscape visualization, and value prediction error analysis. It systematically quantifies the divergence between theoretical assumptions and empirical observations in deep RL. By applying this framework to PPO and TRPO across multiple MuJoCo tasks, it uncovers fundamental biases in gradient estimates, limitations of value networks, and misalignment of surrogate objectives with true rewards. This multi-faceted approach provides novel insights into the mechanisms underlying training instability and offers a pathway for designing more robust algorithms that better adhere to their theoretical foundations.

• �� Evaluate gradient estimates by computing pairwise cosine similarity across multiple runs, measuring how well the estimated gradients correlate with a high-fidelity 'true' gradient obtained from large sample sets.
• �� Visualize reward landscapes by sampling policy parameters around current points, comparing the true reward surface with surrogate objectives at different training stages.
• �� Train value networks to fit the expected return, then measure their prediction errors and impact on gradient variance.
• �� Conduct experiments across different sample regimes, from low (2,000 state-action pairs) to high (100,000+), to assess how sample size influences estimation quality and landscape fidelity.
• �� Use ablation studies to compare baseline choices (zero, estimated, true value) in variance reduction and policy performance.
• �� Analyze the evolution of gradient correlation, landscape alignment, and value prediction errors over training epochs to identify key divergence points.

The experimental setup involves training PPO and TRPO agents on MuJoCo environments (Humanoid-v2, Walker2d-v2, Hopper-v2). Data collection includes millions of state-action pairs, with gradient estimates computed via Monte Carlo sampling. The experiments measure gradient correlation with a high-fidelity 'true' gradient, variance reduction from value functions, and reward landscape alignment through visualization. Different sample regimes test the robustness of estimates. Hyperparameters follow standard configurations, with ablation on baseline choices and sample sizes. Results include quantitative metrics like cosine similarity, bias, and landscape deviation, supported by visualizations illustrating the evolution of these metrics over training.

Gradient correlation declines sharply during training, especially in complex tasks, with late-stage correlation below 0.2. Value networks, though fitting training data well, exhibit up to 50% error in true value prediction, limiting their variance reduction capacity. Reward landscape visualizations reveal early alignment between surrogate and true rewards, but this alignment deteriorates as training progresses, leading to policy updates that sometimes decrease true reward. Larger sample regimes improve landscape fidelity but do not fully eliminate the divergence, indicating fundamental limitations in current estimation methods. These results highlight the need for improved estimators and landscape-aware optimization strategies.

Insights from this study inform the design of more stable deep RL algorithms, applicable in robotics, autonomous driving, and game AI. Better understanding of estimation biases and landscape misalignments can guide the development of algorithms with enhanced robustness, reproducibility, and sample efficiency. Practical deployment requires integrating these findings into training protocols, hyperparameter tuning, and evaluation metrics, ultimately enabling safer and more reliable AI systems in real-world scenarios.

The analysis is primarily based on MuJoCo environments, which may not fully capture real-world complexities. The causal sources of estimation bias remain underexplored, and the generalization to other algorithms or environments needs further validation. Computational costs are high due to large sample requirements, and the impact of hyperparameter variations on results warrants deeper investigation. Future work should address these limitations, extend analysis to diverse settings, and develop methods to mitigate identified biases.