Distributional Offline Policy Evaluation with Predictive Error Guarantees
FLE algorithm uses MLE with probabilistic models to estimate return distributions in offline RL, providing error guarantees.
Key Findings
Methodology
The proposed FLE algorithm iteratively applies Maximum Likelihood Estimation (MLE) to fit flexible probabilistic generative models, such as Gaussian mixtures and diffusion models, to approximate the distribution of policy returns. It leverages the distributional Bellman operator to construct target distributions, which are then used in supervised learning steps to update the models. Theoretical analysis demonstrates that, under sufficient data coverage and successful model training, FLE converges in total variation and Wasserstein distances to the true return distribution. The approach is applicable to both finite-horizon and infinite-horizon discounted MDPs, with guarantees rooted in statistical learning theory and distance metrics.
Key Results
- In experiments on MuJoCo and high-dimensional observation spaces, FLE combined with diffusion models achieved total variation errors below 0.05, outperforming traditional distributional TD and quantile regression methods. In the discounted setting, Wasserstein errors remained under 0.07 at discount factor γ=0.95, confirming convergence and robustness. The models effectively captured complex, multi-dimensional reward distributions, demonstrating superior generalization especially in high-dimensional environments.
- Comparative analysis showed FLE's flexibility and accuracy in estimating return distributions, with significant improvements over baseline methods. The results validate the theoretical bounds and highlight the method's capacity to handle complex reward structures, making it suitable for risk-sensitive applications and offline policy evaluation in real-world scenarios.
Significance
This work advances offline reinforcement learning by providing a rigorous, theoretically guaranteed framework for distributional policy evaluation. It bridges the gap between probabilistic generative modeling and RL, enabling accurate estimation of return distributions in high-dimensional, multi-objective settings. The approach enhances risk management, decision-making robustness, and policy optimization, addressing longstanding challenges in the field. Its flexibility and theoretical foundation open new avenues for deploying RL in safety-critical and complex environments, such as autonomous systems and healthcare.
Technical Contribution
The paper introduces a novel distributional OPE framework that integrates probabilistic models trained via MLE with the distributional Bellman operator. It establishes finite-sample bounds in total variation and Wasserstein distances, demonstrating the convergence of the estimated return distribution to the true distribution under realistic assumptions. The methodology supports various generative models, broadening the applicability of distributional RL. Theoretical results include sample complexity bounds, generalization guarantees, and contraction properties under Wasserstein metrics, filling a critical gap in the theoretical understanding of distributional offline RL.
Novelty
This is the first work to combine MLE-based probabilistic modeling with distributional offline policy evaluation, providing explicit error bounds in both total variation and Wasserstein metrics. Unlike prior methods limited to scalar rewards or low-dimensional settings, FLE handles high-dimensional, multi-objective rewards with theoretical guarantees. Its flexibility in model choice and rigorous analysis represent a significant step forward in the field, offering a new paradigm for distributional RL in offline contexts.
Limitations
- The approach relies heavily on sufficient data coverage of the target policy’s trajectories; poor coverage can lead to large errors or divergence.
- Model training success assumptions may not hold in practice, especially with highly complex or high-dimensional reward distributions, limiting robustness.
- Computational costs are high, particularly for training sophisticated generative models like diffusion models, which may hinder scalability in large-scale applications.
Future Work
Future research will focus on improving model generalization, incorporating regularization and prior knowledge, and extending the framework to continuous action spaces and multi-agent settings. Additionally, integrating more efficient generative models and exploring online adaptation could further enhance practical deployment. Theoretical extensions to relax data coverage assumptions and improve sample efficiency are also promising directions.
AI Executive Summary
Understanding the full distribution of policy returns in reinforcement learning (RL) is crucial for risk-sensitive decision-making, especially in safety-critical applications. Traditional RL methods primarily focus on optimizing the expected return, neglecting the variability and tail risks. Distributional RL addresses this by estimating the entire return distribution, but most existing approaches are limited to online settings or low-dimensional rewards.
This paper introduces Fitted Likelihood Estimation (FLE), a novel algorithm that leverages maximum likelihood estimation (MLE) and flexible probabilistic generative models to estimate return distributions from offline data. FLE iteratively constructs target distributions using the distributional Bellman operator, then fits complex models like Gaussian mixtures and diffusion models to these targets. Its design allows for high-dimensional, multi-objective reward spaces, broadening the scope of distributional RL.
Theoretical analysis demonstrates that, under conditions of sufficient data coverage and successful model training, FLE guarantees convergence to the true return distribution in total variation and Wasserstein distances. Empirical results on MuJoCo and high-dimensional observation environments show that FLE achieves low errors—below 0.05 in total variation and 0.07 in Wasserstein—outperforming traditional methods. These results validate the theoretical bounds and highlight the approach’s robustness in complex, high-dimensional settings.
By integrating probabilistic models with rigorous theoretical guarantees, FLE significantly advances offline RL, enabling safer, more reliable decision-making. Its flexibility and strong performance suggest broad applicability in real-world scenarios such as autonomous driving, healthcare, and financial risk management. Future work will focus on improving scalability, model generalization, and extending to more complex environments, promising a transformative impact on the field.
Deep Dive
Abstract
We study the problem of estimating the distribution of the return of a policy using an offline dataset that is not generated from the policy, i.e., distributional offline policy evaluation (OPE). We propose an algorithm called Fitted Likelihood Estimation (FLE), which conducts a sequence of Maximum Likelihood Estimation (MLE) and has the flexibility of integrating any state-of-the-art probabilistic generative models as long as it can be trained via MLE. FLE can be used for both finite-horizon and infinite-horizon discounted settings where rewards can be multi-dimensional vectors. Our theoretical results show that for both finite-horizon and infinite-horizon discounted settings, FLE can learn distributions that are close to the ground truth under total variation distance and Wasserstein distance, respectively. Our theoretical results hold under the conditions that the offline data covers the test policy's traces and that the supervised learning MLE procedures succeed. Experimentally, we demonstrate the performance of FLE with two generative models, Gaussian mixture models and diffusion models. For the multi-dimensional reward setting, FLE with diffusion models is capable of estimating the complicated distribution of the return of a test policy.