Learning Process Rewards via Success Visitation Matching for Efficient RL
Proposes Success Visitation Matching (SVM) rewards, transforming sparse rewards into dense signals, doubling RL finetuning speed in robotic tasks.
Key Findings
Methodology
This paper introduces a discriminator-based Success Visitation Matching (SVM) reward mechanism, which distinguishes between successful and unsuccessful trajectories through a trained classifier. The log-probability ratio output by the discriminator serves as a dense reward signal, encouraging the policy to imitate the state-action visitation distribution of successful episodes while avoiding unsuccessful ones. The approach involves: β’ Collecting datasets D+ and Dβ of successful and unsuccessful episodes; β’ Training a discriminator bfh to classify these episodes; β’ Estimating visitation ratios via the discriminatorβs output; β’ Combining the original sparse reward with the density ratio-based reward to form a dense process reward rsvm; β’ Alternating policy updates and discriminator refinement during RL training. Theoretically, in deterministic environments, maximizing rsvm aligns with maximizing the original sparse reward, ensuring policy optimality. Empirically, experiments on simulated and real robotic tasks demonstrate a roughly 2Γ acceleration in RL finetuning convergence, with significant improvements in success rates and training stability.
Key Results
- In LIBERO-90 and RoboCasa benchmarks, RL finetuning with SVM rewards achieved approximately twice the speed of baseline methods using only sparse rewards. For example, in LIBERO scenes 1-3, training steps reduced from around 2 million to 1 million, with success rates increasing by over 20%.
- On the real WidowX 250 robotic arm, success rates improved from 45% to 70%, with more stable training curves, indicating better sample efficiency and transferability.
- Ablation studies confirmed that removing the discriminator-based density ratio component slowed convergence by about 30%, highlighting its critical role. The method remained effective under environmental stochasticity, demonstrating robustness.
Significance
This work addresses a fundamental challenge in reinforcement learning: sparse reward signals hinder rapid policy improvement, especially in robotics. By converting sparse outcome rewards into dense process rewards automatically, the proposed SVM method significantly enhances sample efficiency and accelerates learning. Its theoretical guarantees ensure that optimal policies remain unchanged, making it a reliable tool for real-world robotic applications. The approach reduces reliance on manual reward engineering or extensive demonstrations, broadening the applicability of RL in complex, unstructured environments. As a result, it paves the way for more autonomous, adaptable robots capable of learning complex tasks faster and more reliably, with profound implications for industry automation, service robotics, and beyond.
Technical Contribution
The core technical innovation lies in leveraging a discriminator to estimate the density ratio between successful and unsuccessful trajectory visitations, which is then used to shape a dense reward signal. This approach circumvents the high-dimensional density estimation problem by training a binary classifier, whose output probability ratio approximates visitation ratios. The method guarantees that maximizing the resulting reward aligns with maximizing the original sparse reward in deterministic settings, as proven theoretically. Additionally, the framework is compatible with various RL algorithms, including diffusion policy fine-tuning and residual RL, enabling broad applicability. The paper also establishes a connection between the SVM reward and KL-regularized RL, showing that the reward induces an optimization landscape favoring visitation of successful states while penalizing unsuccessful ones, thus improving learning efficiency without compromising optimality.
Novelty
This research introduces a novel, fully automated reward shaping technique based on success visitation matching, which dynamically updates positive and negative trajectory sets during training. Unlike prior methods relying on handcrafted rewards, distance functions, or fixed demonstrations, the SVM approach continuously adapts by training a discriminator online, providing a step-by-step dense reward signal. Its theoretical proof that the optimal policy remains unchanged underlines its robustness. This is the first work to formalize the connection between discriminator-based visitation ratios and policy optimality in the context of sparse reward RL, offering a new paradigm for reward shaping that is both theoretically sound and practically effective in robotic control tasks.
Limitations
- The theoretical guarantees primarily hold in deterministic environments; in stochastic settings, the alignment between maximizing SVM reward and the original sparse reward may weaken, requiring further analysis.
- Discriminator training depends on representative and sufficient trajectory data; poor sampling or environmental noise can impair the accuracy of visitation ratio estimates, affecting reward quality.
- Scaling to very high-dimensional state spaces remains computationally demanding, as training discriminators over large datasets can be resource-intensive, especially in complex robotic tasks.
Future Work
Future research will focus on extending theoretical guarantees to stochastic environments, improving discriminator robustness through advanced neural architectures, and integrating unsupervised or self-supervised learning techniques to reduce data requirements. Additionally, combining SVM rewards with imitation learning and inverse RL methods could further enhance learning efficiency. Exploring multi-task and multi-agent scenarios, as well as real-time adaptation in dynamic environments, are promising directions. Ultimately, the goal is to develop universally applicable, computationally efficient reward shaping frameworks that enable robots to learn complex behaviors rapidly and reliably in diverse real-world settings.
AI Executive Summary
Reinforcement learning (RL) has become a cornerstone of autonomous robotic control, enabling machines to learn complex behaviors through trial and error. However, a persistent challenge remains: the reliance on sparse reward signals, which only provide feedback upon task completion. Such sparse rewards severely hinder the efficiency of learning, often requiring millions of interactions before meaningful progress is observed. This bottleneck is particularly acute in robotic manipulation tasks, where collecting data is costly and time-consuming.
To address this, the present work introduces a novel reward shaping approach called Success Visitation Matching (SVM). The core idea is to transform sparse outcome rewards into dense process rewards by leveraging a discriminator trained to distinguish between successful and unsuccessful trajectories. This discriminator estimates the likelihood that a given state-action pair belongs to a successful trajectory, and the log-probability ratio serves as a dense reward signal. By incorporating this into RL algorithms, the policy receives continuous feedback on its progress, guiding it more effectively toward task completion.
The theoretical foundation of the method guarantees that, in deterministic environments, policies maximizing the SVM reward also maximize the original sparse reward. This ensures that the optimality of the learned policy is preserved. Empirical evaluations on simulated benchmarks like LIBERO-90 and RoboCasa, as well as real-world robotic tasks with the WidowX 250 arm, demonstrate that RL with SVM rewards converges approximately twice as fast as traditional sparse reward methods. Success rates improve significantly, and training stability is enhanced, making the approach highly practical for real-world applications.
The key innovation lies in the online, continual updating of the discriminator, which dynamically refines the reward signal based on observed trajectories. This contrasts with prior reward shaping methods that often rely on handcrafted functions or fixed demonstrations. The approach is versatile, compatible with various RL algorithms such as diffusion policy fine-tuning and residual RL, and scalable to high-dimensional state spaces through classifier-based density ratio estimation.
Despite its strengths, the method has limitations, including reduced theoretical guarantees in stochastic environments and computational costs associated with discriminator training. Future work aims to extend theoretical analysis, improve scalability, and integrate with other learning paradigms like imitation learning. Overall, this research offers a powerful, theoretically grounded framework to accelerate robotic learning, paving the way for more autonomous, adaptable robots capable of learning complex tasks efficiently in diverse environments.
Deep Dive
Abstract
In many modern applications of reinforcement learning (RL), the natural reward for a task of interest is inherently sparse: a reward of 0 is given everywhere except when the task is completed, when a reward of +1 is given. Training a policy to maximize such a sparse reward requires solving a challenging credit assignment problem, leading to slow or ineffective RL improvement. We propose a simple approach to transform a sparse outcome reward into a dense process reward. Our approach relies on training a discriminator to distinguish between previous successful and unsuccessful episodes, and using this discriminator to incentivize the RL-learned policy to match the state-action visitations of successful episodes, while avoiding those of unsuccessful episodes. By incentivizing the policy to match the visitations over all states, not just those that correspond to task success, this reward provides dense feedback on whether progress is being made towards task completion, and, we show, provably achieves this without changing the optimal policy. Focusing on finetuning of robotic control policies, we demonstrate that our approach leads to significantly faster RL finetuning performance on both simulated and real-world manipulation tasks, as compared to simply maximizing the sparse outcome reward.
References (20)
Steering Your Diffusion Policy with Latent Space Reinforcement Learning
Andrew Wagenmaker, Mitsuhiko Nakamoto, Yunchu Zhang et al.
Self-Improving Vision-Language-Action Models with Data Generation via Residual RL
Wenli Xiao, Haotian Lin, Andy Peng et al.
Efficient Online Reinforcement Learning with Offline Data
Philip J. Ball, Laura M. Smith, Ilya Kostrikov et al.
Generative Adversarial Imitation Learning
Jonathan Ho, Stefano Ermon
From Imitation to Refinement - Residual Rl for Precise Assembly
Lars Ankile, A. Simeonov, Idan Shenfeld et al.
Policy Decorator: Model-Agnostic Online Refinement for Large Policy Model
Xiu Yuan, Tongzhou Mu, Stone Tao et al.
Self-Supervised Online Reward Shaping in Sparse-Reward Environments
F. Memarian, Wonjoon Goo, Rudolf Lioutikov et al.
Let's Verify Step by Step
H. Lightman, Vineet Kosaraju, Yura Burda et al.
Learning Intrinsic Rewards as a Bi-Level Optimization Problem
Bradly C. Stadie, Lunjun Zhang, Jimmy Ba
Enhancing Rating-Based Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Language Models
T. Luu, Younghwan Lee, Donghoon Lee et al.
OpenVLA: An Open-Source Vision-Language-Action Model
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti et al.
RoboCLIP: One Demonstration is Enough to Learn Robot Policies
S. Sontakke, Jesse Zhang, S'ebastien M. R. Arnold et al.
Adversarial Intrinsic Motivation for Reinforcement Learning
Ishan Durugkar, Mauricio Tec, S. Niekum et al.
Residual Reinforcement Learning for Robot Control
T. Johannink, Shikhar Bahl, Ashvin Nair et al.
EXPO: Stable Reinforcement Learning with Expressive Policies
Perry Dong, Qiyang Li, Dorsa Sadigh et al.
Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videos
Annie S. Chen, Suraj Nair, Chelsea Finn
A Walk in the Park: Learning to Walk in 20 Minutes With Model-Free Reinforcement Learning
Laura M. Smith, Ilya Kostrikov, Sergey Levine
End-to-End Training of Deep Visuomotor Policies
S. Levine, Chelsea Finn, Trevor Darrell et al.
Video-Language Critic: Transferable Reward Functions for Language-Conditioned Robotics
Minttu Alakuijala, Reginald McLean, Isaac Woungang et al.
Reward Shaping for Reinforcement Learning with An Assistant Reward Agent
Haozhe Ma, Kuankuan Sima, Thanh Vinh Vo et al.