VLA-OPD: Bridging Offline SFT and Online RL for Vision-Language-Action Models via On-Policy Distillation
VLA-OPD combines SFT and RL via Reverse-KL distillation, improving sample efficiency and robustness for Vision-Language-Action models.
Key Findings
Methodology
VLA-OPD integrates the efficiency of offline Supervised Fine-Tuning (SFT) with the robustness of online Reinforcement Learning (RL). Using a Reverse-KL objective, the student model receives dense supervision from a frozen teacher model on self-generated trajectories, avoiding Forward-KL's entropy explosion while preserving action diversity.
Key Results
- On the LIBERO benchmark, VLA-OPD achieved over 90% success rate within 10 steps, a 3× speedup compared to GRPO.
- On RoboTwin2.0 tasks, VLA-OPD outperformed GRPO by 15% in success rate with 1,000-traj initialization.
- Ablation studies showed Reverse-KL significantly mitigates entropy explosion and collapse, ensuring stable optimization.
Significance
This work addresses critical issues in SFT (distribution shift) and RL (sample inefficiency) by introducing a unified framework for post-training Vision-Language-Action models. It reduces training costs while preserving pre-trained generalist capabilities, making it impactful for both academia and industry.
Technical Contribution
Proposed a Reverse-KL distillation objective to avoid Forward-KL's mode-covering and Hard-CE's entropy collapse. By combining dense supervision and on-policy sampling, VLA-OPD enables active error correction and mitigates catastrophic forgetting.
Novelty
This is the first application of Reverse-KL distillation in Vision-Language-Action models, introducing a mode-seeking optimization strategy to address distribution shift and sample inefficiency.
Limitations
- Relies on high-quality teacher models, which may limit generalizability.
- High-entropy teacher distributions in complex environments may introduce noise.
- Experiments focus on single-task settings; cross-task generalization is unexplored.
Future Work
Future work could explore efficient teacher generation methods and extend the framework to multi-task and multi-modal environments.
AI Executive Summary
Vision-Language-Action (VLA) models excel in robotic manipulation but face challenges during post-training. Traditional methods like Supervised Fine-Tuning (SFT) suffer from distribution shift and catastrophic forgetting, while Reinforcement Learning (RL) struggles with sparse rewards and sample inefficiency.
The VLA-OPD framework bridges the strengths of SFT and RL using Reverse-KL distillation. The student model receives dense supervision from a frozen teacher model on self-generated trajectories, enabling active error correction while avoiding Forward-KL's entropy explosion. Experiments on LIBERO and RoboTwin2.0 benchmarks demonstrate significant improvements in sample efficiency and robustness.
Although VLA-OPD relies on high-quality teacher models, its ability to reduce training costs and preserve pre-trained capabilities offers a promising pathway for advancing VLA models. Future research could explore its applications in multi-task environments.
Deep Analysis
Background
Vision-Language-Action (VLA) models integrate visual perception, language understanding, and action control, driving advancements in robotics. Existing post-training methods like SFT and RL have limitations: SFT is stable but prone to distribution shift, while RL improves robustness but is sample-inefficient.
Core Problem
The key challenge is balancing SFT's distribution shift issues and RL's sample inefficiency. Traditional methods like DAGGER and GRPO struggle with optimization objectives, failing to balance mode coverage and action diversity.
Innovation
Key innovations of VLA-OPD include:
- �� Reverse-KL objective: avoids entropy explosion in Forward-KL and promotes mode-seeking.
- �� Dense supervision: teacher model provides detailed guidance on student trajectories.
- �� Dynamic sampling: student samples from its own distribution, addressing distribution shift.
Methodology
VLA-OPD involves:
- �� Student sampling: student interacts with the environment to generate trajectories.
- �� Teacher labeling: frozen teacher model provides action distributions for each state.
- �� Mode-seeking optimization: minimizes Reverse-KL divergence between student and teacher distributions.
Experiments
Experiments were conducted on LIBERO and RoboTwin2.0 benchmarks, evaluating single-arm and dual-arm tasks. Student models were initialized from 1-traj or 1,000-traj SFT, with SimpleVLA-RL as the teacher. Baselines included GRPO and full-dataset SFT.
Results
On LIBERO, VLA-OPD achieved an average success rate of 87.4%, a 40% improvement over 1-traj SFT. On RoboTwin2.0, VLA-OPD converged 3× faster than GRPO and showed higher robustness in complex tasks.
Applications
This method is suitable for post-training in robotic tasks, especially in data-scarce or complex scenarios like industrial automation and home service robots.
Limitations & Outlook
VLA-OPD depends on high-quality teacher models, which may limit performance in unreliable settings. Its generalization to multi-task environments remains to be tested.
Plain Language Accessible to non-experts
Imagine learning to ride a bike. Traditional methods involve watching others (SFT), but you might fall because you don't know how to correct mistakes. Another way is trial and error (RL), but without guidance, progress is slow. VLA-OPD is like having an expert coach who not only shows you the right moves but also corrects you when you make mistakes. This way, you learn faster and avoid repeated falls.
ELI14 Explained like you're 14
Think of playing a robot game. You have a super-smart AI teammate (the teacher), but your own robot keeps messing up. VLA-OPD is like letting the AI teach you how to play. Every time you make a mistake, it tells you the correct move. Soon, your robot becomes as good as the AI!
Glossary
Reverse-KL
An optimization objective encouraging the student model to focus on the teacher's high-probability actions.
Used for mode-seeking optimization in VLA-OPD.
SFT (Supervised Fine-Tuning)
A method to train models using expert demonstration data.
Serves as the initialization step for VLA-OPD.
RL (Reinforcement Learning)
A learning method optimizing policies through environment interaction.
Compared against VLA-OPD as a baseline.
LIBERO
A benchmark for evaluating single-arm manipulation tasks.
Used to assess VLA-OPD's sample efficiency.
RoboTwin2.0
A benchmark for dual-arm robotic manipulation tasks.
Tests VLA-OPD's performance in complex scenarios.
Open Questions Unanswered questions from this research
- 1 How can similar distillation effects be achieved without high-quality teacher models?
- 2 What is the generalization capability of VLA-OPD in multi-task and multi-modal environments?
Applications
Immediate Applications
Robotic Task Optimization
Quickly optimize robotic task strategies in industrial automation, reducing training time.
Home Service Robots
Enhance learning capabilities of home service robots in data-scarce environments.
Long-term Vision
General Robotic Intelligence
Achieve cross-task, cross-environment general intelligence in robotics, lowering development costs.
Abstract
Although pre-trained Vision-Language-Action (VLA) models exhibit impressive generalization in robotic manipulation, post-training remains crucial to ensure reliable performance during deployment. However, standard offline Supervised Fine-Tuning (SFT) suffers from distribution shifts and catastrophic forgetting of pre-trained capabilities, while online Reinforcement Learning (RL) struggles with sparse rewards and poor sample efficiency. In this paper, we propose On-Policy VLA Distillation (VLA-OPD), a framework bridging the efficiency of SFT with the robustness of RL. Instead of relying on sparse environmental rewards, VLA-OPD leverages an expert teacher to provide dense, token-level supervision on the student's self-generated trajectories. This enables active error correction on policy-induced states while preserving pre-trained general capabilities through gentle alignment. Crucially, we formulate VLA-OPD via a Reverse-KL objective. Unlike standard Forward-KL that induces mode-covering entropy explosion, or Hard-CE that causes premature entropy collapse, our bounded mode-seeking objective ensures stable policy learning by filtering out the teacher's epistemic uncertainty while maintaining action diversity. Experiments on LIBERO and RoboTwin2.0 benchmarks demonstrate that VLA-OPD significantly improves sample efficiency over RL and robustness over SFT, while effectively mitigating catastrophic forgetting during post-training.