Beyond Imitation: Self-Improving Robot Policies via Off-Policy Q-Planning
Q-Planning integrates a frozen behavior cloning policy with a small off-policy Q-function, enabling stable self-improvement and success rates up to 99%.
Key Findings
Methodology
This paper introduces the Q-Planning framework, combining a large-scale visuomotor behavior cloning (BC) policy with a compact off-policy Q-function. The Q-function estimates action values from deployment data, including failures, rather than imitating actions directly. During inference, N candidate action chunks are sampled from the frozen BC policy; these are scored by the Q-function; a single-step Q-weighted average determines the executed action. The system continuously collects deployment rollouts—both successes and failures—and updates only the Q-network parameters via gradient descent, leaving the BC weights unchanged. Experiments on LIBERO and RoboTwin demonstrate that ten iterations of this self-improvement loop raise success rates from 93% to 99%, and real robot tasks show improvements from 40% to 90% in contact-rich manipulation tasks, all without human intervention.
Key Results
- In LIBERO-10, success rate improved from 93% to 99%, with an average success rate increase of 5.5 percentage points after ten self-improvement iterations. RoboTwin tasks saw success rates rise from 83.8% to 91.4%. On real robots, the contact-rich tasks 'stack-cups' and 'insert-wallet' improved success from 40% to 90% and 25% to 80%, respectively, within five iterations, solely from deployment rollouts. Compared to methods relying only on successful trajectories, Q-Planning effectively utilizes failure data to enhance policy performance, demonstrating robustness and scalability.
Significance
This work addresses the fundamental limitation of behavior cloning—its inability to learn from failures—by leveraging a small off-policy Q-function for value estimation. The approach enables continuous, stable self-improvement without retraining the entire policy, making it scalable for billion-parameter models. It bridges the gap between imitation learning and reinforcement learning, offering a practical pathway for autonomous robot systems to adapt and improve in complex, real-world scenarios. The method’s ability to learn solely from deployment data, including failures, marks a significant step toward fully autonomous, self-improving robots.
AI Executive Summary
Deep Dive
Abstract
Behaviour Cloning (BC) has driven remarkable progress in robot manipulation, yet it is fundamentally limited by its inability to self-improve: a policy that fails cannot learn from that failure without additional human demonstrations. Reinforcement Learning fine-tuning offers a path to self-improvement but has proven difficult to scale to the multi-billion-parameter models underpinning modern robot policies. We propose Q-Planning, which equips a large visuomotor BC policy with a small off-policy Q-function. Because a Q-function estimates value rather than imitates actions, it can be trained on the same successful demonstrations as the BC policy and later absorb both successful and failed deployment rollouts, an asymmetry BC does not have. We exploit this asymmetry to enable value-guided action selection at inference (a single-step Q-weighted average over BC draws) and online self-improvement that fine-tunes only the Q-function, leaving the BC weights untouched. On LIBERO and bimanual RoboTwin, ten iterations of self-improvement lift every benchmark score we tested (LIBERO-10 93% to 99%, RoboTwin 83.8% to 91.4%) and shorten successful episodes on the near-ceiling suites (LIBERO-Object, LIBERO-Goal). On two contact-rich bimanual real-robot tasks, the same loop (BC frozen, no human intervention) improves purely from its own deployment rollouts: stack-cups 40% to 90% and insert-wallet 25% to 80% in five iterations, whereas SFT on successful rollouts alone stalls at 55% and 30%. Under an identical online budget Q-Planning is the only method, among Best-of-N, filtered SFT, IBRL, DSRL, and DAWR, that improves stably from failures without training an auxiliary actor.