核心发现
方法论
本文提出Q-Planning框架，结合大规模视觉-运动行为克隆（BC）策略与小型离策略Q函数。Q函数通过对成功与失败的部署数据进行训练，估算动作的价值而非模仿，从而实现对策略的在线自我改进。具体机制包括：• 在推理阶段，从冻结的BC策略采样N个候选动作块；• 利用训练好的Q函数对候选动作进行评分；• 通过Q值加权平均选择动作，提升决策质量；• 通过收集部署中的成功与失败轨迹，持续微调Q函数参数，只更新Q网络，不触动BC权重。该方法在LIBERO和RoboTwin任务中实现了10轮自我提升，成功率从93%提升至99%，在真实机器人上的两项接触丰富任务中，成功率从40%提升至90%。
关键结果
- 在LIBERO-10任务中，Q-Planning通过10轮自我提升将成功率从93%提升至99%，平均成功率提升5.5个百分点，显著优于传统基线。RoboTwin任务中，成功率从83.8%跃升至91.4%。在两项真实机器人任务中，stack-cups成功率由40%提升至90%，insert-wallet由25%提升至80%，仅用五轮迭代，无需人类干预。与仅依赖成功轨迹的SFT相比，Q-Planning能有效利用失败信息进行优化。
研究意义
该研究突破了行为模仿的固有限制，提出利用离策略Q函数实现策略的持续自我提升，极大增强了机器人在复杂环境中的适应能力。其无需额外示范或策略更新，便能从失败中学习，解决了大规模机器人策略在实际部署中的瓶颈问题。此方法为未来自主机器人系统提供了新思路，有望推动机器人自主学习、连续改进的研究方向。
AI 总览摘要
深度解读
原文摘要
Behaviour Cloning (BC) has driven remarkable progress in robot manipulation, yet it is fundamentally limited by its inability to self-improve: a policy that fails cannot learn from that failure without additional human demonstrations. Reinforcement Learning fine-tuning offers a path to self-improvement but has proven difficult to scale to the multi-billion-parameter models underpinning modern robot policies. We propose Q-Planning, which equips a large visuomotor BC policy with a small off-policy Q-function. Because a Q-function estimates value rather than imitates actions, it can be trained on the same successful demonstrations as the BC policy and later absorb both successful and failed deployment rollouts, an asymmetry BC does not have. We exploit this asymmetry to enable value-guided action selection at inference (a single-step Q-weighted average over BC draws) and online self-improvement that fine-tunes only the Q-function, leaving the BC weights untouched. On LIBERO and bimanual RoboTwin, ten iterations of self-improvement lift every benchmark score we tested (LIBERO-10 93% to 99%, RoboTwin 83.8% to 91.4%) and shorten successful episodes on the near-ceiling suites (LIBERO-Object, LIBERO-Goal). On two contact-rich bimanual real-robot tasks, the same loop (BC frozen, no human intervention) improves purely from its own deployment rollouts: stack-cups 40% to 90% and insert-wallet 25% to 80% in five iterations, whereas SFT on successful rollouts alone stalls at 55% and 30%. Under an identical online budget Q-Planning is the only method, among Best-of-N, filtered SFT, IBRL, DSRL, and DAWR, that improves stably from failures without training an auxiliary actor.