EvoScene-VLA: Evolving Scene Beliefs Inside the Action Decoder for Chunked Robot Control
EvoScene-VLA updates scene states via actions, boosting RoboTwin task success to 88.8%.
Key Findings
Methodology
EvoScene-VLA introduces a recurrent scene prefix to maintain action-updated scene states across chunks. Key components include Geometric Anchor, Scene Predictor, and action-scene co-denoising.
Key Results
- On 31 RoboTwin tasks, success rates improved from 86.4% to 88.8% (fixed evaluation) and from 85.7% to 87.9% (randomized evaluation).
- Outperformed all baselines on the Galaxea R1-Lite real robot.
- Ablation studies showed cumulative contributions from Geometric Anchor and Scene Predictor.
Significance
Addresses the lack of persistent scene states in robot control, enabling more accurate decision-making for complex tasks. Significant implications for academia and industry.
Technical Contribution
Introduced recurrent scene prefix and action-scene co-denoising, advancing scene representation capabilities and enabling cross-chunk updates for the first time.
Novelty
First to integrate action-updated scene states into robot control, leveraging Geometric Anchor and Scene Predictor for efficient recursive updates.
Limitations
- Scene Predictor is training-only, which may limit generalization.
- Geometric representation for complex scenes needs improvement.
Future Work
Future research could explore more efficient scene prediction mechanisms and applications in multi-robot collaboration.
AI Executive Summary
EvoScene-VLA is a novel robot control framework that updates scene states dynamically across action chunks.
Existing methods fail to account for how actions alter scenes, leading to inefficiencies in complex tasks. EvoScene-VLA uses Geometric Anchor and Scene Predictor for training supervision and employs action-scene co-denoising during inference to update scene states.
The framework demonstrated significant success rate improvements on RoboTwin tasks and excelled in real-world robot experiments. EvoScene-VLA opens new avenues for robot control research and practical applications in dynamic environments.
Deep Analysis
Background
Robot control has advanced significantly, especially with vision-language models. However, these methods struggle to handle dynamic changes caused by robot actions, limiting their effectiveness in complex tasks.
Core Problem
Existing methods fail to maintain action-updated scene states, forcing robots to re-infer scene changes, increasing computational load and reducing task success rates.
Innovation
EvoScene-VLA introduces a recurrent scene prefix and action-scene co-denoising mechanism for dynamic scene updates. Geometric Anchor provides 3D structural supervision, while Scene Predictor generates future scene targets.
Methodology
- �� Recurrent Scene Prefix: Maintains scene states across chunks.
- �� Geometric Anchor: Combines depth supervision and 3D model features.
- �� Scene Predictor: Generates future scene targets during training.
- �� Action-Scene Co-denoising: Updates actions and scene states jointly during inference.
Experiments
Evaluated on RoboTwin and LIBERO benchmarks, and tested on the Galaxea R1-Lite real robot. Ablation studies analyzed component contributions.
Results
EvoScene-VLA improved success rates to 88.8% (fixed evaluation) and 87.9% (randomized evaluation). Ablation studies highlighted the importance of Geometric Anchor and Scene Predictor.
Applications
Applicable to complex robot tasks like object manipulation and dynamic scene interaction, especially where cross-chunk scene updates are critical.
Limitations & Outlook
Geometric representation for complex scenes needs refinement, and Scene Predictor is limited to training, potentially affecting generalization.
Plain Language Accessible to non-experts
Imagine a chef working in a kitchen. Every time they perform an action, like chopping vegetables or opening a cabinet, the kitchen's state changes. If the chef can't remember these changes, they'd have to constantly recheck the entire kitchen. EvoScene-VLA acts like a smart assistant, keeping track of these changes and updating the kitchen's state dynamically, making the chef's work more efficient.
ELI14 Explained like you're 14
Think of playing a robot game where you control it to complete tasks. Every time the robot does something, like opening a door or picking up an object, the scene changes. If the game can't remember these changes, you'd have to keep rechecking everything—so annoying! EvoScene-VLA is like a super memory chip for the robot, helping it remember and update the scene, making tasks faster and smoother!
Glossary
Recurrent Scene Prefix
A method for maintaining scene states across action chunks.
Used to record and update scene changes caused by robot actions.
Geometric Anchor
Supervises scene representation using depth and 3D features.
Provides structural information during training.
Scene Predictor
Predicts future scene states.
Generates target scene representations during training.
Action-Scene Co-denoising
Simultaneously updates actions and scene states.
Used during inference to generate action and scene updates.
RoboTwin Benchmark
A set of robot tasks for evaluating model performance.
Includes 31 tasks covering diverse scenarios.
Open Questions Unanswered questions from this research
- 1 How to achieve more efficient scene prediction during inference?
- 2 How to extend this framework to multi-robot collaboration?
Applications
Immediate Applications
Dynamic Scene Interaction
Suitable for robot tasks requiring real-time scene updates, like object manipulation.
Complex Task Planning
Helps robots maintain consistent scene states during long-duration tasks.
Long-term Vision
Multi-Robot Collaboration
Explores shared scene states among robots for complex cooperative tasks.
Abstract
Chunked vision-language-action (VLA) policies predict multi-step robot controls, conditioning each update on the current visual observation alone. Yet robot actions cause contact, occlusion, and object motion, and the geometry that later decisions depend on can change before the next visual update arrives. Spatial VLAs improve current-frame geometry. Temporal VLAs aggregate past frames. Neither maintains an action-updated scene prior across chunks. We argue for a persistent action-updated scene state across control calls, and introduce EvoScene-VLA. Its recurrent scene prefix carries a geometry-aware scene state across chunks. At each vision-language model (VLM) call, the VLM combines scene information from the current observation with the action-updated prior from the previous chunk; the action decoder outputs both the next action chunk and a compact scene update. This update becomes the next prior, which the VLM corrects against the new observation when the next call arrives. Each control call therefore starts from a scene prior that reflects both recent actions and fresh visual evidence. During training, \textbf{Scene Predictor} supplies future scene-token targets, and Geometric Anchor aligns scene slots with frozen depth and 3D teachers. We discard both modules at deployment. On 31 RoboTwin tasks, EvoScene-VLA raises average success from 87.2% to 89.1% in fixed evaluation and from 86.1% to 88.5% in randomized evaluation. On the Galaxea R1-Lite real robot, EvoScene-VLA outperforms all baselines.