CoRE-VLA: Towards Scalable and Robust Vision-Language-Action Modeling via Conditional Routing of Experts
CoRE-VLA uses conditional routing of experts for scalable and robust vision-language-action modeling, excelling in multi-task and long-horizon tasks.
Key Findings
Methodology
CoRE-VLA employs conditional routing of experts to achieve context-conditioned sparse computation for action generation. The framework uses sensor availability to gate modality-specialized experts and task intent to route action-side representations to task-relevant experts. Depth is used as an auxiliary modality in experiments, demonstrating strong performance on benchmarks like LIBERO and RoboCasa GR1 Tabletop.
Key Results
- On the LIBERO benchmark, CoRE-VLA achieved a 97.6% success rate in long-horizon tasks, significantly outperforming other baselines.
- On the RoboCasa GR1 Tabletop benchmark, CoRE-VLA achieved an average success rate of 56.5%, outperforming other methods.
- In real-world dual-arm manipulation, CoRE-VLA maintained robustness even without auxiliary depth, demonstrating resilience to sensor absence.
Significance
This research provides a novel approach for robotic manipulation, maintaining efficiency and robustness in multi-task and long-horizon tasks. By using conditional expert routing, CoRE-VLA adapts computation paths to different sensor configurations, addressing the performance drop issue in existing methods when sensors are missing.
Technical Contribution
CoRE-VLA introduces conditional expert routing, allowing adaptive computation across different tasks and sensor configurations. This approach not only enhances model robustness but also improves scalability in multi-task learning.
Novelty
CoRE-VLA is the first to apply conditional expert routing in vision-language-action models, achieving robustness and task adaptability in the absence of sensors, marking a significant innovation over existing methods.
Limitations
- In complex tasks, the model may still show sensitivity to the absence of specific sensors.
- Requires substantial training data to fully realize its potential.
Future Work
Future work will extend CoRE-VLA to support more physical sensors like tactile sensing and force feedback, and train on larger cross-robot data scales.
AI Executive Summary
In robotic manipulation, existing vision-language-action models often underperform when faced with diverse sensor configurations. CoRE-VLA addresses this issue by introducing conditional expert routing, allowing the model to maintain efficient action generation even when sensors are absent.
The core of CoRE-VLA lies in treating action generation as context-conditioned sparse computation. Sensor availability gates modality-specialized experts, while task intent routes action-side representations to task-relevant experts. This design enables the model to adapt computation paths across different tasks and sensor configurations.
Experimental results show that CoRE-VLA excels on benchmarks like LIBERO and RoboCasa GR1 Tabletop, particularly in long-horizon tasks. Even in real-world dual-arm manipulation, the model remains robust without auxiliary depth, indicating its significant potential in multi-task and long-horizon tasks.
Deep Analysis
Background
Vision-language-action models have made significant strides in robotic manipulation in recent years. However, existing methods often rely on fixed sensor configurations, leading to performance degradation when sensors are missing. To address this challenge, researchers have proposed various methods, but most have failed to solve the issues of sensor diversity and task complexity.
Core Problem
Robots in real-world applications often face diverse sensor configurations and sensor failures. Existing VLA models, trained under fixed sensor configurations, struggle to adapt to these changes, resulting in significant performance drops when sensors are absent.
Innovation
CoRE-VLA achieves context-conditioned sparse computation for action generation through conditional expert routing. • Sensor availability gates modality-specialized experts, allowing graceful degradation when sensors are missing. • Task intent routes action-side representations to task-relevant experts, enhancing task adaptability.
Methodology
- �� A pretrained vision-language model encodes RGB and language instructions. • A modality encoder processes auxiliary sensor inputs. • An action diffusion transformer generates actions through conditional expert routing, combined with modality dropout and expert masking strategies.
Experiments
Experiments were conducted on the LIBERO and RoboCasa GR1 Tabletop benchmarks, using depth as an auxiliary modality. The model demonstrated strong performance in multi-task and long-horizon tasks, particularly in robustness to sensor absence.
Results
On the LIBERO benchmark, CoRE-VLA achieved a 97.6% success rate in long-horizon tasks, showcasing its task adaptability advantage. On the RoboCasa GR1 Tabletop benchmark, the average success rate was 56.5%, outperforming other methods.
Applications
CoRE-VLA can be applied in robotic manipulation scenarios requiring multi-task and long-horizon tasks, such as industrial automation and service robots. Its robustness to sensor absence makes it widely applicable in real-world settings.
Limitations & Outlook
Despite CoRE-VLA's excellent performance in sensor absence, it may still show sensitivity to the absence of specific sensors in complex tasks. Additionally, the model requires substantial training data to fully realize its potential.
Plain Language Accessible to non-experts
Imagine a factory where workers adapt to different tasks and tools. If a tool breaks, they find alternatives to continue working. CoRE-VLA is like these workers, finding alternative paths when sensors are missing to continue efficient task completion.
ELI14 Explained like you're 14
Imagine playing a robot game where your robot needs to complete various tasks. Sometimes, your robot loses some tools, but it still finds other ways to complete the tasks. CoRE-VLA is like this smart robot, finding alternative paths when sensors are missing to continue task completion.
Glossary
Conditional Routing of Experts
A method that selectively activates computation paths based on task and sensor status.
Used in CoRE-VLA to achieve context-conditioned sparse computation for action generation.
Vision-Language Model
A model that combines visual and language information for task processing.
Used to encode RGB and language instructions.
Modality Dropout
A method that randomly disables auxiliary modalities during training to enhance model robustness.
Prevents over-reliance on auxiliary sensor inputs.
Expert Masking
A mechanism that disables specific experts based on sensor availability.
Maintains model robustness when sensors are missing.
Action Diffusion Transformer
A transformer model for generating actions, incorporating conditional expert routing.
Used in CoRE-VLA for action generation.
Open Questions Unanswered questions from this research
- 1 How can CoRE-VLA be effectively trained on larger cross-robot data scales?
- 2 How does CoRE-VLA perform in more complex physical environments?
Applications
Immediate Applications
Industrial Automation
Apply CoRE-VLA in industrial settings for multi-task manipulation, improving production efficiency.
Long-term Vision
Service Robots
Apply CoRE-VLA in homes and public spaces for automated service of complex tasks.
Abstract
Vision-language-action (VLA) models have advanced generalist robotic manipulation, yet real-world deployment reveals a fundamental challenge: robots are equipped with diverse and heterogeneous sensor configurations, auxiliary sensors can fail unexpectedly during operation, and different robot embodiments often lack certain sensors by design. A unified policy that can exploit auxiliary perceptual inputs when available while remaining reliable under sensor absence, whether incidental or by design, is therefore essential for practical deployment. However, existing VLA policies couple action generation to a fixed sensor set through shared dense computation, making them brittle when sensors are missing and limiting their ability to specialize across diverse tasks and long-horizon behaviors. We propose CoRE-VLA, a scalable and robust VLA framework that formulates action generation as context-conditioned sparse computation. Sensor availability gates modality-specialized experts, enabling graceful degradation under missing sensors without retraining. Task intent further routes action-side representations to task-relevant experts, improving specialization across diverse tasks and long-horizon subgoals. While the framework is designed to accommodate different auxiliary sensors, we focus on depth as a representative and practically important auxiliary modality in our experiments. Experiments on LIBERO, RoboCasa GR1 Tabletop, and real-world dual-arm manipulation show that CoRE-VLA achieves strong results on long-horizon and multi-task benchmarks, and outperforms both a dense-action-generator ablation and a strong pretrained VLA baseline, including in zero-shot generalization to unseen scenarios. Modality analysis shows that CoRE-VLA can exploit auxiliary depth when available while remaining robust when depth is unavailable during deployment.