CoFreeVLA: Short-Horizon Collision-Free Dual-Arm Manipulation via Vision-Language-Action Model and Risk Estimation
CoFreeVLA reduces dual-arm self-collision rates from 0.54 to 0.23 and improves task success rates to 0.61 using a Vision-Language-Action model with risk estimation.
Key Findings
Methodology
CoFreeVLA integrates a Vision-Language-Action (VLA) model with a lightweight short-horizon risk estimator. The estimator predicts self-collision likelihood using visual embeddings, proprioceptive states, and action sequences. It halts risky actions, generates recovery trajectories, and refines policies for safer task execution.
Key Results
- CoFreeVLA reduces average self-collision rates across five tasks from 0.54 to 0.23, a 57.6% reduction.
- Task success rates improve from 0.45 to 0.61, a 34.5% increase.
- Outperforms all baseline methods in both safety and task success across six VLA backbones and five tasks.
Significance
This work addresses the critical challenge of self-collision in dual-arm robotic manipulation, enabling safer deployment in real-world environments. By embedding risk estimation into VLA models, CoFreeVLA achieves a balance between task performance and safety, advancing state-of-the-art robotic control.
Technical Contribution
Introduced the first framework to deeply integrate short-horizon risk estimation with VLA models, significantly reducing self-collision risks. Developed a two-stage training pipeline combining synthetic and real-world data for robust risk estimator calibration. Enabled real-time risk assessment and action optimization across diverse tasks.
Novelty
CoFreeVLA is the first to incorporate explicit self-collision risk prediction into VLA models, enabling real-time safety interventions. Unlike traditional geometric planners, it is tailored for dynamic, complex environments.
Limitations
- The risk estimator heavily relies on the quality of visual embeddings, which may be affected by sensor noise.
- Recovery strategies may introduce delays in highly dynamic scenarios, impacting real-time performance.
- The framework's generalizability to more complex tasks remains untested.
Future Work
Future work includes extending CoFreeVLA to multi-robot collaboration, improving real-time performance of the risk estimator, and exploring applications in more complex tasks.
AI Executive Summary
Dual-arm robotic manipulation poses significant challenges due to the high degrees of freedom and complex spatial coordination required, with self-collision being a major risk. Existing solutions rely on geometric planning, which struggles to predict and prevent collisions in real-time. CoFreeVLA introduces a novel framework combining Vision-Language-Action (VLA) models with a short-horizon risk estimator to address this issue.
The core innovation is a lightweight risk estimator that predicts collision likelihood using visual embeddings, joint states, and candidate action sequences. Integrated into the control loop, it dynamically blocks risky actions, generates recovery trajectories, and optimizes policies for safer execution. Experiments demonstrate that CoFreeVLA significantly reduces self-collision rates (from 0.54 to 0.23) and improves task success rates (from 0.45 to 0.61) across five dual-arm tasks.
While the framework achieves notable advances in safety and task performance, it remains dependent on sensor quality and exhibits latency in highly dynamic scenarios. Future work aims to extend its applicability to multi-robot collaboration and more complex tasks, further enhancing its real-world impact.
Deep Analysis
Background
Vision-Language-Action (VLA) models have recently emerged as a powerful paradigm for robotic control, excelling in single-arm tasks. However, dual-arm systems introduce challenges like spatial coordination and self-collision risks, which remain underexplored in end-to-end VLA frameworks.
Core Problem
Dual-arm manipulation requires precise coordination to avoid self-collisions, a critical safety concern. Existing methods rely on geometric planning, which lacks real-time adaptability and fails to predict imminent collisions in dynamic environments.
Innovation
- �� CoFreeVLA integrates a short-horizon risk estimator with VLA models.
- �� The risk estimator predicts collision likelihood using visual embeddings and action sequences.
- �� Introduced a two-stage training pipeline combining synthetic and real-world data.
- �� Enables real-time risk assessment, action gating, and recovery trajectory generation.
Methodology
- �� Risk Estimator: Predicts collision likelihood, minimum distance, and time-to-collision using cross-attention and multi-task learning.
- �� Dataset Construction: Combines synthetic and real-robot data with collision labels, minimum distances, and collision times.
- �� Real-Time Control: Halts risky actions, adjusts velocities, and synthesizes recovery trajectories.
- �� Policy Optimization: Fine-tunes VLA models with risk-aware objectives to improve safety and performance.
Experiments
Experiments were conducted on an AgileX dual-arm robot across five tasks, including tool handover and precision alignment. Six VLA models, such as TinyVLA and APEX, served as baselines. Metrics included self-collision rate and task success rate, with 30 trials per task-model pair.
Results
CoFreeVLA reduced self-collision rates from 0.54 to 0.23 (57.6% decrease) and improved task success rates from 0.45 to 0.61 (34.5% increase). The framework outperformed all baselines across tasks, particularly in high-interference scenarios like pouring beans.
Applications
CoFreeVLA is ideal for dual-arm tasks requiring high precision and safety, such as industrial assembly, surgical robotics, and collaborative manufacturing. Its real-time capabilities make it suitable for dynamic environments.
Limitations & Outlook
The framework depends on high-quality visual embeddings, which may degrade under sensor noise. Recovery strategies may introduce latency in dynamic tasks. Further validation is needed for broader applications.
Plain Language Accessible to non-experts
Imagine you're using two hands to perform a delicate task, like assembling a model airplane. One hand holds a tiny piece, while the other tries to glue it. If your hands aren't coordinated, you might accidentally bump them together or drop the piece. CoFreeVLA acts like a smart assistant, watching your hands and predicting if they'll collide. If it senses a risk, it stops you, suggests a safer way, or even adjusts your movements automatically. This way, you can focus on the task without worrying about accidents.
ELI14 Explained like you're 14
Think of controlling two robot arms in a game. One arm holds a cup, and the other pours water. If they collide, you lose! CoFreeVLA is like a cheat code—it predicts when the arms might crash and stops them in time. It even adjusts their movements to avoid accidents. Cool, right? Now you can win every level without breaking a sweat!
Glossary
Vision-Language-Action (VLA) Model
Combines vision, language, and action to map instructions to robot actions.
Used to convert task instructions into dual-arm robot actions.
Self-Collision
When a robot's parts collide with each other or with objects it holds.
A critical issue in dual-arm robotic manipulation.
Risk Estimator
Predicts the likelihood of self-collision using sensory inputs and action sequences.
Central to CoFreeVLA's safety mechanism.
Short-Horizon Planning
Action planning over a short time window to ensure real-time adaptability.
Used for predicting immediate collision risks.
Recovery Trajectory
A sequence of actions to return the robot to a safe state after detecting risk.
Generated when high collision risk is detected.
Open Questions Unanswered questions from this research
- 1 How can CoFreeVLA be extended to multi-robot collaboration scenarios?
- 2 How can the real-time performance of the risk estimator be improved?
- 3 What strategies can reduce dependency on sensor quality for robustness?
Applications
Immediate Applications
Industrial Assembly
Dual-arm robots can safely and efficiently perform complex assembly tasks on production lines, reducing human intervention.
Surgical Robotics
Ensures safety in robotic surgeries by preventing self-collisions, enhancing precision and reliability.
Long-term Vision
Multi-Robot Collaboration
Potential to enable safe and efficient coordination among multiple robots for complex tasks.
Abstract
Vision Language Action (VLA) models enable instruction-following manipulation, yet their deployment on coordinated dual-arm platforms remains severely constrained by under-modeled self-collisions between manipulators and grasped objects. To address this critical safety gap, we propose CoFreeVLA, a novel framework that augments end-to-end VLA policies with a lightweight, short-horizon self-collision risk estimator. The estimator predicts collision likelihoods directly from proprioceptive states, visual embeddings, and candidate action sequences. Deeply integrated into the closed-loop control system, this estimator proactively gates risky commands, autonomously synthesizes recovery trajectories to safe states via risk-guided adjustments, and biases policy refinement for safer rollouts. To ensure robust calibration, the estimator utilizes a two-stage training pipeline, pre-training with model-based synthetic collision labels, followed by post-training on real-robot rollouts. Across five bimanual tasks, six VLA backbones, and 30 trials per variant, the task-averaged collision rate decreases from 0.54 to 0.23, while the task-averaged success rate increases from 0.45 to 0.61. Compared to representative baselines, CoFreeVLA substantially reduces self-collision frequencies and improves overall task success rates, providing a crucial step toward the safe deployment of foundational models in multi-arm continuous control.