OC-VLA++: Monocular Geometry-Guided Cross-View Consistency for Viewpoint-Robust Robotic Manipulation
OC-VLA++ enhances viewpoint robustness in monocular robotic manipulation via geometry-guided cross-view supervision and action equivariance, achieving 8-15% success rate improvements under large camera shifts.
Key Findings
Methodology
OC-VLA++ extends OC-VLA by integrating geometry-guided paired-view supervision and explicit cross-view action equivariance. The approach reconstructs local 3D geometry from monocular images using a pretrained monocular depth estimator (e.g., MoGe-2), then synthesizes nearby views via projection. Paired views are processed through the same generative model (diffusion or flow-matching), predicting actions in their respective camera frames. These predictions are transformed into a shared robot coordinate frame, where a cross-view equivariance loss enforces consistency. This process explicitly models how actions should transform across viewpoints, improving generalization to unseen camera poses.
Key Results
- In real-robot experiments, OC-VLA++ outperformed OC-VLA, with success rates increasing by approximately 8-15 percentage points at maximum viewpoint shifts. For instance, success rates reached 48.3% versus 40.8% for OC-VLA under the largest camera displacement. In simulation, the success rate improved by about 4.4%, demonstrating robustness across diverse tasks such as stacking and pouring. Ablation studies confirmed that both geometric paired-view supervision and cross-view equivariance contribute significantly to these gains.
- Across multiple datasets and architectures, OC-VLA++ consistently maintained higher success rates under large viewpoint changes, showing slower degradation compared to baseline methods. The method also preserved in-distribution performance, indicating no compromise on standard task execution while enhancing out-of-distribution generalization.
- Qualitative results illustrated that the model could adapt to unseen camera poses, accurately predicting actions despite large shifts, validating the effectiveness of the geometric regularization.
Significance
This work addresses a critical challenge in robotic manipulation: achieving robust generalization across viewpoints with limited training data. By explicitly modeling the geometric transformation of actions, OC-VLA++ reduces reliance on appearance cues tied to specific views, enabling robots to operate reliably in dynamic, real-world environments. The approach bridges the gap between monocular perception and precise control, facilitating deployment in scenarios where multiple cameras or dense demonstrations are impractical. Its success paves the way for more adaptable autonomous systems in industrial, service, and home robotics, where viewpoint variability is inevitable.
Technical Contribution
The paper introduces a novel training framework that combines monocular geometry reconstruction, view synthesis, and action equivariance regularization. It leverages a pretrained monocular depth estimator to generate paired views, then enforces that action predictions from these views, once transformed into a common robot frame, are consistent. This explicit geometric supervision guarantees that the learned policy respects the underlying physical transformations, providing a theoretical guarantee of action invariance across viewpoints. The framework is compatible with multiple generative models, including diffusion and flow-matching, broadening its applicability. This represents a significant step beyond prior methods that relied solely on appearance augmentation or implicit regularization, offering a principled approach grounded in geometric invariance.
Novelty
OC-VLA++ is the first to explicitly incorporate geometric relations between paired views into the training of monocular manipulation policies, enforcing action equivariance directly in the robot coordinate frame. Unlike previous approaches that primarily relied on data augmentation or camera conditioning, this method models the actual transformation of actions across viewpoints, providing a strong theoretical basis for generalization. Its integration of monocular depth estimation, view synthesis, and equivariance regularization is a novel combination that significantly advances the state-of-the-art in viewpoint-robust manipulation.
Limitations
- The approach depends heavily on the accuracy of monocular depth estimation; errors in depth can impair view synthesis and geometric relations, reducing effectiveness.
- In scenarios with extreme viewpoint shifts or complex occlusions, the geometric assumptions may break down, limiting robustness.
- Offline view synthesis introduces computational overhead, and real-time deployment requires further optimization.
- The method’s performance in highly dynamic or cluttered environments remains to be validated, as current experiments focus on controlled settings.
Future Work
Future directions include integrating end-to-end training of depth estimation and view synthesis modules for improved robustness, extending the framework to multi-robot systems, and exploring online view adaptation. Additionally, combining this approach with multi-modal inputs such as tactile or auditory signals could further enhance manipulation robustness in unstructured environments.
AI Executive Summary
Robotic manipulation from monocular visual inputs faces a persistent challenge: how to ensure reliable performance when the camera viewpoint changes. Traditional models trained on fixed or limited views tend to overfit, struggling to generalize to new perspectives. This limitation hampers real-world deployment, where robots often encounter diverse and unpredictable camera angles. To address this, the recent paper OC-VLA++ introduces a novel framework that leverages geometric principles to improve viewpoint robustness.
The core idea is to generate paired views of the same scene from different angles using monocular depth estimation and view synthesis. These paired observations are processed through a shared generative model, which predicts actions in their respective camera frames. The key innovation is transforming these predictions into a common robot coordinate system and enforcing a cross-view equivariance loss. This loss ensures that actions predicted from different views are consistent after geometric transformation, effectively teaching the model how actions should change with viewpoint.
Experimental results in both real and simulated environments demonstrate the effectiveness of OC-VLA++. Under large camera shifts, success rates improved by 8-15 percentage points compared to baseline methods, with success rates reaching nearly 50% at the largest displacements. The approach maintains high in-distribution performance while significantly enhancing out-of-distribution generalization. Ablation studies confirm that the geometric paired-view supervision and action equivariance are critical to these gains.
This work advances the field by providing a principled way to incorporate geometric invariance into monocular manipulation policies, making robots more adaptable and reliable in complex, real-world scenarios. Future research will focus on end-to-end training, online view adaptation, and multi-modal integration, broadening the impact of this promising approach.
Deep Dive
Abstract
We propose OC-VLA++, an extension of OC-VLA for viewpoint generalization under limited camera coverage. While OC-VLA grounds robot actions in the camera coordinate system to align action supervision with visual observations, camera-space grounding alone can still overfit to the few viewpoints observed during training. OC-VLA++ addresses this limitation by introducing geometry-guided paired-view supervision and an explicit cross-view action-equivariance objective. Given paired observations of the same manipulation scene from geometrically related viewpoints, the model is trained such that their camera-space predictions correspond to the same robot-frame action. This objective explicitly supervises how action predictions should transform across viewpoints, rather than relying solely on image-level augmentation. Experiments demonstrate substantial improvements in unseen-view generalization under limited camera coverage, with performance degrading more gracefully under increasing camera displacement. These results establish cross-view action equivariance as an effective complement to observation-centric action grounding for robust real-world deployment.