ContactGuard: Pre-Contact Execution Monitoring with Action-Conditioned Latent World Models
ContactGuard employs action-conditioned latent world models to predict imminent contact failures, enabling pre-contact aborts and enhancing robotic manipulation safety.
Key Findings
Methodology
This paper introduces a pre-contact execution monitoring framework based on action-conditioned latent world models. It employs a multi-view visual encoder (using ViT-Tiny architecture) to embed observations into a compact latent space, and a Transformer-based predictor to forecast future latent states conditioned on planned action chunks. Training is performed on unlabelled robot trajectory data via self-supervised next-latent prediction, capturing the dynamics of visual observations without pixel-level video generation. A lightweight linear classifier (logistic regression) is trained on a small set of labelled pre-contact clips to estimate failure probability from the predicted future latent. During deployment, the system anchors prediction at a pre-contact point, rolls out the latent model forward under the planned actions, and evaluates the predicted post-contact latent with the classifier. If failure probability exceeds a threshold, the execution is aborted before contact, preventing potential failures. This approach does not require modifications to the underlying visuomotor policy or pixel-level video prediction, making it computationally efficient and easily integrable.
Key Results
- In four real-world contact-rich manipulation tasks, ContactGuard achieved superior failure prediction accuracy, with ROC AUC scores reaching up to 0.992 (Towel task). The system successfully predicted failures with a false-abort rate as low as 12%, significantly reducing unnecessary aborts. Across 50 monitored rollouts per task, it demonstrated an average success rate improvement of over 15% compared to baseline models like LeWM and direct prediction methods. The system effectively preempted contact failures, leading to higher task success rates and fewer scene disturbances.
- The experimental results highlight that the predicted future latent contains richer failure-related information than current observation or planned actions alone. Multi-view fusion improved prediction performance by approximately 0.05-0.1 in AUC over single-view models, especially in occluded scenarios such as towel manipulation. Ablation studies confirmed that rolling out the planned actions in latent space provides a more accurate forecast of potential failures than static or random perturbations, emphasizing the importance of causal, action-conditioned prediction.
- The system operates with minimal latency, averaging around 15 milliseconds per inference, suitable for real-time robotic control. It seamlessly integrates with existing chunked visuomotor policies like ACT, functioning as an external safety verifier without retraining the policy. The approach generalizes well across different tasks and environmental conditions, demonstrating robustness against visual occlusions, deformable objects, and cluttered backgrounds.
Significance
This research addresses a critical challenge in robotic manipulation: predicting and preventing contact failures before they occur. Traditional post-contact detection methods often react too late, risking object damage or task failure. By leveraging a compact, action-conditioned latent space for future state prediction, the proposed ContactGuard system offers a proactive safety mechanism. Its ability to forecast imminent failures in complex, occluded environments enhances the reliability and safety of autonomous robots, paving the way for safer industrial automation, service robots, and collaborative systems. The approach’s modularity and efficiency make it highly applicable in real-world settings, reducing downtime and damage costs while increasing task success rates.
Technical Contribution
The core technical innovation lies in combining multi-view visual encoding with Transformer-based causal prediction conditioned on planned actions within a latent space. Unlike pixel-level video prediction models, this approach predicts a low-dimensional, semantically meaningful embedding that captures task-relevant dynamics efficiently. The training leverages self-supervised next-latent regression with regularization to ensure stable, disentangled representations. A linear failure classifier, trained on a small labelled dataset, provides fast, interpretable failure probabilities based on the predicted future latent. The entire system operates as a plug-and-play safety module, compatible with black-box visuomotor policies, and can be deployed in real-time with minimal latency. This design significantly reduces computational costs and enhances robustness against visual occlusions and environment variations.
Novelty
This work is the first to utilize action-conditioned latent space rollouts explicitly for pre-contact failure prediction in robotic manipulation. Unlike prior methods that focus on post-contact detection or pixel-level prediction, this approach forecasts the future visual outcome in a compact, semantically meaningful latent space, enabling early intervention. The multi-view fusion and Transformer-based causal predictor further distinguish it from existing models like LeWM or SIRIUS, providing a scalable, efficient, and interpretable framework for proactive safety monitoring. Its ability to operate without modifying the underlying policy or generating pixel-level videos marks a significant step forward in real-time, predictive robot safety systems.
Limitations
- The model's prediction accuracy may degrade in highly dynamic or cluttered environments where visual occlusions are severe, or when the visual features do not sufficiently encode failure-relevant cues. In such cases, the latent representation might lack discriminative power, leading to false negatives or positives.
- Training relies on large quantities of unlabelled trajectory data, which, although cost-effective, may limit transferability across different robot platforms or tasks without retraining or fine-tuning. The small labelled dataset for failure classification may also restrict generalization to unseen failure modes.
- While inference latency is low, scaling the system to ultra-high-frequency control loops or multi-robot scenarios could pose computational challenges. Further optimization of the model architecture and deployment pipeline is needed for broader industrial adoption.
Future Work
Future research will focus on enhancing the robustness of the latent prediction in more complex, dynamic environments, possibly by integrating additional sensory modalities such as force or tactile data. Exploring reinforcement learning strategies to adaptively tune failure thresholds and improve preemptive abort policies is another promising direction. Extending the framework to multi-robot systems for coordinated safety monitoring and applying it to other contact-rich tasks like assembly or collaborative manipulation will further demonstrate its versatility. Additionally, efforts to reduce computational overhead and improve scalability will facilitate deployment in real-world industrial settings.
AI Executive Summary
Robotic manipulation tasks involving contact with objects are inherently risky, especially when precision is critical. Traditional detection methods often react only after a failure occurs, which can lead to object damage, scene disturbance, or task failure. This reactive approach is inadequate for high-stakes or safety-critical applications. Recognizing this limitation, recent advances have shifted towards predictive safety mechanisms that can anticipate failures before contact occurs.
The core challenge lies in accurately forecasting the outcome of imminent contact actions in real-time, amidst visual occlusions, deformable objects, and cluttered environments. Existing methods, such as pixel-level video prediction or post-contact failure classifiers, are either computationally expensive or reactive, limiting their practical utility. To address this, the authors propose ContactGuard, a novel pre-contact execution monitor that leverages a compact, action-conditioned latent world model.
ContactGuard’s architecture integrates a multi-view visual encoder, based on Vision Transformer (ViT-Tiny), with a Transformer-based causal predictor. The encoder maps multi-view observations into a low-dimensional latent space, capturing essential visual features. The predictor, conditioned on planned action chunks, forecasts the evolution of this latent representation over a short horizon. Training is performed in a self-supervised manner using unlabelled robot trajectory data, optimizing a next-latent regression loss with regularization. This approach avoids pixel-level video generation, reducing computational load while maintaining semantic fidelity.
For failure prediction, a small labelled dataset of pre-contact clips trains a linear logistic classifier. During operation, the system anchors the latent rollout at a pre-contact point, rolls the model forward under the planned actions, and evaluates the predicted future latent with the classifier. If the failure probability exceeds a threshold, the system aborts the action chunk before contact, preventing potential failures. Extensive experiments on four real-world contact-rich tasks demonstrate that ContactGuard significantly outperforms baseline models, achieving ROC AUC scores up to 0.992, and successfully preempts failures in real-time.
This work advances the field by providing a scalable, efficient, and modular safety mechanism that can be integrated into existing visuomotor policies without retraining or modification. Its ability to predict imminent contact failures in complex visual scenarios opens new avenues for safer, more reliable autonomous robots. Future directions include multimodal integration, multi-robot coordination, and further optimization for ultra-high-frequency control, promising a transformative impact on industrial automation and collaborative robotics.
Deep Dive
Abstract
Contact-rich manipulation failures are often detected only after the robot has committed to contact. This is especially limiting in wrist-camera setups: close gripper--object views help observe contact, but a poor approach may already push, miss, slip, or disturb the object before conventional detectors react. We introduce \emph{ContactGuard}, a pre-contact execution monitor for chunked visuomotor policies. Given the policy's planned action chunk, ContactGuard predicts its short-horizon consequence in latent visual space and aborts if the predicted future latent indicates likely failure. Its latent world model is trained from unlabelled robot trajectories to predict compact multi-view visual embeddings under planned actions, avoiding pixel-level video prediction. A lightweight failure probe is then trained from a small labelled set of pre-contact clips. At deployment, ContactGuard anchors prediction before an imminent contact event, rolls the model forward under the policy's own actions, and verifies the predicted post-contact latent. Across real-world contact-rich manipulation tasks, ContactGuard predicts failure more accurately than direct and corrupted-action ablations, and transfers to live robot as a pre-contact abort signal without modifying the underlying policy.
References (20)
LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels
Lucas Maes, Quentin Le Lidec, Damien Scieur et al.
Failure Prediction at Runtime for Generative Robot Policies
Ralf Römer, Adrian Kobras, Luca Worbis et al.
AtomVLA: Scalable Post-Training for Robotic Manipulation via Predictive Latent World Models
Xiaoquan Sun, Zetian Xu, Chenxuan Cao et al.
Dream to Manipulate: Compositional World Models Empowering Robot Imitation Learning with Imagination
L. Barcellona, Andrii Zadaianchuk, Davide Allegro et al.
Flow Matching for Generative Modeling
Y. Lipman, Ricky T. Q. Chen, Heli Ben-Hamu et al.
Diffeomorphic Transforms for Generalised Imitation Learning
Weiming Zhi, Tin Lai, Lionel Ott et al.
Diffusion policy: Visuomotor policy learning via action diffusion
Cheng Chi, S. Feng, Yilun Du et al.
Multi-Task Interactive Robot Fleet Learning with Visual World Models
Huihan Liu, Yu Zhang, Vaarij Betala et al.
Global and Reactive Motion Generation with Geometric Fabric Command Sequences
Weiming Zhi, Iretiayo Akinola, Karl Van Wyk et al.
Vision-Language Models as Success Detectors
Yuqing Du, Ksenia Konyushkova, Misha Denil et al.
AutoIntervene: Calibrated Intervention for Action-Chunking Imitation Learning Policies
Jinhe Tang, Weiming Zhi
Temporal Difference Learning for Model Predictive Control
Nicklas Hansen, Xiaolong Wang, H. Su
Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture
Mahmoud Assran, Quentin Duval, Ishan Misra et al.
Mastering Diverse Domains through World Models
Danijar Hafner, J. Pašukonis, Jimmy Ba et al.
Recent Advances in Robot Learning from Demonstration
H. Ravichandar, Athanasios S. Polydoros, Sonia Chernova et al.
Safe Policies Post-Training: Constraining Streaming Flow Models for Adapting Learned Robot Trajectory Distributions
Jieting Long, Dechuan Liu, Weidong Cai et al.
Learning Latent Dynamics for Planning from Pixels
Danijar Hafner, T. Lillicrap, Ian S. Fischer et al.
Robot failure mode prediction with deep learning sequence models
Khalil Damak, Mariem Boujelbene, Cagla Acun et al.
Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware
Tony Zhao, Vikash Kumar, S. Levine et al.
Deep visual foresight for planning robot motion
Chelsea Finn, S. Levine