How VLAs Fail Differently: Black-Box Action Monitoring Reveals Architecture-Specific Failure Signatures

The evolution of robotic control has transitioned from rule-based systems to deep learning models capable of end-to-end autonomous operation. Architectures such as VQ-VAE (vector quantized variational autoencoders), diffusion models, and autoregressive policies like ACT have demonstrated remarkable capabilities in generating complex motions from visual and linguistic inputs. Notable prior works include OpenVLA, LeRobot, and pi0, which integrate multi-modal perception with action generation. Despite these advances, the safety aspect—particularly the prediction and prevention of motion failures—has been underexplored. Traditional safety measures rely on boundary checks and velocity limits, which are often insufficient for the nuanced failure modes of modern generative models. As models grow in complexity, failure signatures become more subtle and architecture-dependent, necessitating new monitoring paradigms that can adapt to these differences without requiring access to internal model parameters.

The core challenge addressed in this paper is the lack of systematic understanding of how different VLA architectures fail at the motor command level. Existing safety mechanisms are largely generic and do not account for the distinct failure signatures arising from different model structures. This leads to ineffective failure prediction, risking task failure or unsafe behaviors during deployment. The problem is compounded by the fact that many current systems operate as black boxes, with limited visibility into their internal states. Therefore, developing architecture-aware, model-agnostic monitoring strategies that can reliably predict failures in real-time is crucial. The difficulty lies in identifying robust, architecture-specific failure signatures that can be monitored externally, without retraining or model access, and that can generalize across tasks and environments.

This work introduces several key innovations. First, the SafeContract framework combines conformal calibration with CUSUM change detection to create a robust, training-free monitoring system that guarantees statistical coverage of monitored signals. Second, it systematically compares failure signatures across three distinct architectures—VQ-BeT (discrete token-based), Diffusion Policy (continuous denoising), and ACT (action chunking)—highlighting their fundamental differences. Third, it identifies that failure predictors such as reversal rate are architecture-independent, while jerk and velocity violations are architecture-dependent. Fourth, the paper proposes architecture-matched monitor selection, tailoring specific metrics to each model type, which significantly enhances failure prediction accuracy. These innovations collectively enable scalable, effective safety monitoring without internal model access, addressing a critical gap in current robotic safety research.

• �� Selection of monitoring metrics: Based on the understanding of action generation mechanisms, five key metrics are chosen—reversal rate, jerk (second derivative of action), spectral energy, total variation, and momentum coherence.
• �� Conformal calibration: Using an 80/20 split of demonstration episodes, non-conformity scores are computed on the calibration set. Bounds are constructed to guarantee 97.9% coverage at α=0.05, ensuring statistical reliability.
• �� Real-time change detection: CUSUM detectors are applied to each metric, with thresholds set to detect significant shifts indicative of failures.
• �� Architecture-specific monitor design: For discrete models, jerk and reversal rate are prioritized; for continuous models, reversal rate and momentum coherence are emphasized.
• �� Experimental evaluation: All three architectures are tested on identical tasks, with 450 episodes each. Failure signatures are analyzed via AUROC scores, comparing the predictive power of each metric.
• �� Statistical analysis: AUROC scores are computed for each metric, and significance tests confirm the architecture dependence of failure signatures.
• �� Deployment validation: SafeContract enforces safety bounds during execution, with violations logged and analyzed to assess the impact on task success.

The experiments are conducted on two main platforms: PushT, a 2D manipulation task, and ALOHA, a 14-DOF bimanual manipulation task. All models—VQ-BeT, Diffusion Policy, and ACT—are evaluated under identical conditions, with 200 episodes per architecture for PushT and 50 for ALOHA. The models are tested with the same random seeds and safety bounds, ensuring comparability. SafeContract monitors are integrated without retraining, providing real-time detection of violations. Failure signatures are analyzed through AUROC metrics across multiple monitoring signals. The experiments also include ablation studies, removing or replacing specific metrics to assess their contribution. Results demonstrate architecture-dependent failure signatures and the effectiveness of tailored monitoring strategies in predicting task failures.

The experimental results confirm that reversal rate is a robust, architecture-independent predictor, with AUROC scores exceeding 0.79 across all models. Jerk monitoring is highly predictive for VQ-BeT (AUROC=0.88) but drops to chance levels for diffusion models. Velocity violations, despite being the default safety check, show AUROC scores below 0.7 and sometimes below chance, indicating poor failure prediction capability. The tailored, architecture-matched monitoring strategies significantly outperform generic approaches, achieving AUROC scores above 0.9 for key failure signatures. The experiments also reveal qualitative differences: VQ-BeT exhibits jerky, oscillatory failures, while diffusion models tend to stall with smooth trajectories. These insights validate the importance of architecture-aware monitoring for safe deployment.

The proposed monitoring framework can be directly integrated into existing robotic control systems, especially those employing deep generative models. It enables real-time failure prediction without modifying the underlying models, making it suitable for industrial automation, service robots, and autonomous vehicles. By automatically selecting architecture-specific monitors, it reduces the need for manual tuning and expert intervention. Additionally, the framework can be used in robot debugging and maintenance, providing insights into failure modes and guiding model improvements. In the long term, combining such motion-level safety monitoring with perception and task-level checks can lead to comprehensive, multi-layered safety systems capable of operating reliably in complex, dynamic environments.

The current experiments are limited to simulated environments, which may not fully replicate real-world noise, sensor inaccuracies, and environmental variability. The generalization of identified failure signatures to physical robots remains to be validated. The study focuses on three main architecture types; other emerging models, such as hybrid or transformer-based architectures, may exhibit different failure modes. The monitoring metrics primarily target motion-level failures, leaving perception, reasoning, and semantic errors unaddressed. Additionally, the approach assumes static safety bounds, which may need adaptation for dynamic or uncertain environments. Future work should include real-robot validation, adaptive calibration methods, and multi-modal safety integration to address these limitations.