RouterVLA: Turning Smoke Tests into Supervision for Heterogeneous VLA Selection

In robotics and autonomous systems, selecting the most suitable policy from a diverse pool is a persistent challenge. Traditional approaches rely on extensive online testing, which is costly and time-consuming, especially when deploying heterogeneous vision-language-action (VLA) policies across varied environments. Recognizing this bottleneck, the authors introduce RouterVLA, a novel framework that leverages pre-deployment smoke tests as a form of supervision, transforming the evaluation process into a resource-efficient, offline decision-making paradigm.

The core innovation lies in the concept of outcome-disjoint cross-fitting. By recording probe trials—short, controlled tests—before the actual deployment, the system constructs detailed profiles for each expert. These profiles include success rates, rollout durations, and prior knowledge, which serve as features for expert ranking. Crucially, the framework ensures that the outcome of the scored trial remains hidden during the selection process, preventing information leakage and bias. This separation allows the system to evaluate experts based solely on their probe performance, mimicking real deployment conditions.

Experimentally, the authors analyze a large-scale dataset comprising 34,752 records from the LIBERO-Plus benchmark, involving 28 experts and 398 variants. They compare simple transparent rules, such as probe-success counts, with learned models like logistic regression, gradient-boosted decision trees, and small multilayer perceptrons. The results show that a straightforward probe-success rule can elevate success rates from 46.86% to 61.49%, a significant improvement. Interestingly, learned models perform similarly, indicating that under scalar-only feature constraints, complex models do not offer substantial advantages. The study also reveals that reusing the scored trial within the same evaluation inflates the measured gain by 1.87 times, underscoring the importance of outcome separation for credible evaluation.

These findings have profound implications. They demonstrate that limited probe budgets—just three tests per expert—are sufficient to capture most of the conditional value, enabling resource-efficient expert selection. Moreover, the outcome-disjoint design prevents overestimation of performance, ensuring that offline evaluations reliably predict real-world effectiveness. This approach offers a scalable solution for deploying heterogeneous policy pools in robotics, reducing costs and increasing robustness.

Looking ahead, the authors suggest extending the framework to incorporate richer features, adaptive probing strategies, and online learning mechanisms. Such advancements could further improve the system’s ability to handle complex, dynamic environments and multi-task scenarios. Overall, RouterVLA provides a foundational step toward autonomous, scalable, and reliable robot policy management, with potential applications spanning industrial automation, autonomous vehicles, and beyond. Its emphasis on outcome separation and resource-efficient evaluation marks a significant milestone in the evolution of intelligent robotic systems.