R2S-Eval: Robot Evaluation with Real-to-Sim Calibration via Vision-Language Models
R2S-Eval combines real-to-sim calibration with VLM preference evaluation for stable robot behavior ranking.
Key Findings
Methodology
This approach integrates environment calibration, generating simulated videos via action replay in a calibrated simulator. It then employs pre-trained vision-language models (e.g., Qwen3-VL, LLaVA-OV, Gemma) to analyze videos, produce structured descriptions, and compare behaviors. Preference matrices are constructed from pairwise comparisons, modeled with Bradley–Terry, to derive policy rankings. This framework reduces hardware trials, improves stability, and captures nuanced behavior differences beyond success metrics.
Key Results
- Across simulation and real-world tests, R2S-Eval yields policy rankings with high agreement to human preferences, achieving a mean Spearman’s ρ of 0.823 and preference consistency over 82%.
- Compared to success-rate metrics, it reveals behavioral quality differences, demonstrating robustness and efficiency, with significant reduction in hardware effort.
- In multi-task experiments, the preference-based rankings align well with human judgments, validating its practical utility in real robot scenarios.
Significance
This work addresses the limitations of success-rate-based evaluation by introducing a preference-driven, behavior-aware framework. It enhances stability, informativeness, and automation in robot policy assessment, facilitating faster development cycles and more reliable performance measurement. The methodology bridges the gap between simulation and reality, enabling scalable evaluation crucial for autonomous robot learning and deployment.
Technical Contribution
The key innovations include: 1) environment calibration techniques ensuring simulation fidelity; 2) structured video descriptions and multi-modal preference models for behavior assessment; 3) statistical ranking via Bradley–Terry, avoiding reliance on coarse success metrics. These advancements enable automated, detailed, and stable evaluation of complex robotic behaviors, surpassing prior approaches limited to success detection or scalar scores.
Novelty
This is the first integration of real-to-sim calibration with large-scale vision-language preference models for robot behavior evaluation. Unlike prior work focusing on success detection or single-metric scoring, this framework captures detailed behavioral nuances, providing a comprehensive and automated assessment tool that aligns closely with human judgment.
Limitations
- Calibration accuracy depends on environment reconstruction quality; residual discrepancies may affect video fidelity and preference judgments.
- VLMs may struggle with highly complex or dynamic scenes, limiting their effectiveness in certain scenarios.
- The approach requires substantial computational resources for simulation and model inference, which may hinder real-time deployment in some applications.
Future Work
Future directions include improving environment calibration techniques, developing end-to-end learned preference models, and extending the framework to multi-robot systems and dynamic environments. Enhancing model robustness and reducing computational costs will further facilitate real-world deployment and broader adoption.
AI Executive Summary
The rapid advancement of generalist manipulation models, especially vision-language-action (VLA) systems, has propelled robots closer to autonomous, versatile operation in real-world settings. However, evaluating these policies remains a challenge. Traditional success-rate metrics, while simple, are coarse and often unstable, failing to reflect the nuanced quality of robot behaviors. Manual evaluations, on the other hand, are labor-intensive and subjective. To address these issues, this paper introduces R2S-Eval, an innovative evaluation pipeline that combines environment calibration, simulation, and vision-language preference modeling.
The core idea is to generate behavior videos in a calibrated simulation environment, which closely mimics real-world conditions. These videos are then analyzed by pre-trained vision-language models, which produce structured descriptions and compare behaviors pairwise. The preference judgments are aggregated using the Bradley–Terry model, resulting in a stable and behavior-aware ranking of policies. This approach significantly reduces the need for repeated hardware trials, enhances evaluation stability, and captures behavioral subtleties beyond binary success labels.
Extensive experiments in both simulated and real-world settings demonstrate the effectiveness of R2S-Eval. The method achieves high agreement with human preferences, with correlation coefficients exceeding 0.82, and preference consistency over 82%. It also reveals differences in behavior quality that traditional success metrics overlook, providing richer insights into policy performance. The framework's automation and stability make it a promising tool for robot development, enabling faster iteration and more reliable benchmarking.
Despite its strengths, the approach faces limitations such as calibration accuracy and model understanding in complex scenes. Future work aims to refine environment reconstruction, develop end-to-end preference learning, and extend applicability to multi-robot systems. Overall, R2S-Eval represents a significant step toward automated, behavior-centric robot evaluation, with broad implications for advancing autonomous manipulation technologies.
Deep Dive
Plain Language Accessible to non-experts
想象你在学校参加一个机器人比赛，老师只看你赢没赢，但其实比赛中你做得很漂亮，动作很酷，只是没赢而已。传统的评价就像只看你赢没赢，没有看到你平时的努力和精彩表现。这篇文章发明了一个特别聪明的老师，他可以看比赛的视频，自动判断哪个同学表现得更棒，不用你亲自解释。这个老师还会用模拟的比赛场景，帮你多次练习，确保评价公平又准确。这样一来，不仅节省了很多时间，还能更公正地看到每个人的真实水平。未来，这个系统还能帮机器人学习得更快，让它们在各种任务中都表现得更好，就像你在学校变得越来越厉害一样。
ELI14 Explained like you're 14
想象你在学校里参加比赛，老师只看你是不是赢了，但其实比赛中你做得很棒，动作很酷，只是没赢而已。传统的评价就像只看你赢没赢，没有看到你平时的努力和精彩表现。这篇文章发明了一个特别聪明的老师，他可以看比赛的视频，自动判断哪个同学表现得更棒，不用你亲自解释。这个老师还会用模拟的比赛场景，帮你多次练习，确保评价公平又准确。这样一来，不仅节省了很多时间，还能更公正地看到每个人的真实水平。未来，这个系统还能帮机器人学习得更快，让它们在各种任务中都表现得更好，就像你在学校变得越来越厉害一样。
Abstract
Evaluating robot manipulation policies is becoming increasingly important as generalist models, particularly vision-language-action (VLA) models, are deployed on physical robots. However, conventional real-world evaluation remains labor-intensive, unstable, and insufficiently informative. It requires repeated hardware trials, manual scene resets, and continuous operator monitoring, may produce different policy rankings across repeated evaluations, and primarily relies on success-rate metrics that provide limited information about execution quality. In contrast, humans assess robot performance by observing and comparing complete behaviors rather than relying solely on binary success outcomes. To this end, we propose R2S-Eval, an evaluation pipeline that combines real-to-sim calibration with vision-language model (VLM) preference evaluation. The real-to-sim component efficiently generates rollout videos in a simulator calibrated to the real-world evaluation setting, thereby reducing the need for repeated hardware trials. The VLM evaluator assesses the execution quality of rollout videos and produces pairwise preferences, which are subsequently aggregated into policy rankings. We further introduce a protocol to assess whether the proposed evaluation pipeline yields validated policy conclusions while mitigating the key challenges of conventional real-world evaluation. Experiments in both simulation and real-world settings demonstrate that R2S-Eval produces reliable and stable policy conclusions, achieves agreement with human preferences, substantially reduces repeated hardware-operation effort, and reveals behavior-quality differences that are not captured by binary success labels. In general, R2S-Eval advances robot evaluation from manual success counting toward automated, statistically stable, and quality-aware evaluation of robot behavior. Project page: https://r2s-eval.github.io.