Video2DoorTraversal: Push Door Traversal via Simulated Door Twins
Proposes Video2DoorTraversal, reconstructing instance-aligned, simulation-ready doors from a single RGB video, achieving 96.57% success in real-world door traversal.
Key Findings
Methodology
The approach employs DoorTwin to reconstruct a metrically accurate, articulated door model from a single RGB frame, integrating geometric grounding, procedural program generation, and appearance refinement. The simulation-in-the-loop agent then synthesizes and iteratively refines parameterized expert trajectories based on the reconstructed door, ensuring physical plausibility. ArticuACT, a dual-depth policy conditioned on robot-centric Plücker ray maps, predicts coordinated base, arm, and gripper commands for door opening and traversal. All modules operate onboard, enabling real-time, closed-loop control. The system leverages geometric constraints, visual critique, and appearance transfer to ensure instance fidelity and robustness.
Key Results
- The system achieved an average success rate of 96.57% across five real doors, with an average completion time of 13 seconds, outperforming prior methods like DoorGym (12.5%) and demonstrating high efficiency and reliability. Zero-shot generalization on unseen but structurally similar doors reached 80.95%, indicating strong transferability. The reconstructed doors closely matched ground truth geometry and appearance, validated through quantitative metrics such as Scale Score and mIoU, and qualitative visual comparisons. Extensive simulation tests confirmed robustness under varying door sizes, handle positions, and material properties, validating the approach's adaptability.
Significance
This work advances autonomous robotic manipulation by enabling fully perception-driven, instance-specific door traversal from minimal input. It bridges the gap between perception, simulation, and real-world execution, reducing reliance on prebuilt models or scene scans. The high success and generalization rates demonstrate potential for deployment in domestic, industrial, and emergency scenarios, where robots must adapt to unknown environments. The integration of geometric reconstruction, simulation-based expert refinement, and onboard policy inference sets a new standard for end-to-end autonomous manipulation, fostering progress toward truly intelligent, adaptable robots.
Technical Contribution
The key technical innovations include the DoorTwin pipeline for single-video instance reconstruction, combining geometric grounding with procedural validation and appearance transfer. The simulation-in-the-loop expert trajectory generator employs parameterized skills and iterative diagnosis, ensuring physical feasibility. The ArticuACT policy leverages dual-depth inputs with robot-centric Plücker conditioning and auxiliary interaction prediction, enabling coordinated, contact-aware control. The entire system operates fully onboard, enabling real-time execution without external scene information, representing a significant step beyond prior work that relied on external scans or offline training.
Novelty
This is the first work to reconstruct a detailed, instance-specific, articulated door model directly from a single RGB video, and to utilize this model for closed-loop, real-world door traversal without external scene scans. The integration of geometric grounding, procedural generation, visual critique, and onboard policy inference creates a fully autonomous pipeline from perception to action. Unlike prior methods limited to simulation or static scene understanding, this approach achieves robust, zero-shot generalization to unseen doors, marking a new milestone in vision-based robotic manipulation.
Limitations
- The current system struggles with non-rigid or highly deformable doors, as the geometric assumptions and rigid joint models may not hold. When doors exhibit large angular deviations or complex motion patterns, reconstruction and control can fail. Additionally, reflective or transparent materials pose challenges for depth estimation, affecting model accuracy. Environmental factors such as extreme lighting, occlusion, or clutter can impair perception and robustness. Computational complexity and inference latency also limit real-time performance in highly dynamic scenarios. Extending to multiple doors or complex door mechanisms remains an open challenge.
Future Work
Future efforts will focus on enhancing perception robustness through multi-modal sensing, such as LiDAR or tactile feedback, and extending the framework to handle non-rigid and articulated doors with complex dynamics. Improving computational efficiency and real-time inference capabilities is also a priority. Additionally, scaling the system to multi-door scenarios and integrating higher-level semantic understanding will broaden its applicability. Research into learning-based geometric priors and adaptive control strategies could further improve robustness and generalization in diverse, unstructured environments.
AI Executive Summary
This research introduces Video2DoorTraversal, a novel framework enabling autonomous door opening and traversal solely from a single RGB video. The core challenge addressed is the translation of visual perception into physically executable manipulation strategies in unstructured environments. Traditional approaches often rely on prebuilt models, scene scans, or extensive manual annotations, limiting adaptability and scalability. In contrast, the proposed system leverages a pipeline that reconstructs a detailed, instance-specific, articulated door model—termed DoorTwin—from raw video input. This reconstruction integrates geometric grounding, procedural validation, and appearance transfer, ensuring high fidelity to the observed door.
Building upon this, the system employs a simulation-in-the-loop agent that generates and iteratively refines expert trajectories based on the reconstructed model. These trajectories are parameterized, enabling flexible adjustments and robust optimization through simulation diagnosis. The learned policy, ArticuACT, then predicts coordinated control commands for the robot’s base, arm, and gripper, conditioned on dual-depth inputs and robot-centric Plücker ray maps. This design ensures precise geometric grounding and contact-aware control, crucial for contact-rich tasks like door traversal.
Extensive experiments demonstrate the system’s effectiveness. In real-world tests, it achieves a success rate of 96.57%, completing door traversal in approximately 13 seconds. Remarkably, it maintains an 80.95% success rate on unseen but structurally similar doors, indicating strong generalization. These results surpass existing methods like DoorGym and UniDoorManip, showcasing the approach’s robustness and practical viability. The entire pipeline operates onboard, emphasizing real-time deployment potential.
This work significantly advances autonomous manipulation, offering a scalable, perception-driven solution for complex loco-manipulation tasks. Its ability to learn from minimal visual input and generalize across environments opens new avenues for service robots, industrial automation, and emergency response systems. Future directions include multi-modal perception integration, handling dynamic and deformable doors, and scaling to multi-door scenarios, aiming to realize fully autonomous, adaptable robots capable of operating in diverse, real-world settings.
Deep Dive
Plain Language Accessible to non-experts
想象你在家里，有一扇门挡在面前。你想让机器人帮你开门，但你没有告诉它门的具体信息，只给它拍了一段视频。机器人就像一个聪明的学生，看完视频后，能理解门的形状、把手的位置和运动方式。然后，它会走到门前，伸手抓住把手，转动一下，再用力推开门，就像你用钥匙开门一样。整个过程不需要提前告诉它门的详细参数，只用一段视频，它就能学会怎么操作。这个系统就像一个会看照片就会开门的机器人助手，非常智能和方便。
ELI14 Explained like you're 14
你有没有想过，未来的机器人可以只看一段门的照片，就学会怎么开门？就像你用手机拍一张门的照片，然后机器人就能理解门的样子和把手在哪里。它会自己走到门前，伸出手抓住把手，转动一下，然后用力推开门，最后走过去。这就像你用遥控器开门一样，但机器人不用遥控器，只用看视频学会了。这个系统特别厉害，因为它只需要一段视频，不需要提前告诉它门的具体信息，就能自己学会怎么开门。它还能在不同的门上都试试，表现都很好，就像一个聪明的小帮手一样。
Abstract
Door opening and traversal is a long-horizon loco-manipulation task that requires precise handle interaction and coordinated base-arm control. We present Video2DoorTraversal, a single-video real-to-sim-to-real framework for wheel-legged mobile manipulators. Given one RGB video of a real door, DoorTwin reconstructs an instance-aligned, articulated, and simulation-ready door twin with realistic geometry and appearance. A simulation-in-the-loop agent converts the recovered articulation into a parameterized skill program and iteratively refines failed rollouts to generate physically executable demonstrations. These demonstrations are used to train ArticuACT, a dual-depth policy that predicts coordinated base, arm, and gripper commands using robot-centric camera conditioning and interaction-aware supervision. With all perception and policy inference running onboard, the system achieves a 96.57% average success rate across five real doors and an 80.95% zero-shot success rate on structurally similar unseen doors, while completing the full approach, opening, and traversal sequence in approximately 13s on average. Project Page: https://video2doortraversal.github.io/.