AgenticFocus: Object-Preserving Mixed Reality Synthesis from Human FPV Video for Dexterous Humanoid Learning
AgenticFocus converts human FPV videos into robot-trainable demonstrations with a SPARC score of -5.18.
Key Findings
Methodology
AgenticFocus employs a Mixed Reality synthesis pipeline to convert ordinary first-person-view human videos into robot-trainable demonstrations. This method includes restoring occluded object geometry, reconstructing full-hand motion, and retargeting it to a humanoid embodiment through camera-relative alignment and layered compositing. The resulting dataset pairs focused visual observations with synchronized robot actions and states.
Key Results
- AgenticFocus achieves lower trajectory error and smoother wrist motion than cross-embodiment baselines, with a SPARC score of -5.18 compared to -5.56 for Masquerade and -6.05 for Do as I Do.
- Camera-relative alignment and full-hand retargeting improve spatial matching over baseline human-to-robot transfer pipelines.
- Experiments show AgenticFocus maintains the least negative SPARC across 75 validation episodes, indicating superior wrist motion smoothness.
Significance
AgenticFocus lowers the data barrier for humanoid manipulation by transforming ordinary human videos into robot-trainable supervision. It eliminates the need for specialized capture hardware, making it applicable in domains like household tasks, assistive and service robotics. This method provides an accessible data source for robot-learning research and education, contributing to scalable, hardware-free manipulation demonstration collection for the embodied-AI community.
Technical Contribution
AgenticFocus distinguishes itself technically through mixed reality synthesis, full-hand cross-embodiment retargeting, and a layered compositing strategy. It offers new engineering possibilities, particularly in generating training-ready visual-action data without specialized hardware.
Novelty
AgenticFocus is the first to convert ordinary human videos into humanoid-usable visual-action supervision. Compared to existing methods, it addresses spatial matching and motion smoothness through camera-relative alignment and full-hand retargeting.
Limitations
- In complex scenes, object occlusion and motion blur may lead to inaccurate reconstructions.
- High-quality input videos are required to ensure output accuracy.
Future Work
Future directions include extending to more embodiments and scenarios, and applying in downstream policy training.
AI Executive Summary
AgenticFocus employs a Mixed Reality synthesis pipeline to convert ordinary first-person-view human videos into robot-trainable demonstrations. Existing methods struggle with hand-object occlusion, motion simplification, or require specialized capture hardware. AgenticFocus addresses these issues by restoring occluded object geometry, reconstructing full-hand motion, and retargeting it to a humanoid embodiment through camera-relative alignment and layered compositing. Experimental results show that AgenticFocus achieves lower trajectory error and smoother wrist motion than cross-embodiment baselines, with a SPARC score of -5.18 compared to -5.56 for Masquerade and -6.05 for Do as I Do.
The significance of this method lies in its ability to lower the data barrier for humanoid manipulation, making it applicable in domains like household tasks, assistive and service robotics. By eliminating the need for specialized capture hardware, AgenticFocus provides an accessible data source for robot-learning research and education, contributing to scalable, hardware-free manipulation demonstration collection for the embodied-AI community.
However, AgenticFocus may face challenges in complex scenes where object occlusion and motion blur lead to inaccurate reconstructions. Future work includes extending to more embodiments and scenarios, and applying in downstream policy training.
Deep Analysis
Background
Recent advances in Physical AI have highlighted the need for better supervision. While Vision-Language-Action policies rely on large amounts of robot-usable visual and action data, such data remains difficult to obtain for dexterous humanoids. Existing methods like DexCap and EgoDex demonstrate the value of collecting high-quality supervision using specialized hardware, but these approaches still rely on complex capture systems or scene-specific reconstructions.
Core Problem
Converting human video into robot data requires bridging three main gaps: viewpoint, interaction-region, and action grounding. Human demonstrations are typically captured from head- or chest-mounted cameras, whereas humanoid observations are defined in robot-centered frames. Additionally, the hand frequently occludes the manipulated object, and conventional actor removal or generative editing often fails in these regions.
Innovation
AgenticFocus addresses these challenges through a Mixed Reality synthesis pipeline. First, it identifies and tracks task-relevant objects, restoring stable object templates after human removal. Second, it reconstructs full-hand motion and retargets it to a humanoid embodiment through a camera-relative formulation. Finally, it composes the restored scene and retargeted robot through layered rendering, providing visually stable interactions.
Methodology
- �� Identify and track task-relevant objects using a SAM-style segmentation model. • Remove human actor and reconstruct background using E2FGVI-HQ. • Reconstruct full-hand motion using WiLoR/HaMeR modules. • Retarget motion to humanoid embodiment in a camera-relative frame. • Layered rendering to compose restored scene and retargeted robot.
Experiments
Experiments use datasets like EPIC-KITCHENS, standardized to 30 FPS. Comparisons are made with Masquerade and Do as I Do, with all trajectories resampled in the same coordinate frame. Evaluation metrics include 3D position error and SPARC score.
Results
AgenticFocus achieves lower trajectory error and smoother wrist motion than baselines, with a SPARC score of -5.18. Experiments show AgenticFocus maintains the least negative SPARC across 75 validation episodes, indicating superior wrist motion smoothness.
Applications
AgenticFocus is applicable in domains like household tasks, assistive and service robotics. It provides an accessible data source for robot-learning research and education, contributing to scalable, hardware-free manipulation demonstration collection.
Limitations & Outlook
In complex scenes, object occlusion and motion blur may lead to inaccurate reconstructions. High-quality input videos are required to ensure output accuracy. Future directions include extending to more embodiments and scenarios, and applying in downstream policy training.
Plain Language Accessible to non-experts
Imagine you're cooking in a kitchen. You're wearing special glasses that record everything you do. AgenticFocus is like a smart assistant that turns these videos into tutorials a robot can learn from. It removes your hands from the video, leaving only the objects you're handling, and replaces them with robot hands. This way, the robot learns to cook like you without needing constant guidance.
ELI14 Explained like you're 14
Imagine you're playing a super cool game where your mission is to teach a robot to do things like you. AgenticFocus is like a magic tool that records your moves in the game and then swaps your hands with robot hands. This way, the robot learns your moves without you having to guide it all the time. Isn't that awesome?
Glossary
Mixed Reality
Mixed Reality is a technology that combines elements of the real world and virtual world to create a new environment.
Used in this paper to convert human videos into robot-trainable demonstrations.
SPARC
SPARC is a metric for quantifying movement smoothness, with less negative values indicating smoother motion.
Used to evaluate the smoothness of robot motion trajectories generated by AgenticFocus.
Camera-Relative Alignment
A technique for retargeting human motion to a robot embodiment, maintaining spatial consistency.
Used to address the viewpoint gap between human videos and robot observations.
Layered Compositing
A compositing technique that maintains depth cues in interaction regions through layered rendering.
Used to ensure visual stability in synthesized robot demonstrations.
EPIC-KITCHENS
A large-scale first-person video dataset capturing everyday activities in kitchens.
One of the datasets used to evaluate AgenticFocus.
Open Questions Unanswered questions from this research
- 1 How to accurately reconstruct occluded object geometry in complex scenes? Current methods may not be precise enough for complex occlusions.
- 2 How to maintain output accuracy with low-quality video inputs?
- 3 How to extend AgenticFocus to more embodiments and scenarios?
Applications
Immediate Applications
Household Robots
AgenticFocus can help household robots learn to handle everyday objects, such as washing dishes and organizing items.
Assistive Robots
In healthcare and eldercare, AgenticFocus can help robots learn complex caregiving tasks.
Long-term Vision
Education and Research
AgenticFocus can serve as a tool for robot-learning research and education, providing an accessible data source.
Abstract
Human egocentric video is a scalable supervision source for humanoid policy learning, but current pipelines struggle with hand-object occlusion, oversimplified motion, or specialized capture hardware. We introduce AgenticFocus, a Mixed Reality synthesis pipeline that converts ordinary first-person-view human videos into robot-trainable demonstrations by restoring occluded object geometry, reconstructing full-hand motion, and retargeting it to a humanoid embodiment through camera-relative alignment and layered compositing. The resulting dataset pairs focused visual observations with synchronized robot actions and states. AgenticFocus achieves lower trajectory error and smoother wrist motion than cross-embodiment baselines, with SPARC scores of -5.18 versus -5.56 and -6.05.