Dexterity from Smart Lenses: Multi-Fingered Robot Manipulation with In-the-Wild Human Demonstrations
Proposes AINA, a framework using smart glasses to learn multi-finger robot manipulation from in-the-wild human videos without robot data.
Key Findings
Methodology
AINA integrates high-resolution RGB, stereo depth, and precise hand pose estimation from Aria Gen 2 glasses. It collects diverse outdoor human demonstrations and a single in-scene demo for spatial calibration. Using object segmentation, tracking, and space alignment via Kabsch algorithm, the method transforms raw videos into consistent 3D point clouds and keypoints. A transformer-based point cloud encoder and Vector Neuron MLP predict future fingertip trajectories, enabling end-to-end supervised learning without simulation or reinforcement learning. The trained policies are deployed on a Kinova Gen3 robot with a prosthetic hand, using inverse kinematics for control.
Key Results
- In nine manipulation tasks, AINA achieved an average success rate of 85%, outperforming baselines like scene-only (70%) and wild-only (65%). It demonstrated robustness in cluttered backgrounds and object variations, successfully completing tasks such as door opening, pouring, and wiping. The approach showed strong generalization across different environments and objects, with minimal demonstration effort (~15 minutes).
- Compared to prior imitation learning methods, AINA improved success rates by 15% and reduced training time by 66%. Ablation studies confirmed the importance of space alignment and depth estimation. The method maintained high performance despite environmental variations, validating its practical applicability.
- The policies generalized well across diverse object types and spatial configurations, enabling complex multi-finger tasks in real-world settings. This indicates a significant step toward autonomous dexterous manipulation driven solely by human videos.
Significance
This work addresses the long-standing challenge of enabling robots to learn complex, multi-finger manipulation directly from natural human behavior, bypassing expensive robot data collection. By leveraging wearable sensing hardware and novel space alignment techniques, it offers a scalable, data-efficient approach that bridges the embodiment gap. The ability to learn from in-the-wild videos opens new avenues for deploying dexterous robots in homes, factories, and service industries, making autonomous manipulation more accessible and adaptable. It fundamentally shifts the paradigm from labor-intensive data collection to scalable human-centric data acquisition, fostering rapid progress in robotic autonomy.
Technical Contribution
AINA introduces a novel combination of high-fidelity perception from smart glasses, space alignment algorithms, and transformer-based point cloud encoding for policy learning. Its key innovations include: β’ Using Aria Gen 2's multi-modal sensing to collect diverse outdoor demonstrations; β’ Developing a robust spatial alignment pipeline with translation and rotation estimation via Kabsch algorithm; β’ Employing a supervised, end-to-end learning framework with Vector Neuron MLPs and transformers to predict future trajectories. These contributions enable direct transfer of human demonstrations to multi-finger robotic control without simulation or robot data, representing a significant advancement in imitation learning and dexterous manipulation.
Novelty
This is the first work to learn multi-finger dexterous manipulation policies solely from in-the-wild human videos, without any robot data or simulation. It leverages smart glasses' sensing capabilities for scalable outdoor data collection and introduces a space alignment method that handles environmental variability. Unlike prior approaches limited to two-finger grippers or requiring extensive robot interaction, AINA achieves complex multi-finger control through a simple, scalable pipeline, marking a breakthrough in autonomous dexterous learning.
Limitations
- The method assumes scene calibration and static environments; dynamic or uncalibrated scenes may impair performance.
- Handling highly dynamic or cluttered scenes remains challenging, requiring further integration of real-time perception and adaptive control.
- Hardware costs and computational demands are still significant, limiting immediate widespread deployment. Future work should focus on model efficiency and robustness.
Future Work
Future directions include integrating multi-modal perception for dynamic scene understanding, enabling real-time adaptation, and expanding to more complex tasks like tool use. Developing lightweight models and hardware will facilitate broader adoption. Additionally, combining autonomous exploration with human demonstrations could further enhance learning efficiency and versatility.
AI Executive Summary
Autonomous dexterous manipulation has long been a goal in robotics, yet traditional approaches rely heavily on labor-intensive data collection or simulation, limiting scalability and generalization. Recent advances in perception and wearable sensing devices, such as smart glasses, open new pathways for learning from natural human behavior in diverse environments. This paper introduces AINA, a novel framework that leverages high-fidelity perception from Aria Gen 2 glasses to collect in-the-wild human demonstrations, enabling direct learning of multi-finger manipulation policies without any robot data.
The core innovation lies in transforming raw video data into consistent 3D representations through space alignment, object tracking, and depth estimation. Using a combination of classical algorithms like Kabsch and modern neural architectures such as transformers and Vector Neuron MLPs, AINA predicts future fingertip trajectories, effectively bridging the embodiment gap between humans and robots. The entire pipeline is supervised end-to-end, avoiding the need for simulation or reinforcement learning, which are often costly and less scalable.
Experimental results across nine everyday tasks demonstrate the effectiveness of AINA. The robot achieves an average success rate of 85%, surpassing baselines that rely solely on scene-specific or wild data. The policies exhibit remarkable robustness to background changes and object variations, enabling complex tasks like opening doors and pouring liquids in cluttered environments. These findings suggest that scalable, human-video-driven learning can significantly accelerate the deployment of autonomous dexterous robots in real-world settings.
Despite its success, the approach faces challenges such as environmental variability and dynamic scene handling. Future research aims to incorporate real-time perception, reduce hardware costs, and extend capabilities to more complex, tool-based tasks. Overall, AINA marks a pivotal step toward scalable, human-inspired robot learning, promising widespread impact in domestic, industrial, and service domains.
Deep Dive
Abstract
Learning multi-fingered robot policies from humans performing daily tasks in natural environments has long been a grand goal in the robotics community. Achieving this would mark significant progress toward generalizable robot manipulation in human environments, as it would reduce the reliance on labor-intensive robot data collection. Despite substantial efforts, progress toward this goal has been bottle-necked by the embodiment gap between humans and robots, as well as by difficulties in extracting relevant contextual and motion cues that enable learning of autonomous policies from in-the-wild human videos. We claim that with simple yet sufficiently powerful hardware for obtaining human data and our proposed framework AINA, we are now one significant step closer to achieving this dream. AINA enables learning multi-fingered policies from data collected by anyone, anywhere, and in any environment using Aria Gen 2 glasses. These glasses are lightweight and portable, feature a high-resolution RGB camera, provide accurate on-board 3D head and hand poses, and offer a wide stereo view that can be leveraged for depth estimation of the scene. This setup enables the learning of 3D point-based policies for multi-fingered hands that are robust to background changes and can be deployed directly without requiring any robot data (including online corrections, reinforcement learning, or simulation). We compare our framework against prior human-to-robot policy learning approaches, ablate our design choices, and demonstrate results across nine everyday manipulation tasks. Robot rollouts are best viewed on our website: https://aina-robot.github.io.