RealDexUMI: A Wearable Universal Manipulation Interface for Dexterous Robot Learning
RealDexUMI enables dexterous robot learning via a wearable interface, achieving an 88.75% success rate.
Key Findings
Methodology
RealDexUMI uses a shared dexterous end-effector module, integrating a lightweight dexterous hand, in-hand vision, and fingertip tactile sensing. A palm-side isomorphic teleoperation glove maps human finger inputs to robot-hand joint commands, enabling real-time, retargeting-free, intuitive, and precise hand control.
Key Results
- Across eight real-robot tasks, policies trained on RealDexUMI data achieve an average success rate of 88.75%, generalize to unseen initial poses, and transfer across three embodiments.
- Removing tactile input reduces success from 88.75% to 70.00%, particularly in tasks like plug insertion and tea picking, highlighting the importance of tactile input.
- In cross-embodiment deployment, policies achieve high success rates on different robots without retraining, indicating that dexterous behavior primarily relies on the shared end-effector interface.
Significance
RealDexUMI addresses the inconsistency between demonstration and deployment in dexterous manipulation by providing a zero-gap data collection and deployment interface. It is significant for academia and industry, especially in complex tasks requiring precise hand control.
Technical Contribution
RealDexUMI's technical contributions include its shared dexterous end-effector module and palm-side isomorphic glove design, eliminating the need for retargeting and visual post-processing common in traditional methods, providing a scalable interface for dexterous policy learning.
Novelty
RealDexUMI is the first system to achieve zero-gap data collection and deployment through a shared dexterous end-effector, eliminating retargeting and visual post-processing compared to existing methods.
Limitations
- RealDexUMI's sensing is mainly local, limiting tasks requiring object search, long-range planning, or explicit task-progress reasoning.
- The current hand design's degrees of freedom are limited, unable to cover the full capabilities of higher-DoF dexterous hands.
Future Work
Future work could include adding egocentric or global views to extend task scope while maintaining alignment between wearable collection and robot deployment. Extending to more expressive hand designs while preserving low-burden, precise control is also important.
AI Executive Summary
Learning dexterous manipulation requires demonstrations that preserve fine hand-object interactions while remaining executable at deployment. Existing pipelines either lose deployable dexterity through retargeting or embodiment conversion, or rely on robot-specific teleoperation that is costly to scale and often lacks intuitive, contact-aware control. RealDexUMI provides a wearable universal manipulation interface built around a shared dexterous end-effector module, integrating a lightweight dexterous hand, in-hand vision, and fingertip tactile sensing. A palm-side isomorphic teleoperation glove maps human finger inputs to robot-hand joint commands, enabling real-time, retargeting-free, intuitive, and precise hand control. Experiments show that policies trained on RealDexUMI data achieve an average success rate of 88.75% across eight real-robot tasks, generalize to unseen initial poses, and transfer across three embodiments. These results highlight RealDexUMI as a practical and scalable interface for robot-free collection of deployable dexterous manipulation data.
Deep Analysis
Background
The evolution of dexterous manipulation has progressed from simple mechanical hands to complex dexterous hands. Early research focused on motion control of mechanical hands, while recent studies emphasize the fine manipulation capabilities of dexterous hands. Representative works include DexViTac and DexUMI, which achieve dexterous manipulation data collection and policy learning through various methods.
Core Problem
The core problem of dexterous manipulation is maintaining consistent hand-object interactions during demonstration and deployment. Existing methods often lose dexterity during retargeting or embodiment conversion, or rely on robot-specific teleoperation that is costly to scale and lacks intuitive control.
Innovation
RealDexUMI's core innovations include its shared dexterous end-effector module and palm-side isomorphic glove design. By eliminating the need for retargeting and visual post-processing, it achieves zero-gap dexterous data collection and deployment. This design not only improves data collection efficiency but also ensures policy consistency across different embodiments.
Methodology
- �� Shared dexterous end-effector module: integrates a lightweight dexterous hand, in-hand vision, and fingertip tactile sensing. • Palm-side isomorphic teleoperation glove: maps human finger inputs to robot-hand joint commands for real-time, retargeting-free, intuitive, and precise hand control. • Data collection: records time-aligned hand commands, measured hand states, in-hand RGB observations, fingertip tactile signals, and 6-DoF tracker poses.
Experiments
The experimental design includes eight real-robot tasks, with policies trained on RealDexUMI data evaluated on a Franka FR3. Each policy is evaluated over 20 real-robot trials, with success defined as completing the full task.
Results
Experimental results show that RealDexUMI achieves an average success rate of 88.75% across eight real-robot tasks, generalizes to unseen initial poses, and transfers across three embodiments. Removing tactile input significantly reduces success, highlighting the importance of tactile input.
Applications
RealDexUMI's application scenarios include complex tasks requiring precise hand control, such as multi-object grasping, precision insertion, tool use, twisting, articulated-object interaction, long-horizon execution, and bimanual operation.
Limitations & Outlook
RealDexUMI's limitations include its mainly local sensing, limiting tasks requiring object search, long-range planning, or explicit task-progress reasoning. Additionally, the current hand design's degrees of freedom are limited, unable to cover the full capabilities of higher-DoF dexterous hands.
Plain Language Accessible to non-experts
Imagine you're in a kitchen cooking, and RealDexUMI is like a smart assistant that can precisely mimic your hand movements to complete various complex cooking tasks. It can remember how you chop, stir, and flip, and continue these tasks even when you're not around. In this way, it helps you achieve more efficient and precise operations in the kitchen.
ELI14 Explained like you're 14
Imagine you're playing a super cool robot game, and RealDexUMI is like a magical glove that lets you control a real robot hand just like you control a game character. You just move your fingers, and the robot can precisely complete various tasks, like grabbing objects, inserting plugs, and using tools. Isn't that awesome?
Glossary
Dexterous Hand
A mechanical hand with multiple degrees of freedom capable of complex hand operations.
In RealDexUMI, the dexterous hand is a core component for achieving fine hand-object interactions.
End-Effector
A tool or device at the end of a robot arm used for interacting with the environment.
RealDexUMI uses a shared dexterous end-effector module for data collection and deployment.
Isomorphic Teleoperation Glove
A glove that maps human finger inputs to robot-hand joint commands.
RealDexUMI uses this glove for real-time, retargeting-free hand control.
Tactile Sensing
Sensing that perceives the touch and contact information of objects through sensors.
RealDexUMI's fingertip tactile sensors provide precise contact observations.
Retargeting
The process of converting one form of motion to another.
RealDexUMI eliminates the need for retargeting common in traditional methods.
Open Questions Unanswered questions from this research
- 1 How to add global views while maintaining dexterous manipulation to extend task scope.
- 2 How to achieve low-burden and precise control in higher-DoF dexterous hands.
Applications
Immediate Applications
Industrial Automation
RealDexUMI can be used for precise manipulation tasks in industrial automation, such as complex assembly on production lines.
Medical Robotics
In the medical field, RealDexUMI can be used for precise operations in surgeries, reducing human error.
Long-term Vision
Home Robotics
RealDexUMI could enable automation of complex household tasks, such as cleaning and cooking, in future home robots.
Abstract
Learning dexterous manipulation requires demonstrations that preserve fine hand-object interactions while remaining executable at deployment. Existing pipelines either lose deployable dexterity through retargeting or embodiment conversion, or rely on robot-specific teleoperation that is costly to scale and often lacks intuitive, contact-aware control for dexterous data collection. We present RealDexUMI, a wearable universal manipulation interface built around a shared dexterous end-effector module that integrates a lightweight dexterous hand, in-hand vision, and fingertip tactile sensing. A palm-side isomorphic teleoperation glove maps human finger inputs to robot-hand joint commands, enabling real-time, retargeting-free, intuitive, and precise hand control. The shared hand and sensing modules yield zero-gap end-effector data, with matched in-hand observations, tactile signals, contacts, and hand actions between collection and deployment. Across eight real-robot tasks spanning fine-grained, contact-rich, long-horizon, and bimanual manipulation, policies trained on RealDexUMI data achieve an average success rate of 88.75%, generalize to unseen initial poses, and transfer across three embodiments. Website: https://research.beingbeyond.com/realdexumi