ImMimic: Cross-Domain Imitation from Human Videos via Mapping and Interpolation
ImMimic combines DTW mapping and MixUp, reaching 1.00 success on several tasks with only five robot demonstrations.
Key Findings
Methodology
ImMimic is an embodiment-agnostic co-training framework. MediaPipe and FrankMocap estimate 21 hand joints and wrist pose; an AnyTeleop-style constrained inverse-kinematics solver retargets them to robot joints. Dynamic Time Warping aligns human and robot trajectories using action or visual distances. MixUp then interpolates both conditioning features and actions, with a progressively decreasing α, while a Diffusion Policy is trained jointly on human-derived and robot demonstrations.
Key Results
- Across Pick and Place, Push, Hammer, and Flip on Robotiq, Fin Ray, Allegro, and Ability end-effectors, ImMimic-A generally beats Robot-Only and vanilla Co-Training. Robotiq Pick and Place rises from 0.40 to 1.00, while Hammer rises from 0.20 to 0.50.
- In the Robotiq/Ability comparison, ImMimic-A scores 1.00 on both Pick and Place and Flip for both embodiments. The corresponding Co-Training scores are 0.40, 0.80, 0.80, and 0.90. With 100 human videos, five robot demonstrations suffice for 1.00 Pick and Place success.
- Action-based mapping outperforms visual and random mapping. Pick-and-Place SPARC scores are -9.4424 for Robotiq, -15.6430 for Fin Ray, -10.8409 for Ability, and -13.8940 for Allegro, generally smoother than Robot-Only policies.
Significance
The work addresses a central bottleneck in robot learning: robot-specific demonstrations are expensive, while human videos are abundant but mismatched in appearance, morphology, and physics. It shows that human hand trajectories can provide useful low-level action supervision rather than merely visual context. Because experiments span four substantially different end-effectors, the result is relevant to scalable skill acquisition, few-shot deployment, and hardware transfer beyond strictly anthropomorphic hands.
Technical Contribution
The main contribution is a sequence-level, mapping-guided interpolation pipeline. DTW imposes temporally coherent human–robot correspondences and avoids implausible random pairing; MixUp creates a continuum between mapped source and target domains in both latent condition and action spaces. The framework combines ResNet18 encoders, history window τ, Diffusion Policy prediction, and the joint objective Ltotal=Lrobot+Lhuman. Retargeted human actions therefore function simultaneously as labels, proprioceptive inputs, and adaptation anchors.
Novelty
Unlike human-video pretraining followed by robot fine-tuning, visual masking, or latent visual invariance alone, ImMimic explicitly aligns and interpolates action trajectories. Its fundamental novelty is treating domain transfer as a smooth path populated by intermediate samples. The paper also directly compares visual-DTW and action-DTW, showing that structurally aligned retargeted actions are often more informative than visual resemblance for low-level manipulation.
Limitations
- Embodiment mechanics remain decisive: Ability achieves 0.00 on Hammer and Allegro is at most 0.20 on Flip, owing to short-thumb interference, large hand size, weak grasping, and poor tool contact.
- The method depends on reliable hand estimation, retargeting, and DTW. Poor visual mapping can make the robot loop in place, and the evidence is limited to four tasks, four end-effectors, and small real-world datasets.
- Action distance depends on mounting and arm kinematics; a more human-like hand is not necessarily closer to human demonstrations.
Future Work
Future work should learn robust multimodal alignment and automatically adapt distance weights and interpolation schedules. Incorporating contact, force, tactile, and object-state signals could address tool and heavy-object failures. Larger studies with unsegmented long videos, unseen objects, more embodiments, and compositional multi-task behavior would test scalability. Hardware–policy co-design is another promising direction.
AI Executive Summary
Human videos could provide an inexpensive source of manipulation knowledge, but direct imitation fails because humans and robots differ in appearance, kinematics, action spaces, and contact physics. Conventional approaches often mask embodiments, restrict actions to simple translations, or pretrain visually before relying on costly robot demonstrations for low-level control.
ImMimic, from Georgia Tech, combines 100 human demonstrations with only five teleoperated robot demonstrations. MediaPipe and FrankMocap recover hand motion; an AnyTeleop-style solver retargets it to robot joints. Dynamic Time Warping then pairs human and robot sequences using action or visual similarity. MixUp interpolates paired conditions and future actions, progressively moving training from the human domain toward the robot domain within a Diffusion Policy.
The action-mapping variant produces the strongest results. Robotiq Pick and Place improves from 0.40 with Robot-Only training to 1.00, and Hammer improves from 0.20 to 0.50. In a Robotiq/Ability evaluation, all four ImMimic-A task–embodiment combinations reach 1.00, while Co-Training ranges from 0.40 to 0.90. Five robot demonstrations can therefore match full success where Robot-Only needs 20. The gains are not universal: hand geometry, grasp force, mounting, and mapping errors still determine difficult tool tasks.
Deep Analysis
Background
Human-video robot learning has evolved from retrieval methods such as EgoMimic and from two-stage human pretraining plus robot adaptation toward joint co-training. Prior systems often mask robot appearance, simplify action spaces, or learn domain-invariant visual features. These strategies reduce some perception gaps but commonly leave human hand motion unused and retain dependence on robot data for action decoding.
Core Problem
The objective is to learn executable manipulation policies from many human videos and few robot demonstrations. The challenge combines visual covariate shift, incompatible morphology and kinematics, different physical contacts, and the absence of inverse adaptation: human video creators do not act with the target robot in mind. Incorrect temporal pairing can further produce misleading supervision.
Innovation
- ��Retargeted hand trajectories become action labels for human videos.
- ��Action- or visual-distance DTW creates temporally consistent correspondences.
- ��Mapping-guided MixUp generates intermediate human–robot domains.
- ��A progressive α schedule enables gradual rather than abrupt domain transfer.
- ��The same Diffusion Policy is optimized on robot and adapted human data.
Methodology
- ��Input: agent-view human RGB videos; robot agent-view, wrist-view, and proprioception.
- ��Estimation: MediaPipe crops hands; FrankMocap/SMPL-X predicts 21 3D joints; PnP recovers wrist 6D pose.
- ��Retargeting: solve Σα||pi-fi(qt)||²+β||qt-qt-1||² under joint limits.
- ��Encoding: separate ResNet18 encoders produce visual features, concatenated with τ-step history.
- ��Prediction: Diffusion Policy denoises k-step future actions using Lrobot and Lhuman.
- ��Mapping: DTW constructs Mh→r using dact or dvis.
- ��Interpolation: zmix=αzh+(1−α)zr and amix=αah→r+(1−α)ar.
- ��Inference: temporal ensembling averages overlapping action predictions for stability.
Experiments
Experiments use a Franka Panda with Robotiq 2F-85, Fin Ray, Allegro, and Ability end-effectors. Tasks are Pick and Place, Push, Hammer, and Flip. The main setting uses five robot and 100 human demonstrations. Baselines include Robot-Only, Two-stage Fine-Tuning, Co-Training, Random Mapping, ImMimic-V, and ImMimic-A. Metrics are 10-rollout Success Rate, SPARC smoothness, and post-DTW Action Distance; human- and robot-demo scaling studies are also reported.
Results
ImMimic-A improves success across the four tasks and four embodiments overall. Robotiq Pick and Place rises from 0.40 to 1.00; Ability rises from 0.80 to 1.00; Robotiq Hammer rises from 0.20 to 0.50. In Table 2, all four Robotiq/Ability task combinations reach 1.00 with ImMimic-A. Action mapping beats visual and random mapping, while 50 human videos already raise Robotiq Pick and Place to 1.00. SPARC generally improves over Robot-Only.
Applications
The approach fits learning from public videos for grasping, pushing, tool use, and dexterous manipulation when robot data collection is expensive or hardware changes frequently. Deployment requires RGB sensing, reliable hand and wrist estimation, a robot kinematic model, and a small teleoperation set. Potential users include manufacturing, warehouse automation, household robotics, and educational platforms seeking cheaper skill acquisition.
Limitations & Outlook
ImMimic assumes visible hands and sufficiently accurate retargeting; DTW can fail when visual features or action estimates are noisy. It does not explicitly model force, friction, or collision, explaining failures with heavy hammers, thin fingertips, and oversized tools. The study remains small and does not establish generalization to unseen objects, environments, or long-horizon compositions. Future systems should add multimodal physical-state modeling, uncertainty-aware mapping, and larger cross-domain evaluations.
Plain Language Accessible to non-experts
Imagine training a new cook using many videos of expert chefs, but the trainee has different arms, utensils, and strength. The videos reveal when to reach, grip, push, or flip, yet copying the exact motion may fail because the trainee’s body is not the chef’s body. ImMimic first translates the chef’s hand movement into instructions that the trainee’s equipment can physically follow.
It then compares the human and robot demonstrations like matching two songs with different tempos. The system finds moments that represent similar stages of a task, rather than pairing frames randomly. Next it blends the two demonstrations, creating practice examples that are partly human-like and partly robot-like. During training, the examples gradually become more robot-like, so the policy learns a smooth transition instead of making one sudden leap.
With five robot demonstrations and 100 human videos, the method improved Robotiq Pick and Place from 40% to 100%. It also produced smoother motion. But better teaching cannot change the trainee’s body: short thumbs, large hands, weak grip, and awkward mounting still caused failures in hammering and flipping. The lesson is powerful but realistic—data can compensate for limited experience, not for every physical limitation.
ELI14 Explained like you're 14
Suppose you are teaching a game-playing robot to move a cube, but you have 100 videos of humans and only five videos of the robot. Humans have flexible fingers; the robot may have two grippers. A move that looks identical on screen might require totally different positions and strength.
ImMimic first translates the human hand motion into movements the robot’s joints can use. Then it matches the timing of the two demonstrations, like syncing two songs when one singer is faster. It looks for moments where both are grabbing, pushing, or preparing to strike. After that, it mixes the human and robot versions to create lots of “in-between” practice examples.
Why mix them? Jumping straight from human behavior to robot behavior is like trying a boss level without the training levels. The mixed examples act like those levels. The robot learns gradually, while a Diffusion Policy predicts a short sequence of future moves. In tests, Robotiq’s Pick and Place success went from 40% to 100%, and five robot demos were enough.
Still, no algorithm gives a robot a new body! Ability’s short thumb hurt hammering, and Allegro’s large hand struggled with a spatula. Future versions could add touch and force sensing. Then the robot might know not only where to move, but also how hard to squeeze—pretty important when objects are heavy or slippery!
Glossary
Dynamic Time Warping
A sequence-alignment method that matches events occurring at different speeds. Technically, it finds a low-cost, order-preserving nonlinear alignment path.
ImMimic uses DTW to map human timesteps to robot timesteps using visual or action distances.
MixUp
A data-augmentation method that linearly combines two inputs and their labels. Here it constructs intermediate domains rather than only enlarging a dataset.
The paper interpolates both conditioning features and future actions with coefficient α.
Diffusion Policy
A policy that generates actions by iteratively denoising a random sequence. It can represent multiple plausible continuous behaviors.
ImMimic uses it to predict k-step actions from images, proprioception, and history.
SPARC
Spectral Arc Length, a frequency-domain measure of movement smoothness. Higher values indicate smoother trajectories in the paper’s convention.
It compares rollout smoothness across Robot-Only, Co-Training, and ImMimic-A.
Action Distance
An average difference between aligned translation, hand pose, and orientation trajectories. Smaller values indicate greater structural similarity.
Reported averages are Robotiq 0.066, Fin Ray 0.065, Allegro 0.078, and Ability 0.075.
Open Questions Unanswered questions from this research
- 1 Can the method transfer reliably to unseen objects, environments, and long-horizon compositions? The current evidence covers four tasks and four end-effectors, so broader testing is needed.
- 2 Action distance works well, but its components and weights depend on embodiment and mounting. Learning a universal alignment metric across morphology and contact states remains open.
- 3 Hand and visual estimation can fail under occlusion. Tactile, force, and object-state signals may be necessary for robust physical interaction.
Applications
Immediate Applications
Low-cost skill collection
Robot developers can pretrain grasping, pushing, and tool-use policies from public human videos, then calibrate them with roughly five teleoperated robot demonstrations. Required components are RGB sensing, hand estimation, robot kinematics, and a safe evaluation setup.
Cross-gripper policy transfer
A shared human-video corpus can support Robotiq, Fin Ray, Allegro, or Ability hardware through action-based DTW. Engineers must still validate fingertip geometry, thumb reach, grasp force, and mounting because these factors can dominate task success.
Long-term Vision
Video-driven general-purpose manipulation
Combined with tactile sensing, object-state estimation, and long-video understanding, the approach could let robots acquire skills from the web. Major barriers include safety certification, physical mismatch, dataset bias, copyright, and reliability in rare situations.
Abstract
Learning robot manipulation from abundant human videos offers a scalable alternative to costly robot-specific data collection. However, domain gaps across visual, morphological, and physical aspects hinder direct imitation. To effectively bridge the domain gap, we propose ImMimic, an embodiment-agnostic co-training framework that leverages both human videos and a small amount of teleoperated robot demonstrations. ImMimic uses Dynamic Time Warping (DTW) with either action- or visual-based mapping to map retargeted human hand poses to robot joints, followed by MixUp interpolation between paired human and robot trajectories. Our key insights are (1) retargeted human hand trajectories provide informative action labels, and (2) interpolation over the mapped data creates intermediate domains that facilitate smooth domain adaptation during co-training. Evaluations on four real-world manipulation tasks (Pick and Place, Push, Hammer, Flip) across four robotic embodiments (Robotiq, Fin Ray, Allegro, Ability) show that ImMimic improves task success rates and execution smoothness, highlighting its efficacy to bridge the domain gap for robust robot manipulation. The project website can be found at https://sites.google.com/view/immimic.