Human2Any: Human-to-Robot Transfer via Constraint-Aware Compositional Planning
Human2Any employs object-centric interaction priors learned from human videos, enabling cross-embodiment robot manipulation without real robot demonstrations.
Key Findings
Methodology
Human2Any models object-object interaction motions via a factor graph, separating transferable interaction structures from robot-specific feasibility constraints. It employs conditional diffusion models to learn interaction priors, combined with a constraint-aware sampling strategy during testing. This approach involves object interaction prior learning, robot-object relationship modeling, and in-context feasibility-guided sampling, supporting generalization across robot morphologies and scenes. Key algorithms include diffusion-based trajectory generation and particle filtering for constraint steering.
Key Results
- In diverse simulation tasks, Human2Any achieves over 85% success without robot demonstrations, outperforming DP3 and Im2Flow2Act. On real platforms, success rates reach 80% on Franka and 70% on RBY-1, demonstrating strong transferability. Constraint-guided sampling significantly improves trajectory feasibility and efficiency, especially in complex scenes.
- Cross-embodiment transfer shows the model effectively reuses interaction priors on both Franka and RBY-1 robots, indicating high shape adaptability. Increasing interaction data further boosts performance, confirming the positive impact of data scale.
- Ablation studies reveal that constraint steering doubles sampling efficiency and reduces failure rates compared to unguided sampling. The model outperforms behavior cloning in long-horizon tasks, showing excellent generalization to unseen scenarios.
Significance
This work addresses the core challenge of transferring human demonstration insights to robots across different morphologies and environments. By abstracting interaction structures, it reduces reliance on robot-specific data, enabling scalable, generalizable manipulation. The framework advances autonomous robot learning, promising applications in industrial automation, service robots, and household automation, fostering more adaptable and intelligent robotic systems.
Technical Contribution
The paper introduces a novel object-object interaction graph representation, distinct from end-to-end action regression, enabling cross-morphology transfer. It employs diffusion models for flexible trajectory learning and a particle filtering-based constraint steering mechanism, improving robustness and efficiency. The separation of interaction priors and feasibility constraints allows scalable, embodiment-agnostic manipulation, representing a significant step beyond state-of-the-art methods.
Novelty
This is the first work to explicitly model object-object interaction motions as transferable priors learned via diffusion models, combined with a test-time constraint steering mechanism. Unlike prior methods relying solely on robot demonstrations or end-to-end learning, it abstracts interaction structures, supporting broad generalization and cross-embodiment transfer, marking a significant innovation in manipulation learning.
Limitations
- Currently limited to prehensile manipulation with rigid attachment assumptions; extending to dexterous in-hand manipulation or non-prehensile tasks requires additional modeling. Scene perception relies on point clouds, which may be unreliable under occlusion or sensor noise. Large-scale interaction data is needed for training, and efficiency drops in highly complex scenarios. Future work should incorporate dynamic interaction models and online feedback for robustness.
Future Work
Future directions include integrating reinforcement learning and online adaptation to handle dynamic environments, expanding to multi-finger and non-prehensile manipulation, and incorporating multi-modal sensory data. Combining natural language understanding for task specification and planning is also envisioned, aiming to develop a comprehensive, autonomous manipulation framework capable of handling complex, real-world tasks.
AI Executive Summary
As robots become increasingly integrated into daily life and industrial settings, enabling them to learn versatile manipulation skills efficiently remains a major challenge. Traditional approaches depend heavily on robot demonstrations, which are costly and limited in diversity. Human videos, abundant and rich in object interactions, offer a promising alternative. However, directly transferring human motion to robots is hindered by embodiment differences and scene variability. This paper introduces Human2Any, a novel framework that leverages object-centric interaction priors learned from human videos to facilitate cross-embodiment robot manipulation.
The core idea is to abstract manipulation as object-object interactions, capturing the essential structure of tasks without tying it to human-specific motions. These interaction priors are learned using diffusion models trained on point-cloud tracked trajectories, enabling flexible, diverse motion generation. At deployment, the framework combines these priors with robot-specific feasibility models, including grasping, kinematic, and collision constraints, through a constraint-aware sampling process. This process iteratively refines candidate trajectories, steering them toward feasible, goal-oriented motions.
Extensive experiments in simulation and real-world settings demonstrate the effectiveness of Human2Any. The system achieves over 85% success in simulated tasks like pouring and object sorting, without using robot demonstrations. On physical robots—Franka and RBY-1—it attains success rates of 80% and 70%, respectively, across diverse tasks. The approach exhibits strong generalization to new scene layouts and robot morphologies, validating its cross-embodiment transfer capability. The constraint-guided sampling notably improves efficiency, reducing failure rates and enabling long-horizon task execution.
This work significantly advances autonomous manipulation by decoupling interaction learning from embodiment constraints, paving the way for scalable, versatile robot training. Its potential impact spans industrial automation, service robotics, and household automation, where rapid adaptation to new tasks and environments is crucial. Future research will focus on extending to dexterous in-hand manipulation, integrating multi-modal perception, and enabling online learning for dynamic scenarios, further pushing the boundaries of robot autonomy.
Deep Analysis
Background
Robotics manipulation has seen rapid progress through imitation learning, reinforcement learning, and end-to-end deep models. Notable works include behavior cloning methods like DAgger, and scene understanding approaches such as affordance prediction. Despite these advances, high data collection costs and limited generalization hinder deployment. Human videos, as a vast resource, contain rich object interactions, but their direct use for robot learning remains challenging due to embodiment mismatch. Prior efforts focused on retargeting human motions or learning affordances, yet lacked scalable, embodiment-agnostic models. This research builds on object-centric representations, aiming to extract transferable interaction priors that can adapt across robots and scenes.
Core Problem
The main challenge lies in transferring manipulation skills learned from human videos to robots with different morphologies and scene layouts. Existing methods either require extensive robot demonstrations or struggle to generalize due to embodiment constraints. The core bottleneck is capturing the essential interaction structure—such as which objects participate and how they change—without overfitting to specific motions. Achieving scalable, cross-embodiment transfer demands models that abstract interaction patterns and incorporate scene constraints during execution, enabling robots to perform diverse tasks reliably without costly data collection.
Innovation
This work introduces a novel object-centric interaction graph that models the relative motion between objects, learned via diffusion models trained on human video data. It separates transferable interaction priors from robot-specific feasibility models, allowing cross-embodiment adaptation. The key innovation is the test-time constraint-aware sampling, which guides the generation of feasible trajectories by integrating scene and robot constraints into the diffusion process. This approach enables scalable learning from human videos and robust execution across different robots and environments, a significant step beyond prior end-to-end or imitation-based methods.
Methodology
- �� Extract object point clouds and relative trajectories from human videos using segmentation and tracking algorithms like PointTrack.
- �� Train a conditional diffusion model on these trajectories to learn object-object interaction priors, conditioned on object point clouds.
- �� Develop robot-specific grasping and control priors using simulation data, modeling how each robot interacts with objects.
- �� During deployment, combine learned interaction priors with robot and scene constraints, formulating a joint distribution.
- �� Use a particle filtering-based sampling process during denoising, where each particle’s likelihood is scored based on feasibility metrics such as collision avoidance, kinematic reachability, and grasp stability.
- �� Iteratively resample particles to steer the generated trajectories toward feasible, goal-oriented motions.
- �� Convert the final sampled trajectories into robot control commands for execution.
Experiments
The evaluation includes simulation tasks in MuJoCo, covering pouring, object sorting, and long-horizon manipulation, with variants to test generalization. Real-world tests involve Franka and RBY-1 robots executing tasks like pouring and object transfer without robot demonstrations. Metrics include success rate, task completion time, and sample efficiency. Baselines include behavior cloning and flow-based methods, with ablation studies on the impact of constraint steering and data scale. The experiments demonstrate high success rates, cross-robot transfer, and robustness to scene variations, validating the framework’s effectiveness.
Results
Human2Any achieves over 85% success in simulation, outperforming baselines, and maintains high success rates (~80%) on real robots across diverse tasks. Constraint-guided sampling reduces failure rates by 50%, improves efficiency, and enables long-horizon tasks. Increasing interaction data size correlates with performance gains, confirming the importance of rich supervision. Cross-embodiment transfer is validated by successful deployment on different robot morphologies, demonstrating the model’s generalization and adaptability capabilities.
Abstract
Human videos are a scalable source of supervision for robot manipulation, as they are abundant and naturally capture rich object interactions. However, transferring human demonstrations to robots remains challenging due to embodiment mismatch, scene variation, and robot-specific feasibility constraints. We present Human2Any, a framework for learning reusable object-centric interaction priors from human videos without requiring real-world robot demonstrations in the target task contexts. Human2Any represents manipulation through object-object interaction motion, capturing task-relevant scene changes while abstracting away embodiment-specific details. It composes learned interaction priors with robot-side feasibility reasoning and motion planning, allowing the same human-derived knowledge to adapt to different embodiments, scene geometries, and task contexts. We validate Human2Any across diverse manipulation settings, including real-world experiments on a Franka tabletop setup and an RBY-1 humanoid mobile robot, demonstrating robust interaction-centric manipulation without real-world robot training data. Project website: https://human2any.github.io/.