Learning Human-Intention Priors from Large-Scale Human Demonstrations for Robotic Manipulation

Research in robotic manipulation has long faced challenges of data scarcity and hardware dependency. Traditional robot learning methods often rely on expensive and scarce robot demonstration data, limiting their scalability and adaptability. Recently, the rise of vision-language-action (VLA) models has brought new hope to this field. These models, which combine visual observations and language instructions to generate executable actions, have alleviated the issue of data scarcity to some extent. However, these methods still face challenges such as data sparsity and hardware specificity. To overcome these challenges, researchers have begun exploring the possibility of learning manipulation priors from human demonstrations. Human videos record abundant object interaction information, providing a broader source of manipulation priors than robot data.

Despite the rich manipulation priors contained in human videos, using them for robot learning remains difficult. Raw video clips entangle scene understanding, hand motion, and embodiment-specific actions, making them hard to directly use for robot control. Moreover, many video segments contain visible hands without purposeful manipulation, while useful interaction clips rarely provide temporally aligned action labels or robot-executable controls. In this context, effectively extracting manipulation priors from human videos and applying them to robot control becomes a pressing issue.

The core innovation of the MoT-HRA framework lies in its hierarchical structure and knowledge insulation mechanism. First, the framework decomposes action generation into three coupled experts: a vision-language expert, an intention expert, and a fine expert. The vision-language expert predicts an embodiment-agnostic 3D trajectory, the intention expert models MANO-style hand motion as a latent human-motion prior, and the fine expert maps the intention-aware representation to robot action chunks. Second, by using a shared-attention trunk and read-only key-value transfer, MoT-HRA can utilize human priors without interfering with upstream representations. This design turns heterogeneous human videos into an intermediate intention manifold rather than forcing them into robot-specific action labels.

• �� Dataset Construction: First, a large-scale dataset named HA-2.2M was constructed, containing 2.2M action-language episodes reconstructed from heterogeneous human videos.

• �� Vision-Language Expert: This expert predicts an embodiment-agnostic 3D trajectory, providing spatial anchors to support downstream control.

• �� Intention Expert: It models MANO-style hand motion as a latent human-motion prior and generates hand motion sequences through conditional flow matching.

• �� Fine Expert: It maps the intention-aware representation to robot action chunks, ensuring that the final control is an embodiment-specific realization.

• �� Knowledge Insulation: By using a shared-attention trunk and read-only key-value transfer, it ensures minimal interference between human-prior learning and robot policy learning.

The experimental design includes hand motion generation tests on the Ego4D and OakInk datasets, and robotic manipulation tests on SimplerEnv-WidowX tasks. In hand motion generation tests, average displacement error (ADE), dynamic time warping (DTW), wrist rotation error (Rot), and finger joint rotation error (Joint-Rot) were evaluated. In SimplerEnv-WidowX tasks, success rates across different tasks were evaluated. The experiments also included ablation studies to validate the effectiveness of each component. Key hyperparameters include learning rate, batch size, and training steps.

Experimental results show that MoT-HRA performs excellently in both hand motion generation and robotic manipulation tasks. On the Ego4D and OakInk datasets, MoT-HRA achieved the best results in average displacement error and dynamic time warping. In SimplerEnv-WidowX tasks, MoT-HRA significantly outperformed other baseline methods in average success rate across different tasks. Ablation studies show that the introduction of the 3D trajectory branch and intention expert respectively improved hand motion generation accuracy and SimplerEnv average success rate, validating the effectiveness of the hierarchical structure.

The MoT-HRA framework has potential in multiple application scenarios. Direct applications include improvements in robotic manipulation systems, especially in tasks requiring precise spatial grounding and stable control. The framework can also be used for hand motion generation in augmented reality and virtual reality, providing a more natural user experience. In industry, MoT-HRA can be used for robotic manipulation in automated production lines, improving production efficiency and flexibility.

Despite MoT-HRA's excellent performance in multiple tasks, it still has some limitations. First, the noise and ambiguity in human demonstration data may lead to inaccurate intention priors, affecting the precision of robotic manipulation. Additionally, current evaluations focus mainly on hand motion and manipulation tasks, not covering highly dynamic interactions, multi-object long-horizon planning, or very different embodiments. Future research directions include improving data verification, expanding embodiment coverage, and introducing failure detection mechanisms to enhance reliability in open-world deployment.