ViViDex: Learning Vision-based Dexterous Manipulation from Human Videos

The evolution of dexterous robotic manipulation has transitioned from trajectory optimization and model-based methods to data-driven deep learning approaches. Early methods relied on precise dynamic models, which are often hard to obtain in real-world scenarios. Reinforcement learning algorithms like PPO and GAIL have been employed to learn control policies directly from robot data, but they demand extensive training data and reward engineering. The use of human videos as demonstration sources has gained attention due to their abundance and ease of collection, enabling imitation learning without extensive robot demonstrations. Prior works such as DexMV attempted to extract robot and object poses from videos, but they suffered from noisy data and dependence on privileged object information, limiting real-world deployment. This background sets the stage for the current challenge: how to efficiently leverage human videos for robust, generalizable dexterous manipulation.

The core challenge lies in extracting high-quality, physically plausible trajectories from noisy human videos and translating these into effective robot control policies. Existing methods require large datasets and rely heavily on privileged information like object CAD models, which are impractical in real-world settings. Additionally, the high-dimensional control space of multi-finger hands complicates learning, especially when attempting to generalize across different objects and poses. The problem is further compounded by the need for minimal demonstration data, real-time adaptability, and robustness to environmental variations. Addressing these issues is crucial for deploying autonomous dexterous robots in unstructured environments.

The paper introduces several key innovations: 1) Trajectory-guided RL that uses extracted human trajectories as reference, improving learning stability and reducing noise impact. 2) A physics-based trajectory refinement process that ensures physical plausibility, enabling more realistic control. 3) Point cloud coordinate transformations that align visual inputs with task-relevant frames, enhancing dense interaction features. 4) A dual training scheme employing behavior cloning and diffusion models, improving visual policy robustness and generalization. These innovations collectively enable end-to-end learning from human videos with minimal data, supporting multi-task and unseen object manipulation.

• �� Extract hand and object trajectories from human videos using pose estimation and motion retargeting, aligning human motion to robot joints.
• �� Refine trajectories through physics-based optimization, minimizing joint angle differences and ensuring physical plausibility.
• �� Train a state-based policy with PPO, guided by trajectory-based rewards for pre-grasp and manipulation stages, with data augmentation for generalization.
• �� Roll out successful trajectories, generate point clouds from depth sensors, and transform them into multiple coordinate systems.
• �� Train the visual policy using behavior cloning or diffusion models, inputting transformed point clouds and robot states to predict control commands.
• �� Evaluate policies in simulation and real-world setups, using success rates and trajectory errors as metrics.

Experiments utilize DexYCB and custom datasets, with simulation in MuJoCo and SAPIEN, and real robot tests on UR5 with Allegro hand. Tasks include relocating objects, pouring particles, and placing items. The training involves minimal videos (1-3 per object) and extensive rollouts to generate diverse trajectories. Success rates, trajectory errors, and ablation studies assess the impact of reward design, point cloud density, and model architecture. Comparisons with baselines demonstrate superior performance and generalization, with success rates exceeding 97% on unseen objects.

Results show that ViViDex achieves over 97% success in multiple tasks with minimal demonstration data, outperforming prior methods like DexMV that require hundreds of videos. The point cloud density and coordinate transformations significantly improve control accuracy. Visual policies trained with diffusion models demonstrate robustness and better generalization to unseen objects. Ablation studies confirm the effectiveness of trajectory-guided rewards and visual augmentation, validating the core innovations. The system maintains high performance both in simulation and real-world environments, indicating strong practical potential.

This approach can be applied to industrial automation, home service robots, and assistive devices, where collecting large datasets is impractical. It enables rapid deployment of dexterous manipulators capable of handling diverse objects with minimal supervision, reducing costs and increasing adaptability. The framework supports multi-task learning, making it suitable for complex, unstructured environments, and can be extended to multi-robot coordination and real-time adaptation.

Current limitations include sensitivity to occlusion and video quality during trajectory extraction, which may affect physical plausibility. The reliance on simulation for training introduces domain gaps, requiring further domain adaptation. Computational costs, especially for point cloud processing and diffusion models, pose challenges for real-time deployment. Future work should focus on multi-modal sensory integration, real-world robustness, and efficiency improvements.