Point Policy: Unifying Observations and Actions with Key Points for Robot Manipulation

The evolution of robot policy learning has seen significant progress with imitation learning and reinforcement learning, yet these methods require extensive robot-environment interactions, limiting scalability. Recent works like Behavior Cloning (BC), inverse RL, and multi-task learning have improved sample efficiency but still struggle with generalization and environmental variability. Object-centric representations, such as bounding boxes, pose estimation, and point clouds, have been explored to improve robustness. Key points, as a structured and semantic representation, have gained attention for their ability to generalize across different objects and scenes. Despite these advances, most approaches rely on image-based inputs or depth sensors, which are sensitive to noise and environmental changes. Human videos offer a vast, untapped resource for robot learning, but extracting meaningful information from such data remains challenging due to the morphology gap and environmental complexity. This work builds on these foundations, proposing a key point-based, geometry-aware, offline learning framework that leverages multi-view correspondence models to enable scalable robot policy acquisition.

The core challenge is how to effectively utilize human videos—abundant and diverse—for robot policy learning, overcoming the morphological and visual differences between humans and robots. Traditional methods require extensive robot interaction data, which is costly and time-consuming. The key issues include extracting semantically meaningful, spatially accurate key points from videos, mapping these points into robot space, and predicting future actions robustly across varying environments. Achieving high generalization, especially to unseen objects and cluttered backgrounds, remains difficult. Moreover, ensuring the learned policies are reactive and adaptable without online interaction is an open problem. Addressing these issues is critical for scalable, autonomous robot deployment in real-world scenarios.

This work introduces several innovations: 1) Multi-view semantic correspondence (DIFT) and point tracking (Co-Tracker) for automatic, robust key point extraction; 2) Geometric mapping to transfer human key points into robot coordinate space, enabling observation-action unification; 3) Transformer-based sequence modeling to predict future key point trajectories, facilitating planning without online interaction; 4) A fully offline, end-to-end pipeline that leverages human videos for policy learning, significantly reducing data collection costs. These innovations collectively enable high-fidelity transfer of human demonstrations to robot actions, with strong generalization and robustness.

• �� Collect human demonstration videos, extract hand key points using MediaPipe; • Use multi-view triangulation to obtain 3D human hand and object key points; • Annotate a few frames with semantic object points, transfer via DIFT, and track with Co-Tracker; • Map human key points to robot end-effector points through geometric transformations; • Define robot pose and gripper state based on key points; • Use a transformer model (BAKU) to encode historical key points and predict future trajectories; • Convert predicted trajectories into robot actions via rigid body constraints; • Execute actions at 6Hz, closing the loop for real-time control.

Experiments involve 8 real-world tasks with diverse objects and backgrounds, using 190 human demonstrations and 100 robot demonstrations for co-training. Success metrics include task success rate, robustness to novel objects, and background clutter. Baselines include Behavior Cloning (RGB, RGB-D), Motion Tracks, and P3-PO. The evaluation tests in-domain and out-of-domain generalization, ablation studies on key components, and robustness to environmental variations. Results consistently favor Point Policy, with success rates reaching 88% on average, outperforming baselines by significant margins. The experiments validate the method’s effectiveness and robustness across scenarios.

Point Policy achieves an average success rate of 88% across 8 tasks, outperforming the best baseline (MT-π) by 75%. It maintains high performance on unseen object instances with a 74% success rate increase, e.g., success in 'Put bottle on rack' jumps from 0 to 90%. Under background clutter, success rates remain high, with only slight declines. These results demonstrate superior generalization, robustness, and efficiency, confirming the framework’s potential for scalable robot learning from human videos.

This approach is suitable for industrial automation, household robots, and service robots, especially where collecting robot interaction data is impractical. It enables rapid deployment with minimal human annotation, adaptable to diverse tasks and environments. Long-term, it could lead to autonomous systems capable of continuous self-improvement by observing human behaviors, reducing the need for explicit programming or extensive data collection.

Dependence on scene calibration and initial hand pose estimation limits deployment in uncalibrated or dynamic scenes. Handling occlusions and rapid movements remains challenging. Multi-view sensing increases system complexity, and performance under extreme environmental variability needs further validation. Future work should focus on self-supervised key point detection and online adaptation to improve robustness and ease of deployment.