Affordances from Human Videos as a Versatile Representation for Robotics

Robotics research has seen rapid progress with deep learning, especially in visual perception and control. Early works like GQN and DQN demonstrated scene understanding and decision-making in static or simulated environments. However, deploying robots in real-world, dynamic settings remains difficult due to limited generalization and data scarcity. Recent efforts utilize human videos for behavior understanding, such as HOI and HAP, but mostly focus on static pose or gesture recognition. The advent of large-scale internet videos offers new opportunities to learn scene affordances passively, yet integrating this data into robotic control pipelines is still an open challenge. Prior methods often rely on handcrafted features or simulation, which lack scalability and robustness.

The core challenge is to extract meaningful, robot-centric scene affordances from passive human videos, capturing where and how interactions occur. Existing approaches struggle with multi-modality, scene variability, and transferability to real robots. Additionally, translating visual affordances into actionable control signals that support diverse tasks and environments remains unresolved. The difficulty is compounded by occlusions, dynamic backgrounds, and domain shifts between human videos and robot perception. Addressing these issues requires a scalable, generalizable framework capable of predicting multiple interaction modes and integrating seamlessly with robotic learning algorithms.

The paper's key innovations include: 1) defining a multi-modal affordance representation based on contact points and post-contact trajectories, which are robot-agnostic; 2) employing GMM to model interaction variability, capturing multiple feasible actions; 3) leveraging large-scale passive videos with automatic annotations for scalable supervision; 4) integrating the affordance prior into multiple robot learning paradigms—imitation, exploration, goal-conditioned policies, and structured action spaces—enabling versatile real-world deployment. These innovations collectively advance the state-of-the-art in visual affordance learning and robotic control.

• �� Collect large-scale human interaction videos from the internet, applying hand-object detection for automatic contact and trajectory annotation. • Use homography matrices to compensate for camera egomotion, ensuring spatial consistency across frames. • Fit Gaussian Mixture Models to contact points to model multi-modal interaction possibilities. • Encode scene images with ResNet to extract spatial features, then generate heatmaps via deconvolutional layers. • Use spatial softmax to estimate GMM means, and train a trajectory prediction network with self-attention blocks to forecast post-contact movements. • Map predicted contact points and trajectories into robot control space, enabling multi-task learning and policy optimization.

Experiments involve four real-world environments with tasks like opening cabinets, grasping objects, and manipulating shelves. Two robot platforms—Franka Panda and Stretch—are used. The model is trained on passive videos and evaluated on unseen scenarios, comparing success rates against baselines like Hotspots and HAP. Hyperparameters include a batch size of 32, training over 24 hours, with success measured via task completion metrics. Ablation studies analyze the importance of multi-modality and prediction accuracy. The experiments demonstrate that VRB significantly outperforms baselines, with success rates around 57%, confirming its robustness and transferability.

VRB achieves a success rate of 57% across 10 tasks, outperforming prior methods (Hotspots 25%). It generalizes well to unseen environments and tasks, with hundreds of hours of autonomous robot operation in the wild. Multi-modal contact and trajectory predictions improve interaction accuracy, especially under occlusion or clutter. The framework's ability to transfer from passive videos to real robot control is validated across different platforms, confirming its scalability and practical relevance. Ablation results show that modeling multiple interaction modes via GMM is crucial for high performance.

This approach can be immediately applied to domestic robots, warehouse automation, and outdoor exploration, where passive observation data is abundant. It reduces the need for extensive manual labeling or simulation, lowering deployment barriers. Long-term, it enables robots to autonomously discover and adapt to new objects and tasks, transforming industries by making robots more flexible, scalable, and capable of learning from passive human behavior without explicit supervision.

Despite promising results, the model's multi-modal predictions can be inaccurate in highly cluttered or occluded scenes. Dependence on detection tools and video quality limits robustness. Computational costs for training and inference are high, hindering real-time deployment. Future work should focus on improving prediction robustness, efficiency, and multi-modal fusion to handle more complex, dynamic environments.