ViTacPhys: Physical Property-Aware Grasping from Human Visual-Tactile Demonstrations
ViTacPhys uses multimodal visual-tactile modeling and VLM priors to predict object mass, stiffness, and friction with 97.2% accuracy, enabling adaptive robotic grasping.
Key Findings
Methodology
This work introduces a temporal multimodal framework combining visual and tactile sequences with a cross-attention fusion module, integrating a semantic prior derived from a pre-trained vision-language model (VLM). The visual encoder uses ResNet-18 to extract appearance and flow features, while tactile maps are processed similarly. The semantic prior, generated from the first five RGB frames via GPT-5.4 and encoded with Sentence-BERT, provides high-level object context before contact. The fusion module employs bidirectional cross-attention to align visual and tactile features, followed by task-specific heads predicting mass (ordinal classification), stiffness (regression), and friction coefficient (ordinal classification). The model is trained with a combination of cross-entropy and mean squared error losses, with task-specific weighting via GradNorm. For transfer, limited robot teleoperation data, augmented with matched human demonstrations and visual domain adaptation, fine-tune the model for real-world grasping. The physical property predictions condition an adaptive grasping policy, evaluated on in-distribution and out-of-distribution objects, achieving success rates of 95% and 83.4%, respectively.
Key Results
- On seen objects, the model achieves 97.2% accuracy in mass classification, 98.8% in friction classification, and a 5.51% mean absolute percentage error (MAPE) in stiffness prediction. For unseen objects from known categories, accuracy remains high at 87.5% for mass, 97.5% for friction, and MAPE is 9.08%.
- When deployed on a robot, the property-conditioned grasp policy outperforms baseline ACT methods, with success rates of 95% on in-distribution objects and 83.4% on out-of-distribution objects, demonstrating strong generalization. The force profiles during grasping are more consistent with human teleoperation, indicating better force control.
- The integration of semantic priors from VLMs significantly improves the model’s ability to generalize to unseen objects, validating the effectiveness of multimodal fusion and human demonstration-based training for real-world robotic manipulation.
Significance
This research addresses a critical challenge in robotic manipulation: the perception of physical object properties necessary for adaptive grasping. By leveraging human demonstrations and multimodal learning, it bridges the gap between offline physical property estimation and real-time robotic control. The approach enhances robots’ ability to handle diverse objects with varying material properties, reducing damage and slip risks. It also demonstrates the potential of integrating language-based semantic priors with sensor data, opening new avenues for intelligent, context-aware robotic systems. The successful transfer from human to robot domains highlights the practicality of the method, promising significant impact on industrial automation, service robotics, and assistive technologies.
Technical Contribution
The paper introduces a novel temporal multimodal prediction architecture that fuses visual and tactile sequences using cross-attention, augmented with a VLM-derived semantic prior for physical property inference. The model employs a combination of ordinal classification for mass and friction, and regression for stiffness, optimized jointly with task-specific losses and adaptive weighting via GradNorm. The integration of human demonstration data and limited robot teleoperation for domain transfer represents a key engineering advance, enabling real-time, physically conditioned grasping policies. This work advances the state-of-the-art by combining multimodal deep learning, semantic priors, and human-robot transfer in a unified framework, demonstrating superior accuracy and generalization in physical property estimation.
Novelty
This is the first work to leverage human visual-tactile demonstrations combined with pre-trained vision-language models for predicting physical properties relevant to grasping. Unlike prior offline or single-modal methods, it achieves real-time, continuous estimation of mass, stiffness, and friction, and successfully transfers this knowledge to robotic platforms. The integration of semantic priors with multimodal sensor data for physical property inference represents a novel contribution, significantly improving generalization to unseen objects and dynamic interactions.
Limitations
- The model’s accuracy diminishes when encountering materials with extreme elasticity or very thin surfaces, due to limited dataset diversity in these regimes.
- Transfer relies on limited teleoperation data, which may restrict adaptation in highly novel or unstructured environments.
- Real-time performance under high-speed interactions or complex multi-contact scenarios needs further optimization, and current experiments focus mainly on static or quasi-static interactions.
Future Work
Future research will expand dataset diversity, including more material types and interaction protocols, to improve robustness. Incorporating reinforcement learning could optimize grasp strategies based on predicted properties. Additionally, integrating higher-fidelity sensors and multi-sensor fusion will enhance estimation accuracy and real-time responsiveness, enabling robots to operate more autonomously in complex, unstructured environments.
AI Executive Summary
Deep Dive
Plain Language Accessible to non-experts
想象你在厨房里准备食材。你用眼睛观察食材的颜色和形状，然后用手触摸感受它们的硬度和滑动的感觉。有些食材很软，比如面包；有些很硬，比如坚果。人类在日常生活中就是这样，凭借视觉和触觉的结合，快速判断物体的重量、硬度和表面摩擦力，从而决定用多大力抓取或切割。科学家们让机器人也学会了这种能力，他们用特殊的学习方法，让机器人看和摸东西，然后告诉它们这些东西的重量、硬度和摩擦力。这样，机器人就能像我们一样，用眼睛和手感判断物体，知道用多大力抓取，避免滑落或损坏。这项技术让机器人变得更聪明，能在复杂环境中自主操作，比如仓库、家庭甚至医院，帮助人们完成各种任务。
ELI14 Explained like you're 14
想象你在厨房里做饭，你会用眼睛看食材的颜色和形状，还用手摸一摸它们的硬度和滑动感觉。比如，面包很软，坚果很硬。人类就是这样用视觉和触觉一起判断东西的。现在，科学家们让机器人也学会了这种能力。他们用一种特别的学习方法，让机器人看和摸东西，然后告诉它们这些东西的重量、硬度和摩擦力。这样，机器人就能像我们一样，用眼睛和手感判断物体，知道用多大力抓取，避免滑落或损坏。研究还发现，把人类示范的动作和感觉结合起来，机器人能更好地适应不同的物体。未来，这项技术可以让机器人在仓库、家里帮忙，甚至在医院里做事情，变得更聪明、更贴心！
Abstract
Recent vision-based action models have demonstrated strong capabilities in complex manipulation, but they rarely leverage explicit object physical properties to adapt their policies. We introduce ViTacPhys, a visual-tactile framework and data acquisition system that estimates object mass and friction-coefficient classes, together with continuous stiffness, from human manipulation demonstrations. Trained on data from 60 rigid and deformable objects, ViTacPhys combines temporal visual-tactile modeling, cross-attention multimodal fusion, and a semantic prior derived from a vision-language model. On seen objects, it achieves 97.2% mass classification accuracy, 98.8% friction-coefficient classification accuracy, and a stiffness mean absolute percentage error (MAPE) of 5.51%. On held-out objects from known categories, it achieves 87.5% mass accuracy, 97.5% friction-coefficient accuracy, and a stiffness MAPE of 9.08%. We transfer ViTacPhys from the human domain to the robot domain using limited robot teleoperation data, robot-style video augmentation, and human demonstrations with matched actions, and deploy it as an online module for adaptive grasping. The resulting physical-property-conditioned policy achieves total grasping success rates of 95.0% on in-distribution objects and 83.4% on out-of-distribution objects. For out-of-distribution objects successfully grasped by both methods, its force profiles are more consistent with human teleoperation than those produced by ACT. These results demonstrate the feasibility of explicitly estimating and conditioning on object physical properties for real-world adaptive grasping.