FeelAnyForce: Estimating Contact Force Feedback from Tactile Sensation for Vision-Based Tactile Sensors
FeelAnyForce uses a multi-head Transformer with RGB and depth images to estimate 3D contact forces, achieving 4% MAE on unseen objects.
Key Findings
Methodology
The approach employs a multi-head Transformer architecture, with a pre-trained ViT encoder, trained on a dataset of over 200,000 contact indentations collected via a robotic arm pressing various primitives onto a GelSight Mini sensor. The model simultaneously performs force regression and depth image reconstruction, leveraging multi-task learning to improve generalization. Depth information enhances the model's understanding of contact geometry, while balanced sampling ensures robustness across object types and sensor variations. The training uses a combined loss function with weighted force and depth errors, optimized via Adam for 100 epochs.
Key Results
- The model achieves a mean absolute error of 4% on a test set of unseen real-world objects, outperforming ResNet and single Transformer baselines by at least 20%. It maintains errors between 4% and 6.3% across different sensors and materials, demonstrating strong robustness. In downstream tasks such as object weighing and delicate deformation control, errors remain within 6%-10%, confirming practical applicability.
- Cross-sensor evaluations show that after a simple calibration procedure, the model maintains high accuracy with only 4% error increase on different GelSight Mini and DIGIT sensors. The approach generalizes well to various object textures and shapes, including complex and non-convex geometries.
- The proposed calibration method, using a 3D-printed setup and known weights, enables rapid adaptation to new sensors with minimal effort, ensuring consistent performance across hardware platforms.
Significance
This work addresses a critical bottleneck in robotic tactile sensing—accurate, generalizable force estimation from high-resolution vision-based sensors. By enabling precise force feedback across diverse objects and sensors, it advances robotic manipulation, dexterous handling, and human-robot interaction. The ability to generalize across devices reduces deployment costs and complexity, fostering broader adoption in industrial automation, healthcare, and service robotics. The integration of depth cues and multi-task learning sets a new standard in tactile sensing research, bridging the gap between high-resolution sensing and practical force estimation.
Technical Contribution
The core innovation lies in integrating depth information into a pre-trained ViT-based encoder within a multi-head Transformer framework, trained with a multi-objective loss to simultaneously estimate forces and reconstruct contact depth images. This design enhances the model's ability to focus on force-relevant features, improving generalization across objects and sensors. The calibration procedure, based on a simple 3D-printed fixture and off-the-shelf weights, allows rapid adaptation to different hardware setups. Extensive experiments validate the model's robustness, outperforming existing CNN and marker-based methods by significant margins, with a focus on real-world applicability.
Novelty
This is the first work to combine depth-aware Transformer architectures with large-scale tactile datasets for force estimation, achieving state-of-the-art accuracy and cross-device generalization. Unlike prior methods limited to specific sensors or object types, this approach leverages pre-trained vision transformers and multi-task learning to produce a versatile, high-precision force estimator. The calibration process is simple yet effective, enabling quick deployment across multiple hardware platforms, marking a substantial step forward in tactile sensing technology.
Limitations
- The model's accuracy diminishes slightly under extreme shear forces or rapid dynamic contacts, due to unmodeled nonlinear mechanical behaviors and lighting variations. Further research is needed to incorporate temporal dynamics and nonlinear modeling.
- Current calibration relies on manual setup with known weights, which could be automated for better scalability. The approach may face challenges in highly cluttered or unstructured environments.
- While robust across multiple sensors, the model's performance on highly deformable or elastic objects under high force remains to be fully explored. Computational costs of transformer-based models also limit real-time deployment in resource-constrained systems.
Future Work
Future directions include integrating temporal information for dynamic force estimation, developing fully automated calibration methods, and extending the framework to multi-modal sensing combining tactile, visual, and proprioceptive data. Exploring unsupervised or semi-supervised learning could reduce data collection efforts. Additionally, deploying the system in real-world industrial and healthcare scenarios will validate its robustness and scalability.
AI Executive Summary
Robotic manipulation demands precise force feedback, yet existing tactile sensors face limitations in accuracy and generalization. This study introduces FeelAnyForce, a novel approach leveraging a multi-head Transformer architecture trained on a large, diverse dataset of tactile images and forces collected via a robotic arm. The core innovation integrates depth information with RGB images, enabling the model to accurately estimate 3D contact forces across various objects and sensors. The model employs a pre-trained Vision Transformer (ViT) as encoder, combined with multi-task learning to simultaneously regress forces and reconstruct contact depth images, leading to robust performance and excellent generalization.
The dataset comprises over 200,000 samples from 10 primitive indenters, covering a wide force range up to 15N. The training process balances force and depth reconstruction losses, ensuring the model captures meaningful contact features. Experimental results demonstrate a mean absolute error of only 4% on unseen real-world objects, outperforming traditional CNN-based methods by at least 20%. The approach maintains high accuracy across different sensors, including GelSight Mini and DIGIT, after a simple calibration procedure using a 3D-printed fixture and known weights.
Beyond static force estimation, the model proves effective in downstream tasks such as object weighing and delicate deformation control, achieving errors below 10%. Its robustness across diverse materials and geometries signifies a major step toward practical tactile sensing in robotics. The calibration method simplifies deployment, making the technology accessible for real-world applications.
Overall, FeelAnyForce bridges the gap between high-resolution tactile sensing and accurate force feedback, enabling robots to perform dexterous, delicate tasks with human-like perception. Future work will focus on dynamic scenarios, fully automated calibration, and multi-modal sensing integration, broadening its impact in industry and healthcare. This work marks a significant advancement in tactile perception, promising more intelligent, adaptable robotic systems.
Deep Dive
Plain Language Accessible to non-experts
想象你在厨房里用手摸水果。不同水果有不同的硬度、形状和表面纹理，你可以凭感觉判断它们的重量和新鲜程度。科学家们用一种特殊的“电子手套”模拟这种感觉，这个“手套”能通过拍摄接触时的细微变化，估算出水果的重量和软硬程度。这个技术就像你用手触摸水果一样，能告诉机器人它们的重量和柔软度。它通过学习大量的“触摸照片”，变得越来越聪明，能在没有专用称重器的情况下，准确判断物体的重量和变形。这让机器人在搬运、分类甚至细心处理易碎物品时，变得像人一样灵巧。未来，这项技术还能让机器人在工厂、医院等场合，像人一样细腻地感知世界，帮我们做更多事情。
ELI14 Explained like you're 14
想象你有个超级厉害的机器人手，它能用眼睛看，还能用“皮肤”感觉东西。这个“皮肤”其实是个特别的相机，能拍到接触的细节，就像你用手摸东西一样。科学家们教这个机器人用很多“照片”学习怎么判断它碰到的东西有多重、多硬。以前，机器人要用特别的传感器才能知道这些，但那很贵也不够灵活。现在，这个新方法用“眼睛”和“脑袋”里的聪明算法，让机器人可以用普通的“相机”猜出东西的重量和软硬，就像你用手感觉一样。它还能帮机器人在搬东西、拿易碎品时更小心，甚至在未来帮忙做很多复杂的工作。这个技术就像给机器人装上了“超级感官”，让它变得更聪明、更灵巧。
Abstract
In this paper, we tackle the problem of estimating 3D contact forces using vision-based tactile sensors. In particular, our goal is to estimate contact forces over a large range (up to 15 N) on any objects while generalizing across different vision-based tactile sensors. Thus, we collected a dataset of over 200K indentations using a robotic arm that pressed various indenters onto a GelSight Mini sensor mounted on a force sensor and then used the data to train a multi-head transformer for force regression. Strong generalization is achieved via accurate data collection and multi-objective optimization that leverages depth contact images. Despite being trained only on primitive shapes and textures, the regressor achieves a mean absolute error of 4\% on a dataset of unseen real-world objects. We further evaluate our approach's generalization capability to other GelSight mini and DIGIT sensors, and propose a reproducible calibration procedure for adapting the pre-trained model to other vision-based sensors. Furthermore, the method was evaluated on real-world tasks, including weighing objects and controlling the deformation of delicate objects, which relies on accurate force feedback. Project webpage: http://prg.cs.umd.edu/FeelAnyForce