Neural feels with neural fields: Visuo-tactile perception for in-hand manipulation
NeuralFeels integrates vision and touch, online learning neural fields for real-time unknown object pose and shape reconstruction, enhancing robotic in-hand perception.
Key Findings
Methodology
This paper introduces NeuralFeels, which combines visual, tactile, and proprioceptive data on a multi-fingered robotic hand to learn neural field models online. The system encodes object geometry via neural signed distance functions (Neural SDFs) and jointly optimizes pose graphs for accurate tracking. Using deep neural networks for feature extraction and classical SLAM optimization techniques, it performs real-time object pose and shape estimation during in-hand manipulation. The approach handles unknown objects without prior models, leveraging multimodal data to improve robustness against occlusion and noise, validated through extensive simulation and real-world experiments.
Key Results
- Across 70 experiments, NeuralFeels achieved an average reconstruction F-score of 81%, with pose drift as low as 4.7mm. When using known CAD models, pose error reduced further to 2.3mm. Under heavy occlusion, tactile data improved tracking accuracy by up to 94% compared to vision-only methods. These results demonstrate the system’s robustness and high precision in complex manipulation scenarios.
- In known-object tracking, utilizing CAD models, the system maintained an average pose error of 2mm, with tactile feedback refining estimates further. The experiments confirmed that multimodal fusion significantly outperforms single-modality approaches, especially in occluded or noisy environments, validating the effectiveness of the neural field-based SLAM framework.
- The integration of vision and touch provided substantial improvements in shape reconstruction and pose stability, with tactile sensing playing a critical role in disambiguating visual ambiguities and compensating for occlusion. The system maintained high accuracy across diverse objects and conditions, highlighting its potential for real-world robotic applications.
Significance
This work advances robotic perception by integrating multimodal sensing into a unified neural SLAM framework, enabling real-time, robust object reconstruction and pose estimation during in-hand manipulation. It addresses longstanding challenges of occlusion, noise, and unknown object handling, bridging the gap between perception and dexterous manipulation. The approach paves the way for more autonomous, adaptable robots capable of operating in unstructured environments, with broad implications for service, industrial, and household robotics. Its modular design and online learning capability make it a versatile platform for future developments in spatial AI.
Technical Contribution
The core innovation lies in combining neural fields with multimodal data fusion and pose graph optimization, enabling online, real-time learning of object geometry and pose. The neural signed distance function (Neural SDF) encodes detailed shape information, while the pose graph ensures consistent tracking. The system integrates deep neural feature extractors with classical SLAM optimization, allowing flexible incorporation of tactile and visual data without extensive prior training. This hybrid approach offers high fidelity, robustness, and interpretability, setting new standards for neural perception in manipulation tasks.
Novelty
This is the first demonstration of a full SLAM system driven by multimodal perception—vision, touch, and proprioception—for in-hand object manipulation. Unlike prior works limited to static reconstruction or known objects, NeuralFeels performs online, real-time, unknown object tracking and shape estimation. Its integration of neural fields with pose graph optimization in an embodied, interactive setting is a novel contribution, significantly advancing the state-of-the-art in robotic perception and manipulation.
Limitations
- The system struggles with highly transparent, reflective, or geometrically complex objects due to sensor limitations and neural field expressiveness. Handling such cases requires further sensor and model improvements.
- Real-time performance is constrained by computational costs of neural network inference and optimization, especially for high-resolution reconstructions or complex scenes. Hardware acceleration and model compression are needed for deployment in time-critical applications.
- Robustness under extreme occlusion or rapid object motion remains a challenge, as current models depend on sufficient sensory input and stable data streams. Future work should focus on enhancing robustness and efficiency.
Future Work
Future directions include optimizing neural field architectures for faster inference, integrating adaptive sensor fusion strategies, and extending to multi-object scenarios. Developing more robust models for transparent and reflective objects, along with hardware acceleration, will be key. Additionally, expanding the framework to support diverse manipulation tasks like in-hand reorientation, insertion, and multi-object interaction will broaden its applicability. The ultimate goal is to realize fully autonomous, perceptually aware robots capable of complex, dexterous manipulation in unstructured environments.
AI Executive Summary
Robotic manipulation in unstructured environments demands highly accurate and robust perception of objects during in-hand operations. Traditional vision-based systems often falter under occlusion, poor lighting, or complex geometries, limiting their effectiveness. This paper introduces NeuralFeels, a novel framework that fuses visual, tactile, and proprioceptive data to perform online learning of neural field models, enabling real-time, high-fidelity reconstruction and tracking of unknown objects.
The core innovation lies in encoding object geometry via neural signed distance functions (Neural SDFs) and optimizing pose graphs that jointly refine shape and pose estimates. This multimodal fusion significantly enhances robustness, especially under occlusion, where tactile feedback can improve tracking accuracy by up to 94%. Extensive experiments in simulation and real-world settings demonstrate that NeuralFeels achieves an average F-score of 81% and maintains pose drift below 5mm across diverse objects and conditions.
The system’s ability to perform online, real-time SLAM for unknown objects marks a significant step forward in embodied spatial reasoning. It addresses key challenges of perception under occlusion, noise, and dynamic interaction, paving the way for more autonomous and dexterous robots. Its modular design allows easy integration with different sensors and models, offering a flexible platform for future research.
While promising, the approach faces limitations in handling highly transparent or reflective objects and maintaining real-time performance in complex scenes. Future work aims to optimize neural architectures, extend to multi-object scenarios, and improve robustness against extreme occlusion and rapid movements. Overall, NeuralFeels sets a new benchmark for multimodal, neural-based perception in robotic manipulation, with broad implications for industry and service robotics.
Deep Analysis
Background
机器人感知技术经历了从单一视觉到多模态融合的发展。早期工作如PoseCNN、DenseFusion主要依赖RGB-D数据进行静态场景的姿态估计，解决了有限环境中的重建问题。随着深度学习的兴起，神经场（Neural Fields）如Neural Radiance Fields（NeRF）被引入，用于高质量三维重建，但多模态融合仍有限。触觉传感器如GelSight、DIGIT的出现，使得密集接触信息成为可能，推动了触觉SLAM与重建研究。近年来，结合视觉与触觉的研究逐渐增多，但多模态在线学习与动态交互中的感知仍是挑战。
Core Problem
现有系统多局限于已知物体或静态环境，难以应对未知物体的实时感知与重建。视觉在遮挡、光照变化时表现不佳，触觉虽补充信息但缺乏有效融合机制。多模态感知的在线学习与动态优化仍未成熟，限制了机器人在复杂交互中的自主能力。如何实现高效、鲁棒的多源信息融合，支持未知物体的即时几何与姿态估计，是亟待解决的核心问题。
Innovation
提出NeuralFeels框架，首次实现多模态感知驱动的全SLAM系统，融合视觉、触觉与本体感知，在线学习神经场模型。引入基于pose图的联合优化机制，有效融合多源信息，提升感知鲁棒性。采用深度神经网络进行特征提取，结合经典SLAM优化算法，实现高效、精确的物体追踪与重建。模型支持未知物体的即时学习，突破了传统静态模型的限制。
Methodology
- �� 前端：利用预训练模型提取视觉与触觉深度信息，融合多模态特征。
- �� 神经场训练：采样深度数据，训练神经签名距离场（Neural SDF），编码物体几何。
- �� 后端优化：构建姿态图，利用非线性最小二乘法（如Levenberg-Marquardt）优化姿态与神经场参数。
- �� 联合更新：在交互过程中不断迭代，融合视觉与触觉信息，提升模型一致性与精度。
- �� 目标：实现未知物体的实时几何重建与姿态追踪，支持遮挡与噪声环境。
Experiments
在模拟与真实环境中，使用不同物体（如Rubik’s Cube、大骰子、橡胶鸭）进行70次交互实验。采用多模态传感器，收集视觉、触觉与本体信息，评估重建F-score与姿态漂移。对比基线（仅视觉）与融合模型，进行消融分析，验证触觉的贡献。关键指标包括F-score、ADD-S误差、姿态漂移，实验在不同遮挡与噪声条件下进行，确保模型鲁棒性。
Results
模型在未知物体上实现81%的F-score，平均姿态漂移为4.7mm，利用已知CAD模型可降至2.3mm。在遮挡严重场景中，触觉信息带来的跟踪提升高达94%，显著优于仅视觉方案。融合多模态信息后，重建精度与姿态稳定性均大幅提升，验证了多源感知的有效性。消融实验显示，触觉在遮挡与噪声环境中尤为关键，显著改善感知性能。
Applications
该技术可应用于家庭、工业机器人中的复杂操作任务，如物体重建、装配、插入等。只需配备低成本的视觉与触觉传感器，即可实现高精度感知，提升自主操作能力。未来还可扩展到多物体、多任务环境，支持自主学习与适应，推动机器人智能化发展。
Limitations & Outlook
模型在极端复杂几何、透明或高反光物体的重建仍有限，受传感器性能影响较大。遮挡与高速运动场景中误差仍存在，需优化融合策略。计算成本较高，实时性在高复杂度场景下有待提升，未来需结合硬件加速与模型轻量化技术。
Plain Language Accessible to non-experts
想象你在厨房里做饭，手里拿着各种不同的工具和食材。有时候你用眼睛看清楚，有时候用手摸索确认食材的大小和形状。机器人也是一样，它们用眼睛（视觉）看到东西，但有时候视线被遮挡或光线不好，不能完全看清。触觉就像用手摸东西一样，可以帮忙确认物体的形状和位置。这个研究就像教机器人用眼睛和手一起工作，学会在看不清楚时也能知道东西在哪里、长什么样。它们通过不断试错，学习如何在复杂环境中找到目标，就像我们在黑暗中摸索一样。最终，机器人可以更聪明、更稳健地完成各种任务，比如装配、抓取等。
ELI14 Explained like you're 14
想象你在玩拼图游戏，有时候看不清拼图的边缘，或者被其他拼图片挡住了视线。这时候，你用手摸一摸拼图的边缘，确认它的形状和位置。机器人也是这样，它们用眼睛看东西，但有时候视线被遮挡或光线不好，就会看不清楚。这篇研究让机器人学会用“眼睛”和“手”一起工作，既用视觉，也用触觉，来更好地知道物体长什么样、在哪里。它们通过不断尝试和调整，变得越来越聪明，能在复杂的环境中找到目标。就像你在黑暗中摸索，最后还是能拼好拼图一样，机器人也能在困难环境中完成任务，比如抓取、装配、搬运等。这个技术让机器人变得更像人类一样聪明、灵活！
Abstract
To achieve human-level dexterity, robots must infer spatial awareness from multimodal sensing to reason over contact interactions. During in-hand manipulation of novel objects, such spatial awareness involves estimating the object's pose and shape. The status quo for in-hand perception primarily employs vision, and restricts to tracking a priori known objects. Moreover, visual occlusion of objects in-hand is imminent during manipulation, preventing current systems to push beyond tasks without occlusion. We combine vision and touch sensing on a multi-fingered hand to estimate an object's pose and shape during in-hand manipulation. Our method, NeuralFeels, encodes object geometry by learning a neural field online and jointly tracks it by optimizing a pose graph problem. We study multimodal in-hand perception in simulation and the real-world, interacting with different objects via a proprioception-driven policy. Our experiments show final reconstruction F-scores of $81$% and average pose drifts of $4.7\,\text{mm}$, further reduced to $2.3\,\text{mm}$ with known CAD models. Additionally, we observe that under heavy visual occlusion we can achieve up to $94$% improvements in tracking compared to vision-only methods. Our results demonstrate that touch, at the very least, refines and, at the very best, disambiguates visual estimates during in-hand manipulation. We release our evaluation dataset of 70 experiments, FeelSight, as a step towards benchmarking in this domain. Our neural representation driven by multimodal sensing can serve as a perception backbone towards advancing robot dexterity. Videos can be found on our project website https://suddhu.github.io/neural-feels/