ShapeMap 3-D: Efficient shape mapping through dense touch and vision

The evolution of robotic perception has seen a shift from reliance solely on visual sensors to integrating multiple modalities, including tactile sensing. Early methods focused on point cloud and mesh-based reconstruction using LiDAR or stereo vision, which perform well in structured environments but falter under occlusion or poor lighting. The advent of high-resolution tactile sensors like GelSight introduced new possibilities for local shape sensing, enabling robots to perceive surface details with sub-millimeter accuracy. Prior works such as Shape Completion networks, high-res tactile learning, and Gaussian process surface modeling have demonstrated partial success in local shape estimation. However, these approaches often lack the ability to fuse multiple sensing modalities efficiently, especially in incremental, real-time settings. Moreover, existing models typically do not provide uncertainty estimates, which are vital for active perception and decision-making. The challenge remains to develop a unified framework that can leverage dense tactile data, compensate for visual occlusions, and operate efficiently in real-world, unstructured environments.

The core problem addressed is how to achieve accurate, real-time, and incremental 3D shape reconstruction of unknown objects using limited sensing resources. Specifically, the challenge is to fuse high-density tactile point clouds with noisy depth maps to generate a complete, detailed shape model that accounts for measurement uncertainties. Traditional methods struggle with computational scalability when integrating dense tactile data, and often lack probabilistic estimates of surface confidence. The limited sensing area of tactile sensors and occlusion issues in visual data further complicate the problem. Achieving a balance between local detail recovery and global shape consistency, while maintaining computational efficiency for real-time applications, remains an open problem. Addressing this gap is crucial for enabling robots to operate autonomously in unstructured, cluttered environments where visual information alone is insufficient.

The paper introduces several key innovations: 1) A deep neural network Ω trained in simulation to map GelSight tactile images to local height maps and contact masks, enabling high-fidelity local shape recovery; 2) A Gaussian process implicit surface (GPIS) model that probabilistically represents the object’s shape as an SDF, capturing surface uncertainty; 3) A sparse local kernel approximation within a spatial factor graph (GP-SG), transforming the global GP inference into a set of local Gaussian potentials, significantly reducing computational load; 4) An incremental optimization framework using iSAM2, allowing real-time updates of the shape estimate as new tactile and visual data are acquired. These innovations collectively enable scalable, uncertainty-aware, and high-precision shape mapping in complex environments, surpassing previous point-cloud or mesh-based methods.

• �� Tactile data acquisition: GelSight sensors capture high-resolution tactile images during robot-object contact, which are processed by the neural network Ω to produce height maps and contact masks. • Simulation-based training: Ω is trained on a large dataset of simulated GelSight interactions with diverse household objects, ensuring generalization to real-world data. • Multi-modal data fusion: The local tactile point cloud and depth map are combined, with the tactile data providing dense local details and the depth map offering coarse global context. • Probabilistic modeling: The combined data are modeled as Gaussian potentials within a Gaussian process framework, representing the SDF and its uncertainty across the 3D space. • Spatial graph construction: The 3D space is discretized into query nodes, with local neighborhoods defined by a radius r. Each measurement generates a Gaussian potential constraining nearby nodes, forming a sparse factor graph. • Incremental inference: Using iSAM2, the model updates the posterior distribution of the SDF in real-time, incorporating new measurements as they arrive. • Surface extraction: The zero-level set of the posterior SDF is obtained via Marching Cubes, generating a detailed 3D shape model with uncertainty visualization.

The experimental setup involves both simulated and real-world data collection. In simulation, the YCB dataset provides ground-truth meshes, with synthetic GelSight images generated via tactile simulation, and noise added to emulate real sensor behavior. The robot performs multiple contact interactions from different angles, and the shape reconstruction accuracy is evaluated using Chamfer distance, with results showing rapid convergence within 40 touches. In real-world experiments, a UR5 robot with GelSight mounted on a parallel gripper interacts with household objects secured on a mechanical bench. The Azure Kinect captures depth maps, while tactile data is collected through robot contact. The system processes data in real-time, updating the shape estimate after each contact. The evaluation metrics include Chamfer distance, surface uncertainty, and computational time, demonstrating the method’s robustness and efficiency across diverse object geometries and sensing conditions.

Simulation results show that the proposed method reduces the average Chamfer distance to approximately 18.3 mm after 40 touches, with the surface uncertainty decreasing significantly as more tactile data is integrated. The height map RMSE drops below 0.1 mm, outperforming baseline lookup table methods by nearly 50%. In real experiments, the system achieves similar accuracy, with errors stabilizing after about 30 touches. The incremental inference maintains a consistent computational time (~0.5 seconds per update), enabling real-time operation. Ablation studies confirm that the local kernel radius of 15% of object size balances accuracy and efficiency. The model’s uncertainty estimates effectively guide active sensing, improving shape completeness in occluded regions. Overall, the results validate the framework’s capability to produce high-fidelity, uncertainty-aware 3D reconstructions in both simulated and real environments.

This framework can be directly applied to robotic manipulation tasks requiring precise shape understanding, such as grasping, tool use, and assembly in cluttered environments. It is particularly useful in scenarios where visual data is unreliable due to occlusion or transparency. The ability to estimate surface uncertainty also enables robots to plan active sensing strategies, improving perception completeness. Long-term applications include autonomous warehouse robots capable of handling unknown objects, assistive robots in home environments, and industrial automation systems that require high-precision object modeling. The system’s incremental nature makes it suitable for continuous learning and adaptation in dynamic settings, supporting advanced manipulation and interaction capabilities.

The current approach assumes static objects with known approximate dimensions, limiting its applicability to dynamic or deformable objects. The manual tuning of local kernel parameters (e.g., neighborhood radius) affects adaptability across different scenarios. In cases of severe occlusion, sensor failure, or extreme noise, the robustness of the model may degrade, necessitating active sensing or sensor redundancy strategies. Computational complexity, although reduced, still poses challenges for very high-resolution reconstructions or large-scale scenes, requiring further optimization. Additionally, the model does not explicitly handle deformable objects or real-time shape changes, which are important for soft robotics and biological applications. Future work should address these limitations by incorporating adaptive parameter learning, active sensing, and deformable shape modeling.