Diver Interest via Pointing in Three Dimensions: 3D Pointing Reconstruction for Diver-AUV Communication
DIP-3D integrates sparse stereo triangulation and pose estimation to locate diver-pointed objects in 3D, improving multi-object discrimination in underwater scenes.
Key Findings
Methodology
This approach combines sparse stereo triangulation with human pose detection, focusing on wrist, elbow, and shoulder keypoints. Using SIFT feature matching and Lowe's ratio test, candidate objects are identified in stereo images. These 2D points are reprojected into 3D space via a calibrated reprojection matrix. The system computes the perpendicular distance from each candidate object to an extended pointing vector derived from the pose, selecting the closest target. This modular pipeline effectively distinguishes multiple objects and enhances spatial understanding in underwater environments.
Key Results
- In closed-water tests, DIP-3D achieved an average positional error of 119.95 pixels and over 85% correct object identification. The system accurately determined the diver's pointing direction within 2-3 meters, with human annotation showing an average deviation of 45.9 pixels. Real-time performance was maintained at 0.857 fps, demonstrating suitability for underwater robotic applications.
- Distance estimation accuracy declined beyond 3 meters, highlighting the limitations of sparse triangulation at longer ranges. Filtering invalid depth points and pose anomalies improved robustness. The system outperformed dense stereo methods in noisy underwater conditions, confirming the advantage of sparse triangulation.
- Ablation studies indicated that combining pose estimation with sparse triangulation significantly reduced false positives and improved target localization stability, especially in feature-sparse underwater scenes.
Significance
This work advances underwater human-robot interaction by enabling natural diver-to-AUV communication through pointing gestures. The integration of distance-aware 3D localization addresses longstanding challenges of multi-object discrimination and spatial understanding underwater. It paves the way for safer, more intuitive collaboration in complex tasks like infrastructure inspection, ecological monitoring, and archaeological surveys. The approach enhances the autonomy and situational awareness of AUVs, bridging the gap between manual control and full autonomy, and supporting scalable underwater operations.
Technical Contribution
The core innovation lies in combining sparse stereo triangulation with pose-based gesture recognition, forming a robust 3D localization framework. The method leverages feature matching, reprojected 3D points, and perpendicular distance calculations to accurately identify the diver's pointed object among multiple candidates. Unlike dense disparity-based methods, this approach is more resilient to underwater optical distortions and feature sparsity, offering a scalable, modular solution adaptable to various underwater scenarios. It also introduces a systematic filtering process to improve depth reliability and target discrimination.
Novelty
This is the first work to integrate sparse stereo triangulation with human pose estimation for 3D target localization underwater, specifically addressing multi-object discrimination during diver-AUV interactions. The method overcomes the limitations of traditional dense stereo matching in low-visibility conditions, providing a practical and scalable solution. Its novelty lies in the targeted use of keypoints and geometric filtering to achieve accurate spatial localization, setting a new standard for underwater human-robot communication.
Limitations
- The approach relies heavily on accurate pose detection; non-standard or crossing arm poses can cause misclassification, requiring more robust pose models. Distance estimation accuracy diminishes at ranges beyond 3 meters, limiting long-range applications. Computational load, though acceptable at 0.857 fps, can be further optimized for real-time deployment in resource-constrained systems. Environmental factors like turbidity and low contrast still pose challenges, necessitating future improvements in feature detection and multi-view fusion.
Future Work
Future research will focus on integrating deep learning-based pose estimators to handle diverse and complex underwater postures, improving distance estimation at longer ranges, and enhancing multi-object discrimination. Incorporating multi-view stereo and deep neural networks could further boost robustness against water turbidity and lighting variations. Additionally, developing feedback mechanisms for more intuitive diver-AUV communication and expanding to full gesture classification will promote more natural, scalable underwater human-robot collaboration.
AI Executive Summary
Underwater human-robot interaction faces significant challenges due to poor visibility, optical distortions, and feature sparsity. Traditional dense stereo methods struggle to estimate accurate distances, especially in complex multi-object scenarios. This hampers natural communication between divers and autonomous underwater vehicles (AUVs), limiting operational efficiency in tasks like infrastructure inspection and ecological monitoring.
The presented DIP-3D framework addresses these issues by combining sparse stereo triangulation with human pose estimation. By detecting keypoints on the diver's arm—wrist, elbow, shoulder—and matching features with SIFT, the system reconstructs 3D positions of both the diver and potential targets. Extending the elbow-wrist vector into space, it calculates the perpendicular distance to candidate objects, selecting the most probable target. This modular approach effectively distinguishes multiple objects, even in feature-sparse underwater scenes.
Experimental results in controlled water environments demonstrate that DIP-3D achieves an average positional error of approximately 120 pixels and correctly identifies the diver's pointed object over 85% of the time within 2-3 meters. The system operates at nearly real-time speeds (0.857 fps), validating its practical applicability. The key innovation lies in leveraging sparse triangulation to mitigate water-induced optical distortions, outperforming dense stereo methods in noisy conditions.
This work significantly enhances underwater communication, enabling more natural and accurate diver-AUV collaboration. It opens pathways for scalable underwater operations, including maintenance, ecological surveys, and archaeological exploration. Future improvements will focus on robust pose detection, longer-range accuracy, and multi-view integration, pushing underwater robotics toward greater autonomy and human-like understanding.
Deep Dive
Abstract
This paper presents Diver Interest via Pointing in Three Dimensions (DIP-3D), a method to relay an object of interest from a diver to an autonomous underwater vehicle (AUV) by pointing that includes three-dimensional distance information to discriminate between multiple objects in the AUV's camera image. Traditional dense stereo vision for distance estimation underwater is challenging because of the relative lack of saliency of scene features and degraded lighting conditions. Yet, including distance information is necessary for robotic perception of diver pointing when multiple objects appear within the robot's image plane. We subvert the challenges of underwater distance estimation by using sparse reconstruction of keypoints to perform pose estimation on both the left and right images from the robot's stereo camera. Triangulated pose keypoints, along with a classical object detection method, enable DIP-3D to infer the location of an object of interest when multiple objects are in the AUV's field of view. By allowing the scuba diver to point at an arbitrary object of interest and enabling the AUV to autonomously decide which object the diver is pointing to, this method will permit more natural interaction between AUVs and human scuba divers in underwater-human robot collaborative tasks.