Semantically-Aware Diver Activity Recognition Framework for Effective Underwater Multi-Human-Robot Collaboration

The development of underwater robotics has seen rapid progress over the past decades, driven by applications in marine exploration, environmental monitoring, and resource extraction. Early efforts focused on autonomous navigation and target detection using sonar and acoustic sensors. With the advent of deep learning, vision-based methods employing CNNs such as VGG, ResNet, and later 3D CNN variants like C3D and SlowFast have been applied to underwater scene understanding. However, these models primarily address static object detection or simple motion tracking, lacking the capacity to interpret complex human activities. Existing datasets are limited, often capturing isolated diver poses or static scenes, which restricts the training of models capable of understanding nuanced interactions. Recently, Transformer architectures have revolutionized natural language processing and visual tasks, offering superior long-range dependency modeling. Applying these architectures to underwater activity recognition is promising but uncharted territory. The lack of large-scale, annotated datasets tailored for underwater multi-human-robot interactions further hampers progress. This paper bridges these gaps by introducing a novel framework and dataset, advancing the field toward intelligent underwater perception.

The core challenge in underwater diver activity recognition lies in the environment's inherent complexity—poor visibility, dynamic lighting, and water turbidity obscure visual cues. Additionally, the diversity of activities and interactions among multiple divers and robots creates a high-dimensional, ambiguous recognition problem. Existing models struggle to maintain accuracy under these conditions, especially with limited training data. The absence of large, annotated datasets prevents deep models from learning robust features. Furthermore, subtle differences between activities, such as 'busy' versus 'collaborative' states, are difficult to distinguish based solely on short video clips. These issues collectively hinder the deployment of reliable, real-time activity recognition systems essential for safe and efficient underwater operations.

This work introduces several key innovations:

1. Transformer Integration: Leveraging Transformer modules for temporal modeling enhances the understanding of long-term dependencies in underwater activities, surpassing traditional CNNs.

2. Scene Semantics Supervision: Pixel-level annotations guide the model's focus on relevant scene elements, improving robustness in low-visibility scenarios.

3. Multi-task Learning: Joint optimization of activity classification and scene segmentation fosters comprehensive scene understanding.

4. Dataset Creation: The UDA dataset provides high-quality, pixel-annotated images across six activity categories, filling a critical resource gap.

These innovations collectively enable the model to better interpret complex, noisy underwater scenes, facilitating accurate activity recognition and interaction understanding.

• �� Feature Extraction: Utilized ResNeXt-101 to extract deep features from underwater video frames, incorporating positional encoding to preserve spatial information.
• �� Temporal Modeling: Input features are processed through a Transformer encoder, employing self-attention to capture long-range temporal dependencies.
• �� Multi-task Architecture: The network branches into a classification head (Transformer-based) and a segmentation decoder (encoder-decoder structure), both trained jointly.
• �� Loss Functions: Defined a combined loss with classification cross-entropy and pixel-wise semantic binary cross-entropy, with trainable weights α and β to balance tasks.
• �� Training Strategy: Employed data augmentation techniques, trained on the UDA dataset for 200 epochs using AdamW optimizer, with a learning rate of 10^-5.
• �� Evaluation: Used accuracy, precision, recall, and F1-score on a separate test set, including ablation studies to assess the impact of semantic supervision.

• �� Dataset: The UDA dataset contains over 2600 images with pixel-level annotations, covering six activity categories in controlled water tank environments.
• �� Baselines: Compared DAR-Net against models like 3DResNet, R(2+1)D, SlowFast, and LateTemporal, retrained on the same dataset.
• �� Evaluation Metrics: Assessed performance using accuracy, precision, recall, F1-score, and confusion matrices.
• �� Hyperparameters: Batch size 4, learning rate 10^-5, 200 epochs, data augmentation applied.
• �� Ablation Studies: Compared models with and without semantic supervision, visualized attention maps, and analyzed misclassification cases to validate the importance of scene semantics.

• �� DAR-Net achieved 73.33% accuracy, outperforming all baseline models significantly, with the next best being LateTemporal at 66.67%.
• �� Precision, recall, and F1-score metrics confirmed the model’s balanced performance, with precision at 76.90% and F1-score at 72.17%.
• �� Attention map analysis demonstrated that semantic supervision directs the model’s focus toward relevant scene regions, reducing false positives.
• �� Confusion matrices revealed high accuracy in most categories, with some confusion between 'busy' and 'collaborative' activities, indicating areas for further refinement.
• �� Ablation results confirmed that scene semantics supervision enhances model focus and accuracy, especially in cluttered or low-visibility scenarios.

• �� Immediate: Deployment in autonomous underwater vehicles for real-time diver activity monitoring, enhancing safety and operational efficiency.
• �� Long-term: Development of comprehensive underwater perception systems capable of understanding complex interactions, supporting scientific research, resource management, and disaster response.

• �� Dataset size remains limited, potentially restricting model generalization to diverse real-world scenarios.
• �� Experiments are confined to controlled water tank environments; performance in open ocean conditions with variable currents and turbidity needs validation.
• �� Recognizing subtle or complex activities remains challenging, requiring further multimodal data integration and longer temporal context analysis.