WildFin: An In-the-Wild Dataset for Fish Behavioral Recognition

Marine ecosystems rely heavily on understanding fish behaviors, which influence ecological balance and biodiversity. Historically, behavioral studies depended on manual observation, limited by scope and subjectivity. Recent advances in computer vision have enabled automated analysis, but most datasets are lab-based or from controlled environments, lacking ecological realism. Existing ecological video datasets are small and often lack detailed behavior annotations, restricting model generalization. WildFin addresses this gap by providing a large-scale, real-world dataset capturing authentic underwater behaviors across diverse habitats, facilitating robust model development for ecological monitoring.

Automated recognition of fish behaviors in natural underwater environments faces significant hurdles: occlusion from dense schools, dynamic backgrounds, water turbidity, and lighting variability. These factors cause high false positives/negatives in detection and misclassification of behaviors, especially rare or subtle actions. Moreover, existing models trained on controlled data perform poorly in the wild, limiting their utility for ecological applications. The challenge is to develop models that are robust to environmental noise, capable of handling class imbalance, and adaptable to diverse habitats, all while maintaining high accuracy and efficiency.

This work introduces several innovations: 1) a large, annotated dataset capturing real-world underwater behaviors across multiple scenarios; 2) the integration of static and spatiotemporal foundation models, leveraging transfer learning for robustness; 3) imbalance-aware training techniques like focal loss and class weighting, improving rare behavior detection; 4) a comprehensive evaluation framework that tests models under ecological complexities. These innovations collectively enable more accurate, scalable behavior recognition in challenging underwater environments, surpassing prior lab-centric approaches.

• �� Data collection: fixed CoralCam stations and diver-follow videos capturing diverse reef habitats. • Detection: YOLOv8 trained on expert-labeled bounding boxes to identify fish species. • Tracking: BoTSort algorithm links detections into trajectories, accommodating occlusions. • Annotation: ecologists assign frame-level behavior labels across 23 categories based on expert tracks. • Model training: freeze foundation models (DINOv3, VideoMAE), fine-tune with focal loss and class weights. • Evaluation: measure AP and F1 scores, analyze performance across scenarios, and perform ablation studies to optimize architecture choices.

The dataset is split into training and testing sets (~70/30), ensuring scene diversity. Multiple models are trained, comparing static (ResNet50) and spatiotemporal (VideoMAE, V-JEPA2) architectures. Hyperparameters like learning rate, batch size, and class weights are tuned. Ablation experiments assess the impact of data augmentation, imbalance strategies, and model complexity. Performance is evaluated on detection accuracy, behavior classification, and robustness to occlusion and motion. Cross-scenario tests validate model generalization, and error analyses identify failure modes.

Models achieved [email protected] of 74.2% on CoralCam and F1 of 0.78 on FishFollow. Incorporating imbalance techniques improved rare behavior detection by 15%. Spatiotemporal models outperformed static ones in dynamic scenes, yet still struggled with occlusion and rapid movements. Results demonstrate the effectiveness of transfer learning from foundation models, but also reveal the persistent challenge of environmental complexity, especially in low-visibility conditions. These insights guide future model improvements.

The developed models can be integrated into autonomous underwater vehicles and monitoring stations for real-time behavior analysis, aiding in marine conservation, species monitoring, and ecological research. They enable scalable, continuous data collection, reducing reliance on manual annotation and improving detection of ecologically significant behaviors. This technology can support policy-making, habitat management, and biodiversity assessments, fostering smarter ocean stewardship.

Despite advances, the models are still limited by occlusion, water turbidity, and rapid movements. Annotation biases and tracking inaccuracies affect performance, especially for short or subtle behaviors. High computational costs hinder real-time deployment. Future work should focus on multi-modal data fusion, active learning, and model efficiency to overcome these challenges and extend applicability across broader ecological contexts.