OOD-CV: A Benchmark for Robustness to Out-of-Distribution Shifts of Individual Nuisances in Natural Images

Deep learning has revolutionized computer vision, achieving remarkable performance on standard benchmarks like ImageNet and COCO. However, real-world deployment exposes models to distribution shifts caused by environmental factors, such as weather, lighting, pose, and texture variations. Existing robustness evaluations mainly rely on synthetic corruptions (e.g., ImageNet-C) or cross-dataset tests (e.g., ImageNet-V2, ImageNet-A), which do not fully capture the complexity of real-world nuisances. Prior work on shape and texture biases (e.g., Geirhos et al., 2019) highlighted model vulnerabilities but lacked systematic disentanglement of individual factors. The need for realistic, fine-grained, multi-task datasets to evaluate and improve robustness remains pressing, especially for safety-critical applications like autonomous driving and robotics.

Current robustness assessments often conflate multiple environmental factors, obscuring the specific causes of performance degradation. Synthetic corruptions fail to replicate real-world complexities, and cross-dataset evaluations lack granularity. Consequently, models may perform well on aggregate metrics but fail under specific conditions, such as unusual poses or weather. Addressing this requires datasets with controlled, disentangled annotations of individual nuisances, enabling precise analysis of their impact on different vision tasks. Without such benchmarks, developing truly robust models remains a significant challenge, limiting AI deployment in unpredictable environments.

The paper's key innovations include: 1) a real-image dataset with disentangled annotations of individual nuisance factors (pose, shape, texture, context, weather); 2) a multi-task setup covering classification, detection, and 3D pose estimation; 3) a systematic evaluation of popular architectures and augmentation techniques under controlled single-factor OOD shifts. Data collection combined keyword-based internet searches, manual filtering, and CAD model alignment to ensure each sample varies only in one nuisance. This approach overcomes synthetic and cross-dataset limitations, providing a realistic, detailed benchmark for fine-grained robustness analysis. The dataset enables studying the independent effects of each factor, guiding targeted robustness improvements.

• �� Data collection: Use curated keywords to search for images with specific OOD nuisances, manually filter out multi-factor samples.
• �� Annotation: Label object class, 2D bounding box, and align CAD models for accurate 3D pose estimation.
• �� Data splitting: Create IID and OOD test sets, ensuring each OOD sample only differs in one nuisance factor.
• �� Model training: Fine-tune ResNet, Transformer, and 3D models on the dataset, applying data augmentation techniques like AugMix and style transfer.
• �� Evaluation: Measure performance with accuracy, mAP@50, and pose error; analyze performance drops per nuisance and architecture.
• �� Analysis: Conduct ablation studies on augmentation effects, architecture robustness, and multi-factor interactions.

The experiments involved training baseline models on IID data and testing on OOD splits with single nuisance shifts. Performance metrics included Top-1 accuracy for classification, mAP@50 for detection, and angular error for pose. The evaluation covered models with different capacities and architectures, assessing the impact of each nuisance factor independently. Additional experiments tested data augmentation methods' effectiveness, revealing limited improvements for geometric nuisances. Multi-factor OOD scenarios were also simulated to evaluate combined effects. Results consistently showed that weather and environmental shifts cause the largest performance drops, while shape and pose variations significantly affect 3D pose estimation. These findings highlight the importance of factor-specific robustness strategies.

Models' performance declined notably under certain nuisances; for example, classification accuracy dropped by 16% under weather shifts, detection mAP by 37% under context shifts. Architectures with 3D shape modeling showed better robustness to shape and pose variations. Data augmentation improved appearance-based nuisances but was ineffective for geometric changes. The analysis confirmed that different tasks depend on different cues, and robustness strategies need to be task-specific. The benchmark's detailed annotations enabled precise attribution of performance drops to individual factors, providing a clear roadmap for future robustness improvements.

This benchmark supports the development of more reliable vision systems in autonomous vehicles, robotics, and surveillance, where environmental variability is common. It allows researchers to identify specific weaknesses and tailor robustness techniques accordingly. The detailed annotations facilitate targeted model improvements, potentially leading to safer and more dependable AI applications in real-world scenarios. Additionally, it can guide industry standards for robustness evaluation, accelerating the deployment of trustworthy AI.

The dataset's collection process relies on internet searches and manual filtering, which may introduce bias and limit diversity. It currently covers only 10 object categories, restricting generalization. The evaluation focuses on deep architectures, with limited exploration of lightweight or emerging models. Computational costs for annotation and training are high. Future work should include more categories, multi-factor OOD shifts, and broader model evaluations to enhance applicability.