Task-Agnostic Noisy Label Detection via Standardized Loss Aggregation
Standardized Loss Aggregation (SLA) detects noisy labels by aggregating standardized fold validation losses, outperforming hard-count baselines.
Key Findings
Methodology
SLA employs multiple rounds of K-fold cross-validation, extracting validation losses for each sample, standardizing these losses within each fold, and aggregating across repetitions. Specifically, features are extracted via a frozen pretrained encoder ϕ, reduced with PCA, and classified with a linear discriminant analysis (LDA). In each fold, validation loss is computed, standardized by subtracting the mean and dividing by the maximum standard deviation, then accumulated over repetitions. This process captures both the frequency and magnitude of performance deviations, providing a stable, interpretable noisiness score that reflects label reliability in a model-agnostic manner.
Key Results
- On the public fundus dataset JustRAIGS, SLA significantly outperforms ReCoV across noise ratios from 0.1% to 10%, with AUROC improvements at low noise levels (e.g., 0.5%) from 0.765 to 0.859 after 1,000 repetitions. The standardized scores show clean samples centered around zero, while noisy samples shift towards higher values, demonstrating effective separation. Convergence is rapid, with high detection accuracy achieved within a few thousand repetitions, and computational cost remains low (~0.9 seconds per repetition on CPU).
- The distribution of noisiness scores under SLA reveals clear separation: clean samples cluster near zero, noisy samples form a distinct higher mean distribution. This normalization allows consistent comparison across datasets and models, facilitating practical dataset auditing and re-annotation prioritization.
- Experimental results confirm that SLA converges faster and is more sensitive to subtle loss variations than baseline methods, especially in low-noise scenarios, making it suitable for large-scale, real-world applications where annotation quality is critical.
Significance
This approach addresses a key challenge in medical imaging and other high-stakes domains: reliably identifying mislabeled or ambiguous samples without task-specific tuning. By providing a statistically grounded, task-agnostic, and computationally efficient method, SLA enhances dataset quality control, reduces manual re-labeling efforts, and improves model robustness. Its universal applicability across classification tasks and datasets makes it a valuable tool for data curation, especially in clinical settings where annotation uncertainty is common. The method's ability to produce interpretable scores supports transparent decision-making and systematic dataset refinement, ultimately advancing trustworthy AI deployment.
Technical Contribution
The paper introduces a novel continuous noisiness scoring mechanism based on standardized fold-level validation losses, extending beyond traditional discrete count-based approaches like ReCoV. The framework leverages statistical principles—mean and variance normalization, ensemble averaging—to ensure robustness and interpretability. It integrates PCA for dimensionality reduction and employs simple yet effective classifiers like LDA, enabling large-scale repeated validation with minimal computational overhead. Theoretically, SLA provides an unbiased estimator of label noisiness with decreasing variance as repetitions increase, grounded in the law of large numbers. This combination of statistical rigor and engineering simplicity advances the state-of-the-art in sample-level noise detection.
Novelty
This work is the first to formulate a continuous, statistically grounded noisiness score based on standardized fold validation losses, moving beyond the binary or count-based methods prevalent in prior research. Unlike existing approaches that focus on worst-fold detection, SLA captures the magnitude and frequency of performance deviations, providing a nuanced, interpretable spectrum of label reliability. Its task-agnostic design and theoretical guarantees distinguish it from heuristic or model-dependent techniques, offering a scalable, principled solution for noisy label detection in diverse datasets.
Limitations
- The evaluation relies on synthetic label noise introduced by random flips, which may not fully replicate real-world annotation errors, such as systematic biases or borderline cases.
- High noisiness scores can sometimes reflect ambiguous or borderline samples rather than outright mislabels, requiring additional criteria for definitive filtering.
- Choosing an appropriate threshold for filtering remains task-dependent, and the method's performance under extreme class imbalance or highly complex data distributions warrants further investigation.
Future Work
Future directions include integrating active learning strategies to adaptively determine the number of repetitions, combining SLA with multi-modal data for richer noise signals, and extending the framework to multi-label and regression tasks. Additionally, exploring adaptive thresholding and more sophisticated feature extraction could further enhance detection accuracy and applicability in real-world clinical workflows.
AI Executive Summary
In recent years, deep learning has revolutionized medical image analysis, yet the reliability of training data remains a persistent challenge. Label noise—arising from inter-observer variability, ambiguous cases, and systematic biases—undermines model performance and clinical trustworthiness. Existing solutions, such as robust loss functions and noise transition matrices, often lack the granularity to identify individual unreliable samples, especially in large-scale datasets. To address this, the paper introduces Standardized Loss Aggregation (SLA), a novel, statistically grounded framework that quantifies label noisiness by aggregating standardized validation losses across multiple cross-validation repetitions.
The core idea involves extracting image features via a pretrained encoder, reducing dimensionality with PCA, and training a simple classifier like LDA. During each fold, validation losses are computed, standardized by subtracting the mean and dividing by the maximum standard deviation within each fold, then accumulated over many repetitions. This process captures both the frequency and magnitude of performance deviations, producing a continuous noisiness score for each sample. Unlike discrete count-based methods, SLA offers a smooth, interpretable measure that reflects the stability and reliability of labels.
Experimental validation on the publicly available JustRAIGS fundus dataset demonstrates SLA's superiority over baseline methods like ReCoV. Across noise ratios from 0.1% to 10%, SLA consistently achieves higher AUROC scores, especially at low noise levels where subtle loss variations are informative. The scores follow clear distributions: clean samples cluster near zero, noisy samples shift higher, confirming effective separation. Moreover, SLA converges rapidly, requiring only a few thousand repetitions to reach stable performance, with low computational cost (~0.9 seconds per repetition on CPU). This efficiency makes it practical for large-scale dataset auditing.
The implications of this work are significant. By providing a task-agnostic, interpretable, and scalable tool for label quality assessment, SLA facilitates targeted re-annotation, improves dataset reliability, and enhances model robustness. Its statistical foundation ensures reproducibility and cross-model comparability, supporting transparent AI deployment in clinical and industrial settings. Future work will explore adaptive repetition strategies, multi-modal integration, and extension to more complex annotation scenarios, paving the way for more trustworthy AI systems.
Deep Dive
Abstract
Noisy labels are common in large-scale medical imaging datasets due to inter-observer variability and ambiguous cases. We propose a statistically grounded and task-agnostic framework, Standardized Loss Aggregation (SLA), for detecting noisy labels at the sample level. SLA quantifies label reliability by aggregating standardized fold-level validation losses across repeated cross-validation runs. This formulation generalizes discrete hard-counting schemes into a continuous estimator that captures both the frequency and magnitude of performance deviations, yielding interpretable and statistically stable noisiness scores. Experiments on a public fundus dataset demonstrate that SLA consistently outperforms the hard-counting baseline across all noise levels and converges substantially faster, especially under low noise ratios where subtle loss variations are informative. Samples with high SLA scores indicate potentially ambiguous or mislabeled cases, guiding efficient re-annotation and improving dataset reliability for any classification task.