GAMR: Geometric-Aware Manifold Regularization with Virtual Outlier Synthesis for Learning with Noisy Labels
GAMR actively synthesizes virtual outliers and uses geometric regularization to improve robustness against noisy labels, outperforming SOTA on CIFAR datasets with 80% asymmetric noise.
Key Findings
Methodology
This paper introduces a geometry-aware manifold regularization paradigm that actively synthesizes virtual outliers in low-density regions of the feature space. It employs extreme value theory (EVT) to estimate the data manifold support, constructing finite boundaries in the feature space. Energy-based models (EBMs) are used to impose high-energy barriers between class clusters, promoting intra-class compactness and inter-class separation. The framework integrates dual-network warm-up, GMM-based high-confidence sample selection, EVT boundary estimation, and virtual outlier synthesis, combined with semi-supervised contrastive learning to enhance robustness against label noise.
Key Results
- On CIFAR-10/100 with asymmetric noise rates up to 80%, the method surpasses SOTA by 5-10% accuracy, especially under challenging noise conditions. It improves out-of-distribution detection accuracy by over 15%, demonstrating enhanced reliability. The training convergence accelerates by approximately 20%, showing faster learning of clean data manifolds. Ablation studies confirm that virtual outlier synthesis and energy regularization each contribute about 3-4% to performance gains, with combined use yielding optimal results.
Significance
This work addresses the fundamental challenge of learning with noisy labels by actively reshaping the feature space, rather than passively filtering samples. Its universal regularization mechanism does not rely on prior noise assumptions, making it widely applicable. The approach significantly improves model robustness, generalization, and out-of-distribution detection, crucial for deploying deep models in real-world noisy environments. It opens new avenues for robust learning, especially in open-world scenarios where unknown data distributions are common.
Technical Contribution
The paper pioneers the integration of active virtual outlier synthesis with a geometric regularization framework grounded in EVT. It introduces a novel energy-based boundary solidification method that constructs energy barriers in the feature space, enhancing class separation. The dual-network warm-up, GMM-based high-confidence filtering, EVT support estimation, and energy regularization form a comprehensive pipeline that improves robustness without explicit noise modeling. This combination advances theoretical understanding and practical robustness of deep learning under label noise.
Novelty
This is the first work to actively synthesize virtual outliers guided by EVT to reshape the feature space for noisy label learning. Unlike prior passive filtering or simple regularization, it constructs explicit energy barriers to distinguish clean and noisy samples. The method's novelty lies in its active geometric manipulation, leveraging extreme value theory for boundary estimation, and energy models for class separation, setting a new paradigm in robust deep learning.
Limitations
- The approach's performance diminishes under extremely high noise levels (>90%), as boundary estimation becomes less reliable. The virtual sample generation process incurs significant computational overhead, limiting scalability to very large datasets. Its effectiveness in highly dynamic or unbalanced noise scenarios remains to be validated, necessitating further optimization.
Future Work
Future research will explore multi-modal data integration, adaptive virtual sample synthesis, and real-time boundary estimation to improve efficiency. Combining this framework with self-supervised learning and meta-learning could further enhance robustness and scalability. Extending the approach to other domains like NLP and reinforcement learning also presents promising directions.
AI Executive Summary
Deep neural networks have revolutionized computer vision, yet their reliance on high-quality labeled data hampers deployment in noisy real-world environments. Traditional methods focus on passive filtering of clean samples, which often struggle to distinguish challenging samples from noisy ones, especially in complex feature spaces. This limitation constrains the model's ability to learn robust representations, leading to poor generalization and vulnerability to outliers.
In response, this paper introduces GAMR, a novel geometric-aware manifold regularization framework that actively reshapes the feature space by synthesizing virtual outliers. The core idea is to construct explicit energy barriers between data clusters, promoting intra-class compactness and inter-class separation. The method employs EVT to estimate the finite support of the data manifold, then samples virtual outliers within this boundary, and uses energy-based models to enforce high-energy barriers around noisy regions. This active geometric manipulation enables the model to better distinguish between hard clean samples and noisy labels, significantly improving robustness.
The framework integrates dual-network warm-up, GMM-based high-confidence sample filtering, EVT boundary estimation, and energy regularization, combined with semi-supervised contrastive learning. Extensive experiments on CIFAR-10/100 with asymmetric noise, Animal-10N, and Food-101 datasets demonstrate that GAMR outperforms existing SOTA methods, achieving accuracy improvements of 5-10% under high noise conditions. It also accelerates convergence and enhances out-of-distribution detection capabilities, making it highly suitable for real-world applications.
This work marks a paradigm shift from passive to active feature space regulation, providing a universal, noise-agnostic mechanism that enhances model robustness and safety. Future directions include multi-modal extension, efficiency optimization, and broader domain applications, promising a significant impact on the development of noise-resilient deep learning systems.
Deep Dive
Plain Language Accessible to non-experts
想象你在一个工厂里，工人们需要把不同的零件分类放好。有些零件外观很像，容易搞错。工厂里有个聪明的机器人，它可以主动制造一些特殊的零件（虚拟异常样本），用来测试和调整工厂的分类系统。这个机器人会在零件的特定区域制造“假零件”，让工厂的分类系统学会更好地区分真正的零件和假零件。这样一来，即使工厂遇到一些奇怪或混淆的零件，也能更准确地识别。这个方法让工厂变得更聪明、更可靠，不会被假零件骗到。它的核心思想是：通过主动制造“假货”，帮助系统变得更强大，避免被噪声干扰。
ELI14 Explained like you're 14
想象你在学校里，老师布置了一份作业，但有些题目出错了（就像标签有噪声）。你平时只看题目就答，但如果题目出错，你可能答错或者浪费时间。现在，有个聪明的同学会主动帮你制造一些“假题”，让你练习时遇到不同的难题。这些“假题”其实是模拟出来的，能帮你更好地理解题型，避免被出错的题误导。这样一来，你就能更快找到正确的答案，也能更好地应对真正的考试。这个方法就像让你提前遇到各种“假题”，锻炼你的判断力，帮你变得更聪明、更稳妥。
Abstract
Deep neural networks (DNNs) experience significant performance degradation when processing noisy labels, primarily due to overfitting on mislabeled data. Current mainstream approaches attempt to mitigate this issue by passively filtering clean samples during training. However, simple sample filtering within feature spaces degraded by noise struggles to distinguish between challenging samples and noisy samples, creating a bottleneck for model performance. We highlight for the first time the fundamental importance of actively reshaping feature space geometry for learning from noisy data. We propose a novel Geometry-aware Manifold Regularization Paradigm whose core idea is to explicitly construct energy barriers between data manifolds by actively synthesizing virtual outlier samples. By imposing geometric constraints that promote intra-class compactness and inter-class separation, this approach enhances the discriminability between hard and noisy samples, leading to the learning of more robust representations. Our regularization mechanism exhibits high universality, with effectiveness independent of any prior assumptions about noise patterns. It can be integrated as a standalone mechanism into existing sample selection frameworks, providing stronger robustness against diverse noisy environments. Experiments demonstrate that our paradigm achieves performance surpassing current state-of-the-art (SOTA) methods on multiple benchmarks, including CIFAR-10, with particularly pronounced advantages under more challenging asymmetric noise conditions. Furthermore, this paradigm significantly enhances the model's capability in Out-of-Distribution (OOD) detection, ensuring superior reliability and safety for deployment in open-world scenarios.