MYCROFT: Towards Effective and Efficient External Data Augmentation
Mycroft combines feature distance and gradient matching to efficiently select informative data subsets from private sources, boosting model performance with minimal data sharing.
Key Findings
Methodology
Mycroft integrates feature space distance and gradient similarity to identify small, relevant data subsets from private sources. Using gradient-based similarity (via loss gradients) and feature-based distances, it employs Orthogonal Matching Pursuit (OMP) to select data that closely approximates the utility of full data sharing. The approach adapts to different data types by selecting suitable feature extractors and similarity measures, and combines both modalities into a joint optimization objective. Theoretical analysis shows the regularized similarity function is weakly submodular, ensuring near-optimal subset selection. Empirical results across vision and tabular datasets demonstrate rapid convergence to full-data performance with a fraction of the data, robustness to noise, and effective owner ranking.
Key Results
- Across four classification tasks, Mycroft achieves performance within 2-3% of the full-data baseline while only using 20-30% of the data, with an average performance boost of 21%. In image datasets, it outperforms random sampling by at least 15% at all budgets. For tabular data, just 5 samples selected by Mycroft surpass the performance of 100 samples chosen randomly, reaching full-data performance in 65% of cases. The method maintains robustness under noisy labels and multiple data sources, effectively ranking data owner utility.
- Ablation studies reveal early checkpoints provide more informative gradients for matching, and combining feature and gradient similarity enhances selection quality. The approach generalizes well across different model architectures, including ResNet50 and EfficientNetB0, and across data modalities. The computational cost is manageable, and the method consistently outperforms baselines in diverse scenarios, validating its practicality.
- This work advances private data utilization by enabling targeted, high-quality data sharing, reducing overhead and privacy risks. It democratizes access to high-performance models, especially benefiting small organizations and individual developers. The approach opens new avenues for privacy-aware transfer learning, federated learning integration, and large-scale distributed AI, with significant implications for industry and academia.
Significance
Mycroft addresses a fundamental bottleneck in AI development—how to leverage private, proprietary data without compromising privacy or incurring high costs. By intelligently selecting only the most relevant data subsets, it enables performance gains comparable to full data sharing, but with vastly reduced exposure. This breakthrough facilitates democratization of AI training, making high-quality models accessible to smaller entities and privacy-sensitive sectors like healthcare and finance. Its robustness to noise and multi-source environments further broadens its applicability, paving the way for scalable, privacy-preserving distributed learning frameworks that can operate under strict data governance policies.
Technical Contribution
This research introduces a novel combination of gradient-based and feature-based similarity metrics, unified into a joint optimization framework. The use of OMP for efficient subset selection, coupled with theoretical guarantees of weak submodularity, distinguishes it from prior methods relying solely on random sampling or single-modality similarity. The approach adapts existing tools for different data types, including tabular and image data, and demonstrates that combining multiple similarity notions improves relevance and robustness. Theoretical analysis confirms near-optimality, and extensive experiments validate superior empirical performance, establishing a new standard for privacy-aware data augmentation.
Novelty
This is the first work to unify gradient similarity and feature space distance into a single, joint optimization framework for data subset selection in privacy-constrained settings. Unlike prior methods that depend on sharing entire datasets or simple heuristics, Mycroft leverages multi-modal similarity measures and theoretical properties of weak submodularity to guarantee near-optimal selection. Its adaptive, multi-faceted approach to relevance assessment marks a significant step forward in privacy-preserving transfer learning and data-efficient model training, setting a new benchmark in the field.
Limitations
- The method relies on pre-trained models and feature extractors, which may limit applicability in domains lacking such resources. Gradient matching effectiveness depends on checkpoint selection and model architecture, potentially affecting transferability across different models. In extremely noisy or mislabeled data environments, relevance estimation may degrade. Computational costs, while manageable, still pose challenges for very large datasets, necessitating further optimization or approximation techniques.
Future Work
Future directions include integrating Mycroft with federated learning frameworks to enhance privacy guarantees, exploring unsupervised or semi-supervised relevance metrics, and extending the approach to multi-modal and streaming data. Developing scalable algorithms for larger datasets and real-time relevance assessment will be crucial. Additionally, investigating robustness under adversarial conditions and applying the framework to other domains like healthcare, finance, and IoT will broaden its impact and usability.
AI Executive Summary
Mycroft introduces an innovative framework for privacy-aware external data augmentation, enabling model trainers to identify and acquire highly relevant data subsets from private sources with minimal exposure. Traditional data augmentation relies on sharing entire datasets, but this approach faces significant privacy, legal, and economic barriers. Mycroft circumvents these issues by leveraging the similarity between local data samples and task-specific challenging examples, using a combination of feature space distances and gradient matching techniques.
The core idea involves three steps: first, the trainer shares the difficult test samples (Dh) with data owners; second, each owner employs a tailored similarity measure—either gradient-based or feature-based—to select a small, highly relevant subset (Duseful); third, the trainer evaluates this subset to decide whether further data sharing is warranted. The key technical innovation is the joint optimization of these similarity measures, underpinned by theoretical guarantees of near-optimality via weak submodularity. The algorithms utilize Orthogonal Matching Pursuit (OMP) for efficient subset selection.
Extensive experiments across vision and tabular datasets demonstrate that Mycroft achieves performance close to full data sharing while only using a fraction of the data—often less than 30%. In image classification tasks, it outperforms random sampling by over 15%, converging rapidly to baseline performance. In structured data, just five samples selected via Mycroft can surpass the effectiveness of 100 randomly chosen samples, significantly reducing data exchange costs.
This approach has profound implications for democratizing AI development, especially in privacy-sensitive sectors like healthcare, finance, and cybersecurity. It enables smaller organizations and individuals to leverage private data effectively, fostering broader participation in high-performance model training. While challenges remain—such as computational costs and model transferability—the framework opens new pathways for privacy-preserving, data-efficient AI. Future work will focus on integrating with federated learning, expanding to multi-modal data, and optimizing scalability, promising a transformative impact on the field.
Deep Dive
Abstract
Machine learning (ML) models often require large amounts of data to perform well. When the available data is limited, model trainers may need to acquire more data from external sources. Often, useful data is held by private entities who are hesitant to share their data due to propriety and privacy concerns. This makes it challenging and expensive for model trainers to acquire the data they need to improve model performance. To address this challenge, we propose Mycroft, a data-efficient method that enables model trainers to evaluate the relative utility of different data sources while working with a constrained data-sharing budget. By leveraging feature space distances and gradient matching, Mycroft identifies small but informative data subsets from each owner, allowing model trainers to maximize performance with minimal data exposure. Experimental results across four tasks in two domains show that Mycroft converges rapidly to the performance of the full-information baseline, where all data is shared. Moreover, Mycroft is robust to noise and can effectively rank data owners by utility. Mycroft can pave the way for democratized training of high performance ML models.