Data Selection Through Iterative Self-Filtering for Vision-Language Settings
Proposes an iterative self-filtering method using CLIP to improve vision-language data quality without pretraining, boosting downstream performance.
Key Findings
Methodology
The approach involves iterative training and data selection based on similarity scores (like cosine similarity) derived from the model’s embeddings. Starting with a randomly initialized model, the process trains on a mixed dataset, evaluates sample credibility via embedding similarity, selects top-scoring samples, and then mixes these with the full dataset for subsequent rounds. This cycle continues, enabling the model to progressively filter out noisy samples and enhance data quality without external pretraining. The core mechanism leverages the model’s memorization and generalization capabilities, balancing exploitation of high-confidence samples with maintaining diversity through data mixing.
Key Results
- On the Datacomp benchmark, models trained with the proposed Self-Filtering method outperform those trained on full datasets or datasets filtered by pre-trained CLIP, achieving approximately 1.5% higher accuracy on ImageNet zero-shot classification, reaching 9.9%.
- In experiments with 128 million samples, iterative filtering over 8 rounds led to consistent performance improvements across multiple downstream tasks, validating the method’s robustness.
- Varying filtering ratios (e.g., 30%, 40%) demonstrated that optimal filtering enhances model generalization, with performance gains sustained across diverse datasets.
Significance
This work addresses the dependency on pre-trained models for data filtering, offering a self-sufficient, scalable solution for noisy large-scale datasets. It advances understanding of how model-driven data selection can improve downstream task performance, especially in multimodal settings. The method’s simplicity and effectiveness open pathways for autonomous data curation, reducing biases introduced by external models, and enabling more sustainable training pipelines for vision-language models. Its implications extend to real-world applications like content moderation, data augmentation, and large-scale dataset management, fostering more reliable AI systems.
Technical Contribution
The key technical innovation is an iterative framework that combines model training with self-guided data filtering based on embedding similarity scores. Unlike prior methods relying on external pre-trained models, this approach exploits the model’s evolving understanding to identify high-confidence samples. The process involves: 1) initial random training, 2) scoring samples via cosine similarity, 3) selecting top p% samples, 4) mixing selected samples with the full dataset, and 5) repeating. This creates a feedback loop that progressively refines data quality and model robustness. The method enhances sample efficiency, reduces bias, and is adaptable to various multimodal tasks, offering a new paradigm for noise-robust training.
Novelty
This is the first work to implement an entirely self-supervised, iterative filtering process that does not depend on external pretraining models. It leverages the model’s own embedding similarity to identify trustworthy samples, creating a closed-loop system that improves data quality over multiple rounds. Unlike existing approaches that rely heavily on external models or heuristics, this method dynamically adapts to the data distribution, making it highly scalable and less biased.
Limitations
- The effectiveness depends on the initial model’s capacity; early training stages may produce unreliable similarity scores, limiting filtering accuracy.
- Parameter tuning for filtering ratio and iteration count remains sensitive; improper settings can lead to loss of data diversity or insufficient noise removal.
- In datasets with extremely high noise levels, the model may struggle to distinguish noise from hard but useful samples, requiring further enhancements.
Future Work
Future directions include integrating active learning strategies to adaptively tune filtering parameters, exploring multi-modal feature fusion for better noise discrimination, and scaling the method to larger datasets and more complex tasks. Additionally, developing theoretical guarantees for convergence and robustness will be valuable for broader adoption.
AI Executive Summary
The rapid growth of large-scale vision-language models has underscored the importance of high-quality training data. However, collecting and curating such datasets at scale remains a significant challenge, especially given the prevalence of noisy, mislabeled, or irrelevant samples. Traditional filtering approaches often depend on pre-trained models, which introduce biases and limit flexibility. In response, this paper introduces a novel iterative self-filtering framework that leverages the model’s own embedding similarity scores to progressively refine the dataset without external pretraining. Starting from a randomly initialized model, the method trains on a mixed dataset, evaluates sample credibility via cosine similarity, and selects top-scoring samples for subsequent training rounds. This process creates a feedback loop, where the model’s understanding improves alongside data quality. Extensive experiments on the Datacomp benchmark demonstrate that models trained with this approach outperform those trained on full datasets or datasets filtered by pre-trained models, achieving about 1.5% higher accuracy on ImageNet zero-shot classification. The iterative filtering not only enhances data quality but also boosts downstream task performance, confirming the method’s effectiveness and scalability. Importantly, it eliminates reliance on external models, making it suitable for resource-constrained environments. The results suggest that self-guided data curation can be a powerful tool for building more robust, unbiased, and efficient vision-language systems. Future work will focus on adaptive parameter tuning, multi-modal feature integration, and theoretical analysis to further solidify this promising approach.
Deep Dive
Plain Language Accessible to non-experts
想象你在整理一个巨大的拼图，每一块拼图代表一段图片和文字的配对。刚开始，你不知道哪些拼图是正确的，哪些是错误的。于是，你先随机拼一些，看看拼出来的图像和文字是否匹配。然后，你用自己拼好的部分作为参考，筛选出那些匹配得最好的拼图，放到一起。接着，你用这些“靠谱”的拼图，再拼一遍，筛选出更好的组合。经过多次反复，你的拼图逐渐变得完整、漂亮。这就像论文中的方法，模型不断学习，筛选出最可信的数据，让整体变得更好、更准确。
ELI14 Explained like you're 14
想象你在玩一个超级复杂的拼图游戏，一开始你不知道哪些拼图是正确的。你先拼一些，看看哪些拼图最合适，然后用这些拼出来的部分作为线索，再筛掉那些不匹配的拼图。每次拼完后，你都用自己拼好的部分作为新线索，再筛一遍。经过几轮，你的拼图变得越来越完整，最后拼出一幅漂亮的画。这就像论文里的自我过滤方法，模型通过自己不断学习，筛选出最靠谱的数据，让整体变得更好。
Glossary
Self-Filtering (自我过滤)
一种利用模型自身的相似性评分，反复筛选可信样本的技术，避免依赖外部预训练模型。
论文提出的核心机制，用于逐步提升数据质量。
CLIP (对比语言-图像预训练)
由OpenAI开发的多模态模型，通过对比学习将图像与文本映射到共同空间，实现跨模态检索。
用于筛选和评分图像-文本对。
Cosine Similarity (余弦相似度)
衡量两个向量夹角余弦值的指标，用于评估图像与文本的匹配程度。
作为样本可信度评分的依据。
Open Questions Unanswered questions from this research
- 1 在极高噪声比例下，如何确保筛选的鲁棒性？当前方法在高噪声环境中表现有限，仍需改进。
- 2 多模态信息融合的潜力未充分开发，结合视觉和文本特征可能提升筛选效果。
- 3 筛选参数（比例、轮次）尚缺自适应调节机制，未来需研究自动调优策略。
Applications
Immediate Applications
大规模多模态数据清洗
企业或研究机构可用此方法自动筛除噪声，提升模型训练效率，适用于内容审核、数据增强等场景。
自主数据管理系统
无需外部预训练模型，构建自动筛选流程，降低成本，适合资源有限环境。
Long-term Vision
智能化数据筛选平台
未来可发展为全自动、多模态的智能数据管理系统，实现大规模数据的高效清洗和优化。
Abstract
The availability of large amounts of clean data is paramount to training neural networks. However, at large scales, manual oversight is impractical, resulting in sizeable datasets that can be very noisy. Attempts to mitigate this obstacle to producing performant vision-language models have so far involved heuristics, curated reference datasets, and using pre-trained models. Here we propose a novel, bootstrapped method in which a CLIP model is trained on an evolving, self-selected dataset. This evolving dataset constitutes a balance of filtered, highly probable clean samples as well as diverse samples from the entire distribution. Our proposed Self-Filtering method iterates between training the model and selecting a subsequently improved data mixture. Training on vision-language datasets filtered by the proposed approach improves downstream performance without the need for additional data or pre-trained models.