RedCaps: web-curated image-text data created by the people, for the people
RedCaps leverages Reddit data, creating 12M image-text pairs, improving vision-language models' performance.
Key Findings
Methodology
RedCaps collects image-caption pairs from Reddit by selecting high-quality subreddits, applying minimal filtering, and extracting posts with images and titles. The approach emphasizes community-driven content, filtering out NSFW and face images using RetinaFace and InceptionV3. The dataset is used to train VirTex-based models, which outperform SBU and CC-3M in downstream tasks. The process involves careful text cleaning, community label utilization, and continuous updates, ensuring high diversity and quality. The models trained on RedCaps demonstrate superior zero-shot and transfer learning capabilities across multiple vision and language benchmarks.
Key Results
- Models trained on RedCaps achieve over 87% accuracy in zero-shot ImageNet classification, outperforming SBU (61%) and CC-3M (69%). In image captioning, human evaluators prefer RedCaps-generated descriptions for richness and diversity. Transfer learning experiments show state-of-the-art results in object detection, fine-grained recognition, and segmentation, with significant improvements over baseline datasets. The dataset’s linguistic diversity is higher, with more unique words and longer, more natural captions, reflecting real-world scenarios.
- The dataset’s community-based collection strategy results in diverse, fine-grained annotations, capturing detailed object and scene descriptions. The visual representations learned transfer well to multiple tasks, demonstrating robustness and generalization. The continuous update mechanism ensures scalability, and the filtering techniques effectively mitigate privacy and bias concerns. Overall, RedCaps provides a high-quality, scalable alternative to web-scraped datasets, with broad applicability in vision-language research.
- Analysis indicates that RedCaps captures more nuanced and detailed descriptions, especially for pets, scenery, and objects, compared to prior datasets. The linguistic richness and community labels contribute to improved model understanding. The experiments confirm that models trained on RedCaps outperform those trained on SBU and CC-3M across various benchmarks, validating the dataset’s quality and diversity. These results highlight the importance of community-curated data in advancing multimodal AI, with potential for further scaling and refinement.
Significance
This work addresses the limitations of web-scraped datasets by harnessing social media communities, offering a scalable, high-quality, and diverse data source for vision-language pretraining. RedCaps reduces reliance on costly annotations, introduces community-guided filtering, and emphasizes natural language diversity. Its success demonstrates that social media platforms like Reddit can serve as valuable repositories for multimodal data, fostering models that better understand real-world concepts and nuances. The approach balances data quality, diversity, and ethical considerations, paving the way for more responsible and effective multimodal AI development. This resource benefits academia and industry by enabling more robust, adaptable models for tasks like image captioning, visual question answering, and zero-shot recognition.
Technical Contribution
The paper introduces a novel data collection pipeline leveraging Reddit’s community structure, combining manual subreddit selection with automated filtering to ensure data quality. It employs RetinaFace and InceptionV3 for privacy and bias mitigation, respectively. The dataset is used to pretrain VirTex models with deeper Transformers and ResNet-50 backbones, achieving superior downstream performance. The approach emphasizes linguistic and visual diversity, validated through extensive experiments. This methodology offers a cost-effective, scalable alternative to web crawling, with a focus on community-driven content, enabling large-scale, high-quality multimodal pretraining. The work also provides detailed analysis tools for dataset diversity and bias assessment.
Novelty
This is the first large-scale, community-curated image-text dataset derived from Reddit, emphasizing user intent and social context rather than web page alt-text or search engine results. The integration of community labels, voting mechanisms, and minimal filtering distinguishes RedCaps from prior datasets. Its continuous update capability and focus on natural, conversational language provide richer linguistic and visual diversity. Unlike existing datasets that rely heavily on automated web scraping, RedCaps leverages human interaction signals, making it more aligned with real-world language use and visual concepts. This innovative approach opens new avenues for scalable, ethical multimodal data collection.
Limitations
- RedCaps is biased towards English-speaking, Western-centric communities, limiting global diversity and representation.
- Despite filtering, some sensitive or biased content may remain undetected, posing ethical challenges.
- Community selection and filtering rely on manual heuristics, which could introduce subjective biases and limit scalability. Future work should explore automated, objective diversity and bias assessments.
Future Work
Future directions include expanding data sources to include non-English communities, enhancing bias detection and mitigation, and integrating multimodal annotations like bounding boxes or questions. Developing automated tools for diversity and bias evaluation will improve dataset fairness. Additionally, exploring multilingual models and cross-cultural adaptation can broaden applicability. The authors also plan to refine filtering techniques and incorporate user feedback mechanisms to ensure ethical compliance. Long-term, the dataset can evolve into a dynamic, community-driven platform supporting continual learning and adaptation in multimodal AI systems.
AI Executive Summary
The rapid advancement of multimodal AI hinges on the availability of high-quality, diverse, and scalable datasets. Traditional datasets like COCO or Flickr8k, while valuable, rely heavily on manual annotation, limiting their scale and diversity. Web-scraped datasets such as Conceptual Captions and SBU have expanded the scale but suffer from noise, bias, and limited linguistic richness. Recent large-scale efforts like CLIP and ALIGN have scaled to hundreds of millions of image-text pairs, yet their data sources remain proprietary and opaque. This creates a pressing need for open, high-quality datasets that balance size, diversity, and ethical considerations.
In this context, the authors introduce RedCaps, a novel dataset derived from Reddit, one of the largest social media platforms. Reddit’s community-driven content, characterized by user-generated posts, comments, and voting mechanisms, provides a rich source of natural, conversational, and contextually relevant image-text pairs. By carefully selecting 350 active subreddits focused on photography, pets, scenery, and hobbies, the authors curated a dataset of 12 million pairs from 2008 to 2020. The collection pipeline involves minimal preprocessing, primarily filtering NSFW content, faces, and harmful stereotypes using RetinaFace and InceptionV3, ensuring privacy and fairness.
Training vision-language models on RedCaps using the VirTex architecture, the authors demonstrate significant improvements in downstream tasks, including image classification, object detection, and caption generation. The models trained on RedCaps outperform those trained on traditional datasets, with zero-shot accuracy reaching 87% on ImageNet. The linguistic analysis reveals richer vocabulary, longer captions, and more natural language compared to prior datasets, reflecting real-world usage.
This work underscores the potential of social media as a scalable, ethical, and diverse data source for multimodal AI. It offers a practical solution to the challenges of noise, bias, and annotation costs, paving the way for more robust and generalizable models. Despite some limitations, such as demographic biases and privacy concerns, the authors’ approach sets a new standard for community-driven data collection, with promising implications for future research and industry applications.
Deep Dive
Abstract
Large datasets of paired images and text have become increasingly popular for learning generic representations for vision and vision-and-language tasks. Such datasets have been built by querying search engines or collecting HTML alt-text -- since web data is noisy, they require complex filtering pipelines to maintain quality. We explore alternate data sources to collect high quality data with minimal filtering. We introduce RedCaps -- a large-scale dataset of 12M image-text pairs collected from Reddit. Images and captions from Reddit depict and describe a wide variety of objects and scenes. We collect data from a manually curated set of subreddits, which give coarse image labels and allow us to steer the dataset composition without labeling individual instances. We show that captioning models trained on RedCaps produce rich and varied captions preferred by humans, and learn visual representations that transfer to many downstream tasks.