Common Corpus: The Largest Collection of Ethical Data for LLM Pre-Training

The evolution of large language models has been driven by the availability of massive datasets like Common Crawl and C4, which enabled models such as GPT-3 and BERT to achieve unprecedented performance. However, reliance on web scraping introduces legal uncertainties, bias, and data quality issues. Recent incidents, including copyright lawsuits and takedown of datasets like Books3 and LAION, highlight the fragility of current data sources. Consequently, the AI community recognizes the need for open, licensed, and diverse datasets that can support fair and sustainable development of multilingual models, especially for low-resource languages. Prior efforts like C4C, Open License Corpus, and Common Pile have made progress but remain limited in scope, often English-centric or domain-specific. This context underscores the importance of a comprehensive, legally compliant, and multilingual dataset like Common Corpus.

Existing large-scale datasets face critical issues: copyright restrictions, bias, low diversity, and lack of representation for low-resource languages. These limitations hinder the development of truly inclusive and fair AI systems. Moreover, legal challenges and takedown incidents threaten reproducibility and open science efforts. The core problem is to create a massive, ethically sourced, multilingual dataset that complies with legal standards, covers diverse domains, and supports scalable model training. Achieving this requires systematic data collection, rigorous licensing validation, and high-quality filtering, which are complex and resource-intensive tasks. Addressing these challenges is essential for democratizing AI and ensuring its societal benefits.

This work introduces a multi-layered data collection pipeline that integrates data from government, cultural, scientific, web, and code sources, all with open licenses or public domain status. Key innovations include: 1) comprehensive license verification using automated tools; 2) multi-language filtering with fastText; 3) OCR correction for cultural heritage texts; 4) toxicity and bias filtering; 5) multi-domain coverage ensuring broad representativeness. The dataset’s scale (2 trillion tokens) and diversity surpass existing open resources, enabling high-quality multilingual pretraining. The approach combines automation with manual curation, setting a new standard for responsible dataset construction, and demonstrates the feasibility of large-scale open data for AI research.

• �� Data collection from six domains: government, culture, science, web, code, and semantic sources, ensuring open licenses. • License validation: automated recognition of license types, filtering out restricted content. • Data cleaning: OCR correction for scanned documents, removal of low-quality and toxic content. • Language detection: using fastText to identify and categorize languages, ensuring diverse coverage. • Deduplication: hash-based algorithms to remove duplicates, maintaining dataset uniqueness. • Filtering: rule-based and model-based filtering for bias, toxicity, and relevance. • Data integration: merging multi-source datasets into a unified corpus, with detailed metadata for traceability.

Two small Transformer models (based on GPT-2 and BERT architectures) were trained on Common Corpus. Hyperparameters included a learning rate of 1e-4, batch size of 64, and 100k training steps. Evaluation used multilingual benchmarks such as Tatoeba, XTREME, and code understanding datasets. Ablation studies examined the impact of filtering steps and data diversity. Results showed BLEU and F1 scores above 85%, outperforming models trained on less curated datasets. Toxicity and bias assessments confirmed the ethical standards of the dataset, with significant reductions compared to Web-scraped data.

Models trained on Common Corpus achieved BLEU scores of 87% and F1 scores of 85% across multiple languages, outperforming baseline models trained on Web datasets by 5-7%. Low-resource languages like Swahili and Urdu showed marked performance improvements, with BLEU scores rising by 10%. Code understanding tasks also saw performance gains, validating the utility of code data. Filtering processes effectively reduced toxicity scores by 30%, demonstrating enhanced safety. The dataset’s diversity and quality directly contributed to these improvements, confirming its suitability for multilingual and low-resource language training.

The dataset enables training of inclusive multilingual models for translation, question-answering, and content generation. It supports development of legal, scientific, and cultural AI applications, especially in low-resource languages. Industries such as education, healthcare, and government can leverage it for multilingual communication tools, document processing, and knowledge extraction. Its open license facilitates collaborative research, reducing legal barriers and fostering innovation across sectors.

While comprehensive, the dataset may still contain biases inherited from source materials, affecting fairness. The filtering process, though rigorous, cannot eliminate all low-quality or toxic content. Scalability to larger models requires further validation. Additionally, expanding low-resource language coverage remains challenging due to limited available open data. Future work should focus on improving filtering algorithms, expanding data sources, and developing automated bias mitigation techniques.