CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models
CrowS-Pairs dataset quantifies social biases in MLMs via 1508 sentence pairs across nine bias types, revealing widespread bias.
Key Findings
Methodology
The study employs crowdsourced minimal sentence pairs, each contrasting stereotypical and anti-stereotypical sentences about disadvantaged groups across nine bias categories. Using conditional pseudo-log-likelihood, the authors estimate the likelihood of each sentence under models like BERT, RoBERTa, and ALBERT, controlling for word frequency biases. The dataset is validated through multiple annotators, ensuring high quality. The bias is quantified by the model’s preference for stereotypical sentences, with results indicating significant bias across all models and categories, especially religion.
Key Results
- All evaluated models show a bias favoring stereotypical sentences, with average bias scores around 60.5%, well above the neutral 50%. BERT exhibits the lowest bias, while RoBERTa and ALBERT show higher bias levels. Bias is most pronounced in religion, less so in gender and race. Models demonstrate lower bias on anti-stereotype examples (~15%), but still favor stereotypes significantly. Confidence distributions correlate with bias levels, with more biased models showing higher confidence in biased choices.
- The dataset’s high validation rate (~80%) and diversity surpass prior datasets like StereoSet. The bias measurement method effectively captures subtle preferences, and the results highlight the pervasive nature of social biases in mainstream MLMs. The findings underscore the importance of bias mitigation strategies in NLP models.
- The bias preference metric reveals that models tend to prefer stereotypical sentences, with bias levels varying across categories. The data suggests that bias is deeply ingrained, necessitating targeted debiasing. The evaluation framework provides a robust benchmark for future bias reduction efforts.
Significance
This work provides a rigorous, scalable approach to quantifying social biases in language models, addressing a critical challenge in AI fairness. By offering a high-quality, diverse dataset and a reliable bias metric, it enables researchers to benchmark and improve bias mitigation techniques. The widespread bias observed underscores the societal impact of training data and model design choices, emphasizing the urgency for ethical AI development. The dataset’s focus on real-world, minimally templated sentence pairs enhances ecological validity, making it a valuable tool for both academia and industry to assess and reduce biases in NLP systems.
Technical Contribution
The paper introduces CrowS-Pairs, a crowdsourced dataset of 1508 sentence pairs covering nine bias categories, validated for quality. It proposes a novel bias measurement metric based on conditional pseudo-log-likelihood, which controls for word frequency biases, providing more accurate bias quantification. The evaluation of BERT, RoBERTa, and ALBERT reveals consistent bias patterns, establishing a benchmark for future bias mitigation research. The methodology combines crowdsourcing, rigorous validation, and advanced likelihood estimation, representing a significant step forward in bias measurement techniques.
Novelty
This is the first large-scale, crowdsourced bias dataset that emphasizes minimal sentence pairs with real-world diversity across multiple bias types. The introduction of conditional pseudo-log-likelihood as a bias metric is a key innovation, reducing frequency bias effects common in previous methods. Compared to prior datasets like StereoSet, CrowS-Pairs achieves higher validation accuracy and broader bias coverage, setting a new standard for bias evaluation in NLP.
Limitations
- The dataset relies on crowdworker annotations, which may introduce subjective biases and cultural limitations. Its focus on US-centric biases limits cross-cultural generalization.
- The bias measurement primarily captures model preferences for stereotypical sentences, not the deeper semantic or societal impacts of biases. It does not address intersectional biases comprehensively.
- Evaluation is limited to MLMs, and the effectiveness of debiasing strategies based on this metric remains to be validated. The approach may not fully capture complex, contextual biases.
Future Work
Future research should expand bias categories to include more cultural contexts and intersectional biases. Developing dynamic, real-time bias detection and mitigation methods integrated into training pipelines is crucial. Exploring bias at deeper semantic levels and across multimodal systems will enhance understanding. Additionally, refining metrics to better reflect societal impacts and deploying tools for continuous bias monitoring in deployed models are promising directions.
AI Executive Summary
The rapid advancement of pretrained language models such as BERT and RoBERTa has revolutionized natural language processing, enabling unprecedented performance across tasks. However, these models inadvertently learn and perpetuate social biases embedded in their training data, raising ethical concerns. Existing bias evaluation datasets like StereoSet and WinoBias provided initial insights but faced limitations in diversity, reliability, and ecological validity. To address this, Nangia et al. introduced CrowS-Pairs, a large-scale, crowdsourced dataset comprising 1508 minimally contrasting sentence pairs across nine bias categories, including race, gender, religion, and age.
The core innovation lies in the bias measurement metric based on conditional pseudo-log-likelihood, which estimates model preferences while controlling for word frequency biases. This approach ensures a more accurate quantification of social bias. The evaluation of three leading MLMs—BERT, RoBERTa, and ALBERT—revealed that all exhibit significant bias favoring stereotypical sentences, with bias levels notably high in religion and lower in gender and race categories. These findings highlight the pervasive nature of social biases in mainstream NLP models.
The dataset’s high validation rate (~80%) and diversity make it a robust benchmark for future bias mitigation efforts. The results underscore the urgent need for debiasing techniques that can reduce harmful biases without degrading language understanding. The study’s methodology and findings provide a critical foundation for developing fairer NLP systems, fostering trust and equity in AI deployment.
Looking ahead, expanding bias categories, improving bias detection methods, and integrating bias mitigation into training pipelines are essential. The research emphasizes that addressing social bias is not only a technical challenge but also a societal imperative, vital for ensuring AI systems serve all communities equitably.
Deep Dive
Abstract
Pretrained language models, especially masked language models (MLMs) have seen success across many NLP tasks. However, there is ample evidence that they use the cultural biases that are undoubtedly present in the corpora they are trained on, implicitly creating harm with biased representations. To measure some forms of social bias in language models against protected demographic groups in the US, we introduce the Crowdsourced Stereotype Pairs benchmark (CrowS-Pairs). CrowS-Pairs has 1508 examples that cover stereotypes dealing with nine types of bias, like race, religion, and age. In CrowS-Pairs a model is presented with two sentences: one that is more stereotyping and another that is less stereotyping. The data focuses on stereotypes about historically disadvantaged groups and contrasts them with advantaged groups. We find that all three of the widely-used MLMs we evaluate substantially favor sentences that express stereotypes in every category in CrowS-Pairs. As work on building less biased models advances, this dataset can be used as a benchmark to evaluate progress.