RedBench: A Universal Dataset for Comprehensive Red Teaming of Large Language Models
RedBench consolidates 37 datasets with 22 risk categories and 19 domains, enabling comprehensive LLM vulnerability evaluation.
Key Findings
Methodology
RedBench integrates 37 high-quality datasets from venues like NeurIPS, ACL, ICLR, totaling 29,362 samples. It employs a standardized taxonomy with 22 risk categories and 19 domains, annotated via semi-automated processes combining Qwen2.5-72B-Instruct and human validation. The analysis includes distribution patterns, baseline performance of models like Llama 3.1, Gemma, GPT-4, assessing vulnerabilities in attack and refusal scenarios. Data sources span diverse fields, emphasizing attack and defense contexts, ensuring broad evaluation coverage.
Key Results
- Attack samples are dominated by Abusive Content (3,523) and Cybersecurity Threats (2,906), indicating prevalent vulnerabilities. Models like RainbowPlus achieve up to 97.81% attack success rate, while GPT-4 variants show robustness with success rates below 7%. In refusal tasks, Llama 3.1 exhibits a high average rejection rate of 28.53%, suggesting over-cautious behavior, whereas Gemma-2 shows lower rejection (13.46%), highlighting model safety versus usability trade-offs.
- Analysis reveals significant model weaknesses in risks such as Extremism and Misinformation, especially within Technology and Healthcare domains. The dataset's imbalance underscores areas needing targeted expansion, guiding future robustness improvements.
Significance
RedBench pioneers a unified, standardized platform for large-scale red teaming, addressing inconsistencies in existing datasets. Its comprehensive analysis informs model design, helping researchers understand vulnerabilities across diverse scenarios. By providing open-source benchmarks, it accelerates industry adoption of safer, more reliable LLMs, fostering trust and deployment in critical sectors like healthcare, finance, and law.
Technical Contribution
The work introduces a unified taxonomy with 22 risk categories and 19 domains, combined with a semi-automated annotation pipeline leveraging state-of-the-art LLMs. It consolidates 37 datasets into a high-density, multi-faceted resource, establishing baseline performances for models like Llama 3.1, Gemma, and GPT-4. This framework standardizes evaluation protocols, enabling consistent, scalable robustness assessments across models and scenarios.
Novelty
This is the first effort to unify diverse red-teaming datasets under a common taxonomy, addressing fragmentation issues. The semi-automated annotation process, integrating LLMs with human oversight, ensures high-quality labels. Additionally, it systematically benchmarks recent models like Qwen 2.5 and Llama 3.1, filling a gap in current research on modern model robustness across multiple risk domains.
Limitations
- Certain risk categories like Confabulation and specific domains such as Religion are underrepresented, limiting comprehensive coverage. The semi-automated labeling, despite validation, may introduce biases. Evaluation focuses mainly on select models, requiring broader testing for generalization. Dataset imbalance suggests need for targeted expansion in future iterations.
Future Work
Future efforts include expanding dataset diversity, especially in underrepresented categories and domains. Developing multi-modal evaluation methods and automating annotation further will enhance scalability. Cross-sector collaborations are essential to establish industry-wide safety standards. Additionally, integrating reinforcement learning techniques could improve model robustness, ensuring safer deployment in real-world applications.
AI Executive Summary
As large language models (LLMs) become integral to numerous applications, their safety and robustness are increasingly critical. Existing red teaming datasets, such as AdvBench and HarmBench, suffer from fragmented risk definitions and limited coverage, hampering systematic vulnerability assessment. To address this, RedBench consolidates 37 datasets, totaling 29,362 samples, into a unified platform with a standardized taxonomy of 22 risk categories and 19 domains.
This comprehensive dataset employs a hybrid annotation approach, combining advanced LLMs like Qwen2.5-72B-Instruct with human validation, ensuring high-quality, consistent labels. Analysis of the dataset reveals that attack samples predominantly target harmful content and security threats, exposing vulnerabilities in models like Llama 3.1, which shows a high over-defense tendency (average rejection rate of 28.53%). Conversely, models like GPT-4 demonstrate stronger resilience, with attack success rates below 7%.
The evaluation benchmarks established using RedBench highlight significant disparities in model robustness across risk categories and domains. For instance, RainbowPlus achieves near-perfect attack success rates (~97.81%) on open-source models, indicating the effectiveness of sophisticated red-teaming methods. The dataset’s balanced yet detailed coverage enables precise identification of weaknesses, guiding future model improvements.
Open-sourcing RedBench promotes transparency, reproducibility, and community collaboration, fostering the development of safer, more reliable LLMs. Future directions include expanding dataset diversity, enhancing multi-modal evaluation, and establishing industry-wide safety standards, ultimately ensuring that large models can be deployed securely in real-world scenarios without compromising trust or safety.
Deep Analysis
Background
Recent advances in LLMs such as GPT-4, Llama, and PaLM have revolutionized NLP, enabling multi-task, multilingual, and domain-specific applications. Despite these achievements, safety concerns—like generating harmful content, misinformation, or biased outputs—persist. Existing datasets like AdvBench and HarmBench provide targeted evaluations but lack a unified framework, leading to fragmented assessments. As models grow larger and more capable, their vulnerabilities become more complex, necessitating comprehensive, standardized evaluation platforms. This background underscores the importance of developing systematic red teaming tools that can evaluate models across diverse scenarios, ensuring safer deployment in sensitive domains like healthcare, finance, and legal systems.
Core Problem
Current red teaming efforts face significant challenges: inconsistent risk definitions, limited domain coverage, and lack of a unified benchmarking platform. These issues hinder cross-model comparisons and comprehensive vulnerability understanding. Moreover, existing datasets often focus narrowly on specific risks, leaving gaps in areas like over-refusal behaviors or emerging threats. As newer models like Qwen 2.5 and Llama 3.1 are released, evaluating their robustness becomes more urgent. The core problem is establishing a scalable, standardized, and detailed evaluation framework that captures the multifaceted vulnerabilities of modern LLMs, facilitating both research and industry safety standards.
Innovation
RedBench introduces several innovations: 1) a unified, comprehensive dataset integrating 37 sources, 2) a standardized taxonomy with 22 risk categories and 19 domains, 3) a semi-automated annotation pipeline leveraging Qwen2.5-72B-Instruct and human validation, 4) baseline evaluations of recent models like Llama 3.1, Gemma, and GPT-4, 5) detailed analysis of risk and domain distributions, revealing vulnerabilities and coverage gaps. These innovations address the fragmentation and inconsistency issues of prior datasets, enabling more precise, scalable, and comparable assessments of LLM robustness across diverse scenarios.
Methodology
- �� Data collection: Aggregated 37 datasets from top conferences and repositories, selecting based on relevance, source credibility, and diversity.
- �� Taxonomy design: Defined 22 risk categories (e.g., toxicity, misinformation, bias) and 19 domains (e.g., healthcare, technology, politics) to classify prompts.
- �� Annotation process: Used Qwen2.5-72B-Instruct to automatically assign risk and domain labels, followed by human validation on a subset for accuracy.
- �� Data harmonization: Standardized formats, removed duplicates, balanced samples across categories.
- �� Distribution analysis: Quantified risk and domain coverage, identified imbalances.
- �� Baseline testing: Evaluated models like Llama 3.1, Gemma, GPT-4 on attack and refusal tasks, measuring attack success rate (ASR) and rejection rate (RR).
Experiments
The evaluation involved six models—three open-source (Llama 3.1, Gemma, Ministral) and three closed-source (GPT-4 variants). Attack success was measured via ASR on datasets like HarmBench, using methods such as RainbowPlus and HumanJailbreak. Rejection rates were assessed on datasets like CoCoNot and ORBench, with GPT-4o as a judge. Experiments tested model vulnerabilities across risk categories and domains, providing a comprehensive robustness profile. Hyperparameters were tuned for attack strength and detection thresholds, ensuring fair comparisons. Results highlighted model-specific strengths and weaknesses, guiding future improvements.
Results
RainbowPlus achieved near-perfect attack success (~97.81%) on open-source models, exposing significant vulnerabilities. GPT-4 models showed low attack success (<7%), indicating robustness. Rejection rates varied: Llama 3.1 averaged 28.53%, indicating over-cautiousness, while Gemma-2 was more permissive (~13.46%). Vulnerabilities were concentrated in categories like Extremism and Misinformation, especially in Technology and Healthcare domains. These findings demonstrate the dataset’s effectiveness in revealing model weaknesses and guiding targeted robustness enhancements.
Applications
RedBench can be used by AI developers and safety teams to benchmark and improve model robustness before deployment. It supports content moderation, bias detection, and security assessments across sectors like healthcare, finance, and legal. In the long term, RedBench aims to standardize safety evaluation protocols industry-wide, enabling consistent, transparent assessments that foster trustworthy AI deployment in critical applications.
Limitations & Outlook
Some risk categories and domains are underrepresented, limiting coverage. Annotation relies partly on LLMs, which may introduce biases. Evaluation focuses on select models, requiring broader testing. Dataset imbalance suggests future work should include targeted sample expansion and multi-modal scenarios for comprehensive robustness assessment.
Plain Language Accessible to non-experts
想象你在学校里有个超级智能的老师，他可以帮你解答各种问题，但有时候也会因为太谨慎而拒绝回答一些正常的问题。RedBench就像是这个老师的试题库，里面有很多不同类型的问题，有些是用来测试老师是否会说错话，有些是用来看看老师是否会过度保护自己，不让学生提正常的问题。通过这些题目，老师可以知道自己在哪些方面容易出错，在哪些方面太过谨慎。这样，老师就能不断改进，让自己变得既聪明又可靠。RedBench的目标就是建立这样一个全面、标准的检测体系，让人工智能变得更安全、更可信。
ELI14 Explained like you're 14
想象你有个超级厉害的机器人老师，它能帮你解答各种问题，但有时候它会太紧张，拒绝回答一些正常的问题，或者被坏人骗去说不该说的话。RedBench就像是给这个机器人准备的“考试题库”，里面有很多不同类型的问题，有些是用来测试它是否会说错话，有些是用来看看它是否会过度保护自己。通过这些题目，机器人可以知道自己在哪些方面容易出错，在哪些方面太过谨慎。这样，未来的机器人老师就能变得更聪明、更安全，不会随意拒绝，也不会被坏人骗。这就像是让人工智能变得像一个既聪明又可靠的好老师一样。
Abstract
As large language models (LLMs) become integral to safety-critical applications, ensuring their robustness against adversarial prompts is paramount. However, existing red teaming datasets suffer from inconsistent risk categorizations, limited domain coverage, and outdated evaluations, hindering systematic vulnerability assessments. To address these challenges, we introduce RedBench, a universal dataset aggregating 37 benchmark datasets from leading conferences and repositories, comprising 29,362 samples across attack and refusal prompts. RedBench employs a standardized taxonomy with 22 risk categories and 19 domains, enabling consistent and comprehensive evaluations of LLM vulnerabilities. We provide a detailed analysis of existing datasets, establish baselines for modern LLMs, and open-source the dataset and evaluation code. Our contributions facilitate robust comparisons, foster future research, and promote the development of secure and reliable LLMs for real-world deployment. Code: https://github.com/knoveleng/redeval