COLD: A Benchmark for Chinese Offensive Language Detection

The rapid development of deep learning and pre-trained language models (e.g., BERT, GPT) has significantly advanced NLP tasks, including hate speech and toxicity detection. In English, large-scale datasets like HateXplain and Jigsaw Toxicity have fostered robust models. However, Chinese NLP research on offensive content remains underdeveloped due to scarce resources and cultural-linguistic differences. Existing Chinese datasets are limited, often focusing on profanity or gender bias, lacking comprehensive coverage of racial and regional biases. The rise of generative models (e.g., CDialGPT, CPM, EVA) has introduced new safety concerns, as these models can produce biased or offensive outputs, especially when prompted with sensitive keywords. Addressing these issues requires high-quality datasets and effective detection methods tailored to Chinese language characteristics.

The core challenge lies in developing reliable, scalable tools for Chinese offensive language detection amid data scarcity and linguistic complexity. Existing methods rely on keyword filtering or rule-based systems, which are insufficient for nuanced bias detection. The subtlety and diversity of bias expressions, coupled with the potential for generative models to produce offensive content, demand sophisticated, data-driven approaches. Furthermore, the lack of standardized benchmarks hampers progress, making it difficult to evaluate and compare detection techniques. These issues threaten the deployment of safe AI systems in Chinese social platforms, necessitating a dedicated dataset and robust models to mitigate risks effectively.

This work introduces several key innovations: 1) COLDATASET, the first large-scale, multi-dimensional Chinese bias and offensive dataset, covering race, gender, and regional biases; 2) COLDETECTOR, a BERT-based classifier trained with semi-automatic annotation, achieving 81% accuracy; 3) systematic evaluation of popular Chinese generative models’ bias and offensive outputs under various prompts, revealing their risk profiles. The dataset’s construction employs keyword queries, sub-topic crawling, and model-guided annotation, ensuring diversity and high quality. The detection framework combines keyword filtering, model fine-tuning, and human verification, setting a new standard for Chinese offensive content detection.

• �� Data collection: Utilize keyword queries and sub-topic crawling on platforms like Zhihu and Weibo to gather biased and offensive comments. • Annotation: Implement a model-in-the-loop approach—initial manual labeling of 500 samples, training a classifier, then semi-automatically labeling remaining data, with iterative validation. • Data categorization: Divide samples into attack (individual/group), anti-bias, and other non-offensive categories, ensuring comprehensive coverage. • Model training: Fine-tune bert-base-chinese with cross-entropy loss, optimizing detection accuracy. • Evaluation: Measure performance using accuracy, precision, recall, F1-score across categories. • Generative model assessment: Input diverse prompts, analyze bias and offensive outputs, identify triggers.

The experiments involve training COLDETECTOR on the COLDATASET, comparing it with baseline tools like Baidu Text Censor and TransJigsaw. Hyperparameters include a learning rate of 2e-5, batch size 32, and 3 training epochs. The evaluation covers overall accuracy, sub-category detection, and cross-model generalization. Additional tests assess generative models’ bias under different prompts, analyzing their propensity to produce offensive content. The annotation process involves multiple annotators with high inter-annotator agreement (Fleiss’ κ=0.819). The robustness of the detection system is validated through ablation studies and cross-scenario tests.

COLDETECTOR achieves 81% overall accuracy, outperforming Baidu Text Censor (63%) and TransJigsaw (60%). It detects attack categories with nearly 80-85% accuracy, but anti-bias detection remains weak at 38%. Generative models like CDialGPT and CPM show increased bias and offensive outputs when prompted with keywords or negative sentiments, indicating safety risks. The results highlight that bias and offensive content can be triggered by subtle cues, emphasizing the need for robust detection and mitigation strategies. The dataset and models provide a solid foundation for future research and practical deployment.

The developed detection system can be integrated into social media moderation tools, AI content filters, and online community management platforms to automatically identify biased or offensive content. It supports real-time filtering, reducing harmful interactions and promoting healthier online environments. The dataset and methodology also serve as benchmarks for further research, enabling continuous improvement in bias mitigation. Long-term, these tools can be extended to multi-modal content and multilingual settings, fostering safer AI deployment globally.

The dataset relies on keyword-based collection, which may miss subtle or implicit biases, limiting coverage. The detection model struggles with nuanced anti-bias expressions, leading to false negatives. Semi-automatic annotation introduces potential noise, requiring further manual validation. The evaluation focuses on text-based content, not considering multimodal or contextual factors. Future work should address these limitations by expanding data sources, improving annotation quality, and exploring explainability techniques.