Toxicity in Multilingual Machine Translation at Scale

The evolution of machine translation has transitioned from rule-based systems to neural models, notably Transformer architectures (Vaswani et al., 2017), which enabled large-scale multilingual models like mBERT and XLM-R (Conneau et al., 2020). Projects such as NLLB (No Language Left Behind, 2022) have expanded translation capabilities to over 600 languages, emphasizing inclusivity. Despite these advances, issues of bias, toxicity, and fairness have surfaced, especially as models generate harmful content unintentionally. Prior works (Sheng et al., 2021; Webster et al., 2020) have documented biases but lacked systematic mechanisms to analyze and interpret toxicity sources across languages. HOLISTICBIAS (Smith et al., 2022) provides a comprehensive benchmark for demographic bias evaluation, but understanding how toxicity arises during translation—particularly via hallucination and mistranslation—remains an open challenge. Addressing this gap is critical for deploying responsible multilingual AI systems.

Multilingual MT systems risk introducing toxicity, especially in low-resource languages and sensitive demographic axes. Existing detection methods are limited by reliance on static word lists, which cannot capture context-dependent or evolving expressions. Hallucination and mistranslation are identified as key mechanisms leading to toxicity, but their roles are not well-quantified or understood. This impairs the ability to develop targeted mitigation strategies. The core problem is how to accurately detect, interpret, and reduce added toxicity, considering the complex interplay of language resource levels, model biases, and translation robustness. Solving this is vital for ensuring safe, fair, and inclusive AI-powered communication tools.

This work introduces a multi-faceted framework combining HOLISTICBIAS, toxicity wordlist detection, and ALTI+ attribution to analyze toxicity mechanisms systematically. It innovates by quantifying source contribution to toxicity, revealing hallucination and mistranslation as primary causes. The use of Gini impurity to measure translation robustness offers a novel interpretability angle, enabling fine-grained risk assessment. Unlike prior approaches limited to surface-level detection, this methodology provides explainability and actionable insights, especially across diverse languages and demographic axes. The integration of these techniques represents a significant step forward in bias analysis and mitigation in multilingual NLP.

• �� Translate HOLISTICBIAS sentences into 164 languages using NLLB 3.3B model, covering demographic axes like gender, race, and ability. • Detect toxicity via toxicity wordlists, created through human translation, to identify potentially harmful terms. • Validate automatic detection with human annotators, assessing false positive and negative rates. • Apply ALTI+ to compute input attribution scores, measuring source contribution to each translated word, especially toxic ones. • Calculate Gini impurity across aligned descriptor words to evaluate translation robustness. • Analyze the correlation between source contribution, robustness, and toxicity, identifying high-risk translation patterns. • Use thresholds (e.g., 40% source contribution) to flag potentially hallucinated or mistranslated toxic outputs. • Perform statistical analysis across languages and axes, identifying patterns and high-risk scenarios. • Incorporate human evaluation to verify false positives and negatives, refining detection thresholds and interpretability tools.

The experiments involve translating a large-scale dataset (HOLISTICBIAS, 472k sentences) into 164 languages using NLLB 3.3B. Toxicity detection relies on curated wordlists, validated by native speakers, to identify toxic content. The evaluation measures include toxicity rate statistics, source contribution analysis via ALTI+, and robustness assessment through Gini impurity. The study conducts ablation experiments to verify the impact of source contribution thresholds and robustness metrics on toxicity detection accuracy. Human evaluators assess false positives and negatives, providing ground truth for automatic methods. The experiments also compare toxicity across demographic axes and language resource levels, revealing key patterns and risk factors. The setup emphasizes scalability, interpretability, and cross-lingual generalization, aiming to produce actionable insights for bias mitigation.

Toxicity rates across languages range from 0% to 5%, with low-resource languages like Yoruba and Tok Pisin exhibiting higher rates (>2%). 84% of toxic outputs show significant correlation with low source contribution (<40%), indicating hallucination or mistranslation as primary causes. The combined analysis of source contribution and Gini impurity identifies translation instability as a key factor in toxicity risk. Human evaluation confirms detection accuracy, with false positive rates below 1%, though some languages like Chinese and Persian show higher false positives. The results underscore the importance of data quality, resource level, and demographic sensitivity in toxicity mitigation. These findings provide a quantitative basis for developing safer translation models and bias-aware evaluation protocols.

• �� Immediate: Implement toxicity risk flags in multilingual translation platforms, enabling real-time content moderation and reducing harmful outputs. • Long-term: Develop self-correcting translation systems with integrated bias detection and explainability modules, fostering responsible AI deployment across diverse cultural contexts. These applications can improve user trust, compliance with ethical standards, and inclusivity in global communication tools.

Current detection relies heavily on predefined toxicity wordlists, which may miss context-dependent or emerging toxic expressions, especially in low-resource languages. Attribution metrics like source contribution and Gini impurity, while effective, may not fully capture complex bias pathways in adversarial or ambiguous cases. The datasets used (HOLISTICBIAS, FLORES) do not encompass all real-world scenarios, limiting generalization. Further research is needed to incorporate contextual, multimodal, and adaptive detection methods to address these gaps.