Ready to Translate, Not to Represent? Bias and Performance Gaps in Multilingual LLMs Across Language Families and Domains
Proposes Translation Tangles framework, integrating multi-metric, multi-domain, bias detection for 24 language pairs, analyzing translation quality and biases.
Key Findings
Methodology
The study employs a comprehensive multilingual benchmark assessing 24 language pairs with metrics like BLEU, BERTScore, and COMET. It introduces a hybrid bias detection pipeline combining rule-based heuristics, semantic similarity, and LLM validation, using datasets such as WMT, Lit-Corpus, and ELRC-Medical. The bias detection integrates keyword matching, named entity recognition (NER), and semantic analysis, with a threshold set at 0.75 for similarity scores. Human annotations serve as gold standard for evaluating automatic bias detection. Models evaluated include Gemma-7B, Llama-3 series, and Mixtral-8x7B, across domains and resource levels, providing a detailed analysis of translation performance and bias prevalence.
Key Results
- Larger models like Llama-3.2-90B achieve BLEU scores up to 44.2 in intra-family translation, with performance gaps narrowing in cross-family translation as model size increases. Bias analysis shows cultural and social biases dominate, with over 75% of detected biases. Bias detection accuracy against LLM verification is around 50%, with significant computational efficiency advantages. Performance varies across domains, with legal texts scoring highest and literary texts lowest, highlighting domain-specific challenges.
- Bias frequency varies notably across models and language pairs, with under-resourced languages like Gujarati and Kazakh exhibiting higher bias counts. The automatic bias detection pipeline achieves nearly 50% agreement with LLM-based validation, but with lower computational cost, making it suitable for large-scale screening. The results underscore the importance of model scaling and resource availability in reducing biases and improving translation quality across diverse languages and domains.
- The experiments demonstrate that while increasing model size improves general translation performance, biases persist, especially in low-resource and culturally sensitive contexts. The combined evaluation framework effectively identifies biases and guides model refinement, emphasizing the need for multi-faceted fairness assessments in multilingual NLP. The study provides a foundation for future research on bias mitigation and fair AI deployment.
Significance
This research addresses a critical gap in multilingual NLP by providing a systematic, scalable framework for evaluating translation quality and fairness across diverse languages and domains. It advances the understanding of how model size, resource availability, and cultural factors influence biases, offering practical tools for bias detection and mitigation. The comprehensive evaluation approach supports the development of more equitable AI systems, especially for low-resource and marginalized languages, fostering inclusive global communication. Its multi-metric, multi-domain design sets a new standard for fairness assessment in multilingual models, with broad implications for academia and industry.
Technical Contribution
The study introduces the Translation Tangles framework, integrating rule-based, semantic, and LLM validation methods into a unified bias detection pipeline. It innovates by combining multiple metrics and domain-specific evaluations, providing a detailed, interpretable assessment of translation fairness. The use of a high-quality, bias-annotated dataset and the hybrid detection approach enhances detection accuracy and efficiency. This work extends current benchmarks by incorporating bias analysis into translation quality evaluation, offering theoretical insights into bias propagation and practical tools for bias mitigation in multilingual LLMs.
Novelty
This is the first comprehensive system integrating multi-metric, multi-domain, and bias detection mechanisms into a unified evaluation framework for multilingual translation. It uniquely combines rule-based heuristics, semantic similarity, and LLM-based validation, providing a scalable, interpretable solution for bias detection. Unlike prior work focusing solely on accuracy metrics, this approach emphasizes fairness and bias analysis, setting a new standard for holistic evaluation of multilingual models. Its dataset and pipeline enable systematic bias quantification across languages and domains, representing a significant methodological advance.
Limitations
- Bias detection relies on predefined keywords and entity categories, potentially missing implicit or culturally specific biases, and threshold tuning remains somewhat subjective.
- Evaluation datasets are limited in scope, primarily covering formal domains, which may not reflect real-world linguistic diversity and bias scenarios.
- Computational costs for large models and LLM validation remain high, limiting scalability in resource-constrained settings. The approach may also underperform in detecting subtle biases or biases rooted in context-specific cultural norms.
Future Work
Future research will explore adaptive thresholding and multi-task learning to enhance bias detection robustness. Integrating multimodal data (images, audio) could improve bias identification in complex scenarios. Developing unsupervised or semi-supervised bias detection methods may reduce reliance on predefined lexicons. Extending the framework to real-world applications like content moderation and cross-cultural communication will further promote fairness. Additionally, efforts will focus on bias mitigation techniques to actively reduce biases during model training and deployment.
AI Executive Summary
As multilingual large language models (LLMs) become central to global communication, their translation capabilities are increasingly scrutinized for fairness and bias. Traditional evaluation metrics like BLEU, while useful, fail to capture the nuanced biases embedded in model outputs, especially across diverse languages and cultural contexts. This gap hampers efforts to develop equitable AI systems capable of serving marginalized and low-resource languages.
In response, this study introduces the Translation Tangles framework, a comprehensive evaluation system that combines multiple metrics, domain-specific assessments, and a hybrid bias detection pipeline. The framework evaluates 24 language pairs across domains such as legal, literary, and medical texts, providing a detailed analysis of translation quality and bias prevalence. It employs rule-based heuristics, semantic similarity measures, and entity recognition to detect biases, with validation via LLM-based verification. Human annotations serve as a gold standard, ensuring the reliability of automatic detection.
Experimental results reveal that larger models like Llama-3.2-90B significantly improve intra-family translation scores, reaching BLEU scores of 44.2, while cross-family performance still lags but shows improvement with scale. Bias analysis indicates cultural and social biases dominate, especially in low-resource languages, with bias detection accuracy around 50%. The framework’s efficiency and interpretability make it a practical tool for large-scale bias screening, guiding model refinement.
This work offers a vital step toward fairer multilingual AI, providing scalable, multi-dimensional evaluation tools that address both quality and bias. Its implications extend to deploying more equitable translation systems, reducing biases in AI-generated content, and fostering inclusive global communication. Limitations include reliance on predefined bias categories and high computational costs, but ongoing research aims to address these challenges. Overall, the framework sets a new standard for fairness assessment in multilingual NLP, with promising avenues for future development in bias mitigation and multimodal evaluation.
Deep Dive
Abstract
The rise of Large Language Models (LLMs) has redefined Machine Translation (MT), enabling context-aware and fluent translations across hundreds of languages and textual domains. Despite their remarkable capabilities, LLMs often exhibit uneven performance across language families and specialized domains. Moreover, recent evidence reveals that these models can encode and amplify different biases present in their training data, posing serious concerns for fairness, especially in low-resource languages. To address these gaps, we introduce Translation Tangles, a unified framework and dataset for evaluating the translation quality and fairness of open-source LLMs. Our approach benchmarks 24 bidirectional language pairs across multiple domains using different metrics. We further propose a hybrid bias detection pipeline that integrates rule-based heuristics, semantic similarity filtering, and LLM-based validation. We also introduce a high-quality, bias-annotated dataset based on human evaluations of 1,439 translation-reference pairs. The code and dataset are accessible on GitHub: https://github.com/faiyazabdullah/TranslationTangles