On the Limitations of Cross-Lingual Consistency in Multilingual Text-to-image Generation
Introducing LingT2I benchmark, analyzing cross-lingual performance gaps and cultural influences in multilingual T2I models.
Key Findings
Methodology
Using the LingT2I dataset with 33K prompts across 10 languages, the study applies multi-dimensional metrics like CLIPScore and TRIGScore to evaluate content and text rendering. Comparing models such as SD3.5, Qwen-Image, and PixArt, it quantifies performance disparities, especially in low-resource languages and non-Latin scripts. Data quality is ensured through automated translation, manual validation, and multi-stage filtering. The analysis links linguistic resource distribution, script complexity, and cultural context to performance gaps, providing a comprehensive understanding of cross-lingual biases.
Key Results
- Models perform best on high-resource languages (average CLIPScore 0.78), but performance drops sharply on low-resource languages like Hindi and Korean (average 0.38), indicating over 40% disparity. Non-Latin scripts exhibit poor text rendering accuracy (<0.3), with frequent character errors. Language-specific biases emerge: Korean results skew towards oil painting styles, while English remains realistic, reflecting cultural influences on generation patterns.
- Performance variance across dimensions (realism, faithfulness, style) reveals systematic biases, with certain languages favoring particular styles or content fidelity. These biases highlight the influence of linguistic and cultural factors on model outputs, emphasizing the need for more inclusive training and evaluation strategies.
- The study demonstrates that current models lack equitable multilingual capabilities, especially for underrepresented languages and scripts, which limits their global applicability. Addressing these gaps is crucial for developing fair and culturally aware AI systems.
Significance
This research exposes fundamental limitations in current multilingual text-to-image generation, particularly regarding linguistic inequality and cultural bias. It provides a structured benchmark for evaluating and diagnosing performance gaps, guiding future development towards more inclusive and robust models. The findings have broad implications for AI fairness, cross-cultural communication, and global content creation, encouraging the community to prioritize linguistic diversity and cultural sensitivity in model design. By systematically analyzing performance disparities, the study offers actionable insights for academia and industry, fostering AI systems that serve diverse populations more equitably.
Technical Contribution
The paper introduces LingT2I, a comprehensive benchmark with multi-dimensional evaluation metrics, addressing content quality, style, faithfulness, and robustness across 10 languages. It innovatively combines automated translation, multi-modal assessment, and cultural analysis to quantify performance gaps. The methodology integrates advanced metrics like CLIPScore and TRIGScore, along with a detailed bias analysis linking linguistic resource distribution, script complexity, and cultural factors. This approach enables precise diagnosis of performance bottlenecks and bias sources, setting a new standard for multilingual T2I evaluation. The study also highlights the importance of culturally aware training strategies and resource augmentation for improving fairness.
Novelty
This is the first systematic, large-scale evaluation of multilingual T2I models across diverse languages and scripts, emphasizing cultural and linguistic influences. Unlike prior work limited to English or isolated phenomena, it provides a holistic framework combining quantitative metrics with cultural analysis. The introduction of a multi-language, multi-dimensional benchmark fills a critical gap, enabling fairer comparisons and targeted improvements. Its emphasis on cultural context and script complexity as performance determinants marks a significant step forward in developing truly inclusive generative models.
Limitations
- The dataset, while diverse, still favors high-resource languages, leaving low-resource languages underrepresented and less accurately evaluated.
- Handling of complex scripts remains imperfect, with frequent rendering errors for non-Latin characters, limiting practical deployment.
- Bias analysis relies on static metrics, lacking real-world user feedback and dynamic cultural context considerations, which are essential for holistic fairness assessment.
Future Work
Future efforts will expand low-resource language coverage, improve character system processing, and incorporate cultural context modeling. Developing adaptive training strategies, multilingual data augmentation, and user-centric evaluation methods will be prioritized. Additionally, integrating dynamic feedback loops and user studies can enhance model fairness and cultural sensitivity, ultimately leading to more equitable AI systems capable of serving a truly global audience.
AI Executive Summary
Multilingual text-to-image (T2I) generation has seen rapid advancements, yet most models excel only in English, revealing significant performance gaps across languages. This disparity stems from resource imbalance, script complexity, and cultural differences, which current models struggle to handle effectively. To address this, the study introduces LingT2I, a benchmark comprising 33,000 prompts across 10 diverse languages, designed to evaluate both content generation and text rendering tasks systematically.
The evaluation of state-of-the-art models such as SD3.5, Qwen-Image, and PixArt reveals stark disparities: high-resource languages like English and Chinese achieve average CLIPScores of 0.78, whereas low-resource languages like Hindi and Korean drop below 0.38. Non-Latin scripts, including Arabic and Han characters, exhibit severe rendering errors, with accuracy below 0.3. Furthermore, the analysis uncovers language-dependent biases, with models favoring certain styles or content fidelity based on linguistic and cultural cues. For example, Korean prompts tend to produce oil painting styles, while English prompts favor realism.
These findings expose critical limitations in current multilingual models, emphasizing the need for more inclusive training data, character system adaptations, and cultural awareness. The research advocates for developing models that can fairly represent all languages and cultures, fostering AI systems capable of global, equitable content creation. Future directions include expanding low-resource language datasets, refining script handling, and integrating cultural context modeling to build truly universal generative AI. Ultimately, this work aims to bridge performance gaps, promote cultural inclusivity, and advance AI's role in global communication and creativity.
Deep Dive
Abstract
Text-to-image (T2I) generation has achieved remarkable progress in recent years. However, existing research has largely focused on English-only settings, leaving cross-lingual performance gaps and language-specific effects insufficiently explored. To fill this gap, we introduce LingT2I, a benchmark covering 10 widely used languages with 33K prompts, designed to evaluate cross-lingual effects in both content generation and text rendering. Building on this benchmark, we conduct a comprehensive cross-lingual analysis, uncovering linguistic inequality and language-dependent trade-offs across evaluation dimensions. Beyond quantitative evaluation, we further reveal a range of language-dependent generation patterns, highlighting how linguistic factors and their corresponding cultural contexts systematically impact model outputs. Our benchmark and analysis provide a foundation for studying cross-lingual behavior in T2I generation and facilitate the development of more robust and inclusive models. Code and dataset are available at https://github.com/RISys-Lab/LingT2I.