Left-Branching Transformers Excel at Right-Branching Languages: Data Shapes Word Order Preferences in Language Models
This study compares 192 artificial and natural languages, revealing that language models develop left-branching bias artificially but favor right-branching SVO in natural languages, driven by data resources.
Key Findings
Methodology
The research trains decoder-only models on 192 artificially generated languages covering all six basic word orders with five binary switches, controlling constituent order variations. It evaluates model preferences using perplexity (PPL) and surprisal across multiple data scales. For natural languages, typological databases (WALS, Grambank, APiCS) map language features, and models are trained on FLORES-200 and PUD datasets. Both monolingual and multilingual models are analyzed to observe how preferences evolve with data volume, employing cross-linguistic universal tests and typological mappings to disentangle architecture and data effects.
Key Results
- In artificial languages, models exhibit a strong left-branching bias, inconsistent with human universal tendencies, with preferences reversing early in training, influenced by syntactic entropy rather than innate bias. The preference magnitude varies across constituent switches, especially favoring prenominal relative clauses, contrary to typological distributions.
- In natural languages, models show no clear bias at small data scales but develop a preference for right-branching SVO structures as data size increases (up to 1000MB). This preference correlates with language resource levels rather than inherent word order, with resource-rich languages like English and Chinese showing a strong SVO bias. Multilingual models reflect similar trends, risking reduction in language diversity.
- Overall, the findings suggest that language model biases are primarily data-driven, with resource availability shaping preferences, which could lead to homogenization of global language typologies, especially favoring dominant SVO languages.
Significance
This work fundamentally shifts the understanding of language model biases from architecture to data influence, highlighting the risk of language homogenization driven by resource imbalance. It underscores the importance of balanced, diverse training data to preserve linguistic diversity and informs future multilingual AI development. The insights bridge linguistic typology and AI, emphasizing data curation's role in fostering fair and diverse language technologies.
Technical Contribution
The study introduces a comprehensive framework combining artificial language generation with typological mapping, enabling precise control and comparison of word order biases. It leverages multi-scale training and surprisal analysis to quantify biases dynamically. The integration of typological universals with neural model behavior provides a novel methodology for bias analysis, advancing understanding of data-driven versus architecture-driven preferences.
Novelty
This is the first systematic comparison of artificial and natural language models across a broad typological spectrum, demonstrating that biases are predominantly data-driven. The approach of controlling for data resource levels and using typological mappings to interpret model preferences is innovative, providing new insights into the origins of linguistic biases in AI.
Limitations
- The study mainly focuses on data size and resource levels, with less emphasis on architectural variations, which may also influence biases. Further research is needed to isolate architecture-specific effects.
- Artificial languages, while controlled, lack semantic, pragmatic, and discourse complexities, potentially limiting the generalizability of findings to real-world language use.
- The early training dynamics and bias reversals require deeper analysis, including intermediate checkpoints, to fully understand the mechanisms behind preference shifts.
Future Work
Future research should explore architecture modifications to mitigate bias, develop balanced multilingual corpora, and incorporate semantic and pragmatic features into artificial language experiments. Extending analysis to more diverse models and real-world low-resource languages will help design fairer, more inclusive language technologies.
AI Executive Summary
The rapid deployment of large language models (LLMs) across diverse languages raises critical questions about their inherent biases and impact on linguistic diversity. Historically, linguistic typology suggests universal tendencies, such as the prevalence of SOV and SVO orders, but whether models reflect these universal patterns remains unclear. This study offers a comprehensive analysis by systematically comparing models trained on 192 artificially generated languages and multiple natural languages from FLORES-200 and PUD datasets.
Artificial languages, generated via probabilistic context-free grammars, reveal that models develop a strong left-branching bias, which conflicts with human language universals. Interestingly, this bias shifts early during training, influenced more by syntactic entropy than innate architecture. Conversely, in natural languages, models initially show no clear bias at small data scales but gradually favor right-branching SVO structures as data volume increases, especially in resource-rich languages like English and Chinese. This bias correlates strongly with resource availability rather than inherent word order preferences.
The findings have profound implications. They suggest that the biases observed in language models are primarily data-driven, driven by resource imbalance, which risks reducing the linguistic diversity of future AI systems. As models favor dominant SVO languages, less-resourced languages and those with flexible word orders may face further marginalization. The research emphasizes the importance of balanced data curation and typological awareness in training multilingual models.
Methodologically, the study combines artificial language generation with typological mapping, using surprisal and perplexity metrics to quantify biases dynamically across training stages. This approach provides a novel framework for disentangling architecture and data effects, offering a pathway to mitigate bias and promote linguistic diversity in AI applications.
Deep Analysis
Background
语言的词序变化是跨语言研究的核心内容之一。Greenberg的普遍性理论指出，SOV和SVO是最常见的基本词序，且与后置或前置介词、修饰语位置密切相关。近年来，深度学习模型在多语种环境中的应用引发关注，研究模型在不同语言中的偏好偏向，尤其是是否会偏向某些特定词序，从而影响语言多样性。已有研究多集中于人工生成的语言或有限自然语料，揭示模型偏好与语法结构的关系，但缺乏系统性比较。此研究结合人工语言和自然语言，利用大规模数据和多模型训练，试图揭示偏差的根源，强调数据资源丰富度对偏序偏好的影响，推动模型公平性和多样性保护。
Core Problem
核心问题在于：大规模语言模型是否存在系统性偏好偏向？偏向的根源是模型架构还是训练数据？偏差是否会导致语言多样性的丧失？这些问题关系到模型的公平性、语言保护和未来多语种应用的可持续性。传统研究多关注模型性能，忽视偏好偏向的深层机制，尤其是偏向资源丰富语言的趋势可能引发语言同质化，影响全球语言多样性。
Innovation
本研究的创新点在于：• 结合192个人工语言和多语自然语言，系统分析模型偏好偏向；• 利用多尺度数据（5MB到1000MB）观察偏好变化；• 引入Typological映射，揭示偏好与语言资源的关系；• 发现偏好偏向与数据资源密切相关，偏左在人工语言中，偏右在自然语言中，强调数据偏差的作用。这为理解模型偏差提供了全新视角。
Methodology
- �� 生成192个人工语言，控制六种基本词序和五个二元切换参数，确保所有变体在概率上等价。• 训练GPT-2风格模型，评估困惑度（PPL）和Surprisal，分析偏好偏向。• 利用FLORES-200和PUD语料库，映射自然语言的词序特征。• 比较单语、多语、多语种模型在不同数据规模下的偏好变化。• 采用Typological数据库，分析偏好与人类语言的对应关系。• 通过对比人工和自然语言，控制变量，揭示偏差的根源。
Experiments
- �� 在人工语言上，训练不同数据规模（5MB、10MB）模型，分析偏好偏向，观察偏好反转。• 在自然语言上，使用Goldfish和多语模型，评估偏好随数据增长的变化。• 采用困惑度、Surprisal和Typological映射指标，量化偏差。• 设计对比实验，验证偏好是否符合跨语言普遍性。• 结合模型训练动态，分析偏好变化的机制。
Results
- �� 人工语言中，模型偏左，偏好在早期反转，偏差由语法结构影响，偏离人类普遍偏好。• 自然语言中，小数据无偏好，大数据偏向SVO，资源越丰富越明显，偏好与资源相关。• 多语模型表现出类似趋势，偏差可能导致语言词序多样性减少，尤其在多词序语言中影响显著。
Applications
- �� 促进多语种模型的公平性设计，避免偏差引发的语言同质化。• 支持语言保护项目，通过调整训练数据，维护语言多样性。• 在多语应用中优化模型偏好，提升少数语言的表现。
Limitations & Outlook
- �� 主要依赖训练数据规模和资源丰富度，未充分探讨模型架构偏向。• 人工语言缺乏语义和语用复杂性，影响偏好分析的普适性。• 偏好反转机制的早期动态未完全解析，未来需结合训练中间状态深入研究。
Plain Language Accessible to non-experts
想象你在一个工厂里，工人们每天按照一定的顺序组装产品。有些工人喜欢先把零件放在左边，再装到产品上，有些则喜欢先装到右边。这个偏好可能是因为工厂的机器布局或流程设计造成的。现在，假设我们用一种智能机器人（模型）来学习这个流程。这个机器人会观察大量的工厂操作数据，逐渐学会哪个顺序更容易、更快完成任务。研究发现，如果工厂用的机器和流程多是资源丰富的，机器人就会偏向某一种顺序（比如先左后右），但如果工厂资源少，偏好就会不同。这说明，机器人偏好的形成，主要是因为它所看到的数据和流程，而不是它的设计本身。这个发现提醒我们，要想让机器人学习多样的流程，就必须提供平衡丰富的训练数据，否则它可能只学到一种偏见，导致未来的操作变得单一化。
ELI14 Explained like you're 14
想象你在学校里学做菜。有的厨师喜欢先把菜切好再炒，有的喜欢边切边炒。你会发现，厨师的习惯其实跟厨房的布局和工具有关。如果厨房里所有的刀子都在左边，厨师自然会先用左边的刀切菜；如果刀子都在右边，厨师就会偏向用右边的刀。现在，假设有个机器人厨师，它通过观察很多厨房的操作，学会了哪种顺序更快、更好。研究发现，如果这个机器人只看到用左边的刀切菜的厨房，它就会偏向左边；如果看到用右边刀的厨房，它就会偏向右边。这说明，机器人的偏好其实是被它看到的厨房环境和工具影响的，而不是它自己天生喜欢哪一种。这个发现告诉我们，要让机器人学会多样的做菜方法，就得给它看各种不同的厨房环境，否则它只会学到一种习惯，未来可能就变得单一了。
Abstract
We systematically compare word order preferences in decoder-only language models across 192 artificial languages and typologically diverse natural languages. On artificial languages, models exhibit a left-branching preference that aligns with neither natural language universals nor human word order learning biases. On natural languages, monolingual models show no clear base word order bias at small scales, but as data grows, a preference for right-branching subject-verb-object (SVO) languages emerges while SOV falls behind despite being the most frequent order cross-linguistically. This SVO advantage extends to multilingual models and correlates with language resource level and data quality rather than word order. Thus, the same architecture exhibits opposite preferences on artificial and natural languages, establishing that word order biases observed in practice are data-driven. Since highly-resourced languages are overwhelmingly SVO, these biases risk gradually reducing word order diversity, particularly in languages that productively use multiple word orders, with the widespread adoption of LLMs.