Should We Still Pretrain Encoders with Masked Language Modeling?
This study compares MLM and CLM pretraining, finding MLM superior in downstream tasks, but CLM more data-efficient and stable.
Key Findings
Methodology
Using large-scale, controlled experiments, 38 models (210M-1B params) were trained under identical data and architecture conditions. The study systematically compared MLM, CLM, and hybrid strategies across multiple NLP tasks, including classification, QA, and IR. Key algorithms include BERT-style bidirectional attention for MLM and autoregressive causal attention for CLM. Experiments involved hyperparameter tuning, masking ratios, and training budgets, with extensive fine-tuning evaluations totaling over 15,000 runs. The methodology emphasizes isolating the effects of training objectives on model performance, stability, and data efficiency, employing rigorous statistical analysis and ablation studies.
Key Results
- MLM models outperform CLM in most text representation tasks, with improvements of 3-5 points in F1 or accuracy, especially on QA and classification datasets, confirming the importance of bidirectional context modeling.
- CLM models show higher data efficiency and training stability, performing better in early training stages and being less sensitive to hyperparameters like learning rate, making them suitable for resource-constrained scenarios.
- A two-stage approach—initial CLM pretraining followed by MLM fine-tuning—achieves the best overall performance within fixed compute budgets, outperforming single-objective training, especially when starting from pretrained CLM models.
Significance
This research clarifies the roles of MLM and CLM in text representation learning, demonstrating that while MLM remains dominant for performance, CLM offers advantages in efficiency and stability. The findings guide the design of more effective pretraining strategies, reducing computational costs and enabling robust models in low-resource or multi-task settings. It advances understanding of how training objectives influence downstream capabilities, contributing to the development of more versatile NLP models.
Technical Contribution
The paper introduces a systematic framework for comparing MLM and CLM under controlled conditions, revealing their complementary strengths. It proposes a hybrid two-stage training protocol that leverages CLM’s data efficiency and MLM’s representation quality, validated through extensive experiments. The work also provides insights into training stability and hyperparameter robustness, offering a new paradigm for multi-objective pretraining. The approach bridges the gap between autoregressive and bidirectional models, opening avenues for future research in unified training objectives.
Novelty
This is the first comprehensive, controlled comparison of MLM and CLM at the same scale and data conditions, demonstrating the benefits of a sequential two-stage training strategy. Unlike prior work focusing on either architecture or large-scale data, this study systematically evaluates their interplay, establishing a new standard for hybrid pretraining. It also highlights the practical advantages of starting from pretrained CLM models for efficient encoder development.
Limitations
- The experiments are primarily conducted on English datasets and specific model architectures, limiting immediate generalization to other languages or architectures.
- High computational costs restrict exploration of larger models or more diverse tasks, potentially impacting scalability.
- Further investigation is needed on transferability to multi-task and low-resource settings, as well as on multi-lingual models, to fully realize the approach's potential.
Future Work
Future research will focus on automating multi-objective pretraining, integrating multi-task learning, and exploring adaptive strategies for balancing objectives dynamically. Extending experiments to multilingual datasets and low-resource languages will be prioritized. Additionally, developing lightweight training techniques and reducing computational costs remain key directions to facilitate broader adoption.
AI Executive Summary
In natural language processing, the quest for high-quality text representations has traditionally centered on Masked Language Modeling (MLM), exemplified by models like BERT. MLM’s bidirectional attention mechanism enables deep contextual understanding, leading to state-of-the-art performance across various tasks such as classification, question answering, and named entity recognition. However, recent developments have shown that decoder-only models pretrained with Causal Language Modeling (CLM), like GPT, can be repurposed as encoders and sometimes outperform traditional MLM models, especially when scaled up significantly.
Despite these promising results, the fundamental reasons behind CLM’s success remain unclear. Is it an inherent advantage of the causal objective, or merely a consequence of larger models and more data? To address this, the study conducted extensive, controlled experiments comparing MLM and CLM under identical conditions, training 38 models ranging from 210 million to 1 billion parameters. The experiments involved rigorous evaluation across multiple NLP benchmarks, including sequence classification, token classification, question answering, and information retrieval.
The findings reveal that MLM generally yields better downstream performance, especially on tasks requiring rich bidirectional context. Nonetheless, CLM models demonstrate superior data efficiency and training stability, particularly in early training stages, and are less sensitive to hyperparameters like learning rate. Notably, a hybrid two-stage training strategy—initial CLM pretraining followed by MLM fine-tuning—achieves optimal results within fixed computational budgets. This approach leverages the strengths of both objectives, reducing training costs and improving robustness.
The implications are significant: for resource-constrained environments, starting with CLM and then fine-tuning with MLM offers a practical and effective pathway to high-quality encoders. The research clarifies the roles of different pretraining objectives, guiding future model design and training strategies. All models and code are publicly available, fostering further innovation in efficient NLP representation learning. Overall, this work advances understanding of pretraining dynamics, balancing performance, efficiency, and stability in the development of next-generation language models.
Deep Analysis
Background
近年来，预训练模型在自然语言处理中的地位不断提升。BERT引领的MLM策略通过双向注意力机制实现了对上下文的深度理解，推动了文本分类、问答等任务的性能提升。与此同时，解码器架构如GPT在生成任务中表现出色，但在表示学习方面相对不足。近期，研究发现将解码器模型迁移为编码器（如通过微调或结构调整）也能获得优异表现，尤其是在大规模数据和模型条件下。不同预训练目标（MLM和CLM）在模型性能、数据效率和训练稳定性方面表现出差异，但缺乏系统性比较。本研究旨在填补这一空白，通过控制变量，深入分析两者的优劣，为预训练目标设计提供理论依据。
Core Problem
核心问题在于，MLM作为双向预训练目标在文本表示中表现优异，但训练成本较高；而CLM在数据效率和训练稳定性方面表现优越，但在某些任务上性能略逊。现有研究多集中在模型规模扩大或目标微调，缺乏系统性比较，难以明确两者的本质差异。此外，如何结合两者优势，设计高效的预训练策略，也是亟待解决的问题。尤其是在资源有限或多任务环境中，找到既能保证性能，又能降低成本的方法尤为重要。
Innovation
本研究创新点主要包括：1）系统性比较MLM与CLM在相同模型和数据条件下的表现差异，揭示二者的本质区别；2）提出了结合两者的两阶段预训练策略，先CLM后MLM，有效兼顾数据效率和表示质量；3）验证该策略在不同模型规模和任务上的优越性，特别是在资源有限的情况下表现出显著优势。此策略突破了传统单一目标预训练的局限，为多目标、多阶段预训练提供了新思路。
Methodology
- �� 采用EuroBERT架构，模型尺寸涵盖210M、610M、1B参数。
- �� 使用FineWeb-Edu数据集，确保训练样本一致性。
- �� 实现三种预训练目标：MLM（随机遮蔽，比例20%-50%）、CLM（自回归预测）及其结合（CLM后MLM）
- �� 设计控制实验，比较不同目标在相同数据、模型规模下的性能差异。
- �� 采用WSD学习率调度，训练42,000步，确保充分学习。
- �� 进行多任务微调，评估分类、问答、信息检索任务，使用不同指标（准确率、F1、NDCG）衡量。
- �� 大规模实验验证不同训练策略（从零开始、持续预训练、两阶段）对性能的影响。
Experiments
- �� 实验数据来自多任务数据集，包括SST-2、MNLI、QQP、SQuAD、MS MARCO等。
- �� 比较MLM、CLM及其结合策略在不同模型参数（210M-1B）上的表现。
- �� 采用多次随机初始化和超参数调优，确保结果稳健。
- �� 重点分析训练稳定性、数据效率和微调性能差异。
- �� 通过控制遮蔽比例、训练步数，评估模型在不同资源预算下的表现。
Results
- �� MLM模型在所有任务中表现优异，问答和分类任务性能提升达3-5点F1或准确率，验证其在双向上下文建模中的优势。
- �� CLM模型在早期训练中表现更好，数据效率高，训练稳定性强，尤其在低资源环境中更具优势。
- �� 两阶段训练（CLM后MLM）在固定计算预算下实现最优性能，验证了目标互补的有效性，特别是在从预训练的CLM模型继续微调时效果明显优于纯MLM。
Applications
- �� 该策略适用于需要高效文本表示的搜索引擎、问答系统和多任务学习场景。
- �� 资源有限的研究或工业环境中，可利用预训练的CLM模型作为起点，减少训练成本。
- �� 长远来看，有望推动多目标预训练模型的标准化，降低大规模模型训练门槛，促进多语种、多任务的应用。
Limitations & Outlook
- �� 实验主要集中在英语和特定架构，泛化到多语言、多模型类型仍需验证。
- �� 大规模训练成本高，限制了更大模型和更复杂任务的探索。
- �� 对迁移学习和少样本微调的适应性尚未深入，未来需结合多任务学习优化预训练目标。
Plain Language Accessible to non-experts
想象你在一家工厂里，工人们需要学习如何组装不同的产品。有些工人只看说明书（MLM），他们会反复阅读每个部分，理解每个零件的关系，这样可以组装得更快、更准。而另一些工人只看生产线（CLM），他们按照顺序逐步操作，学会了流程，但可能对整体理解不够深。现在，最聪明的做法是先让工人按照流程熟悉整个生产线（CLM），然后再让他们反复看说明书（MLM），这样既能快速掌握流程，又能理解每个细节。研究发现，这样的两步走方法，不仅节省时间，还能让工人更稳当，不容易出错。这个比喻说明，结合不同学习方式，可以让机器学习模型变得更快、更聪明，也更可靠。
ELI14 Explained like you're 14
想象你在学校学新技能，比如弹钢琴。有时候，你会先学弹奏的顺序（像CLM，按顺序练习），这样你能快速掌握流程，但可能不太懂每个音符的细节。然后，你会反复练习每个音符（像MLM，遮蔽部分练习），这样可以理解每个音符的细微差别。研究发现，先用CLM让你更快入门，节省时间；之后再反复练习细节（MLM），可以让你弹得更好、更稳。这两步结合，比只做其中一件事效果都更好。对机器学习模型来说，也是一样的道理。先用CLM让模型理解整体结构，再用MLM让模型掌握细节，这样可以更快、更好地学习语言。
Abstract
Learning high-quality text representations is fundamental to a wide range of NLP tasks. While encoder pretraining has traditionally relied on Masked Language Modeling (MLM), recent evidence suggests that decoder models pretrained with Causal Language Modeling (CLM) can be effectively repurposed as encoders, often surpassing traditional encoders on text representation benchmarks. However, it remains unclear whether these gains reflect an inherent advantage of the CLM objective or arise from confounding factors such as model and data scale. In this paper, we address this question through a series of large-scale, carefully controlled pretraining ablations, training a total of 38 models ranging from 210 million to 1 billion parameters, and conducting over 15,000 fine-tuning and evaluation runs. We find that while training with MLM generally yields better performance across text representation tasks, CLM-trained models are more data-efficient and demonstrate improved fine-tuning stability. Building on these findings, we experimentally show that a biphasic training strategy that sequentially applies CLM and then MLM, achieves optimal performance under a fixed computational training budget. Moreover, we demonstrate that this strategy becomes more appealing when initializing from readily available pretrained CLM models, reducing the computational burden needed to train best-in-class encoder models. We release all project artifacts at https://hf.co/MLMvsCLM to foster further research.