GLM: General Language Model Pretraining with Autoregressive Blank Infilling
GLM introduces autoregressive blank infilling with 2D positional encoding, outperforming BERT, T5, and GPT on multiple NLP tasks.
Key Findings
Methodology
GLM employs an innovative autoregressive blank infilling objective, integrating 2D positional encodings and span shuffling to unify bidirectional encoding and unidirectional generation. The model is built on a Transformer architecture, with modifications such as rearranged layer normalization and GeLU activations. It supports multi-task pretraining, combining NLU, conditional, and unconditional generation tasks. During training, spans are sampled from a Poisson distribution, masked, and randomly permuted, enabling the model to learn flexible span prediction in arbitrary order. The 2D positional encoding distinguishes intra- and inter-span positions, enhancing long-text understanding. The training data includes BookCorpus and Wikipedia, with parameter sizes ranging from 110M to 515M, trained over 250,000 steps to ensure comprehensive semantic and structural learning.
Key Results
- On SuperGLUE, GLM Large surpasses BERT and T5, with an average improvement of 4.6% (BERT base) to 5.0% (BERT large), demonstrating superior performance across multiple NLP tasks;
- In natural language understanding, conditional and unconditional generation, GLM outperforms comparable SOTA models, especially after multi-task pretraining;
- With parameters at 1.25× BERT Large and comparable training data, GLM exhibits strong generalization across diverse NLP scenarios.
Significance
This work advances the field by unifying bidirectional and autoregressive modeling within a single framework, overcoming the limitations of previous models that excel only in specific tasks. The innovative use of autoregressive blank infilling with span shuffling and 2D positional encoding enables the model to excel in both understanding and generation, reducing the need for task-specific architectures. Its ability to handle multiple NLP tasks with a single pretrained model paves the way for more flexible AI systems, impacting applications like question answering, summarization, and dialogue generation. The approach also offers new insights into model design, emphasizing the importance of flexible span prediction and multi-task training for building truly general-purpose language models.
Technical Contribution
The paper introduces a novel autoregressive blank infilling objective that supports arbitrary span prediction order, facilitated by 2D positional encoding. It modifies the Transformer architecture to better suit multi-task learning, including rearranged layer normalization and GeLU activations. The multi-task training strategy combines NLU, sequence-to-sequence, and language modeling objectives, enabling the model to perform well across diverse tasks. Empirically, the model demonstrates significant improvements over BERT, T5, and RoBERTa on benchmarks like SuperGLUE, CNN/DailyMail, and XSum, validating its theoretical and engineering innovations.
Novelty
This study's key novelty lies in the autoregressive span prediction with span shuffling and 2D positional encoding, allowing the model to predict spans in any order, unlike previous models such as T5 or XLNet. It effectively unifies bidirectional and unidirectional pretraining within a single architecture, supporting multi-task learning seamlessly. This approach is the first to demonstrate that a single model can excel simultaneously in natural language understanding and multiple generation tasks, marking a significant step forward in model versatility.
Limitations
- Despite its flexibility, the model's performance on extremely long texts or highly complex dependencies remains limited, partly due to positional encoding constraints;
- Training large models requires substantial computational resources, making deployment challenging for some applications;
- Balancing multiple tasks during training is non-trivial, and suboptimal weighting can hinder performance on specific tasks.
Future Work
Future research will focus on enhancing long-text modeling capabilities, optimizing training efficiency, and exploring multi-modal extensions. Incorporating dynamic span sampling and adaptive weighting strategies could further improve task-specific performance. Additionally, integrating multimodal data (images, audio) will broaden the applicability of GLM in real-world AI systems.
AI Executive Summary
Pretrained language models have revolutionized NLP, yet their specialization often limits their versatility across tasks. BERT excels in understanding, GPT in generation, and T5 attempts to unify both but at the cost of complexity and size. Recognizing these limitations, this work introduces GLM, a novel model that bridges the gap through autoregressive blank infilling combined with innovative training techniques.
GLM’s core innovation lies in its span shuffling and 2D positional encoding strategies, which enable the model to predict text spans in any order, effectively blending bidirectional and unidirectional capabilities within a single Transformer architecture. During pretraining, the model is exposed to multiple tasks—masked span filling, long text generation, and sentence completion—using a multi-task learning framework. The training data includes large-scale corpora such as BookCorpus and Wikipedia, with model sizes ranging from 110M to 515M parameters. The training process involves extensive steps (up to 250,000) to ensure the model captures rich semantic and structural information.
Experimental results demonstrate that GLM significantly outperforms existing models like BERT, T5, and RoBERTa on benchmarks such as SuperGLUE, CNN/DailyMail, and XSum. For instance, on SuperGLUE, GLM Large achieves an average improvement of 4.6% over BERT, with performance gains across multiple tasks. The model’s ability to handle both understanding and generation tasks with a unified architecture marks a substantial step toward truly general-purpose NLP models.
This research offers profound implications for future NLP development, enabling more flexible, efficient, and capable AI systems. Its multi-task training paradigm reduces the need for task-specific models, streamlining deployment and maintenance. While challenges remain—such as computational costs and handling extremely long texts—the foundational ideas introduced here open new avenues for research, including multimodal integration and adaptive span prediction strategies, promising a more intelligent and versatile NLP ecosystem.
Deep Analysis
Background
近年来，预训练模型在自然语言处理（NLP）领域取得了巨大突破。BERT（Devlin et al., 2019）通过掩码语言模型实现了双向编码，极大提升了理解能力。GPT（Radford et al., 2018）采用自回归预测，擅长文本生成。T5（Raffel et al., 2020）结合编码-解码架构，支持多任务学习，兼顾理解与生成。然而，这些模型各有局限：BERT在生成任务中表现不足，GPT在理解任务中受限，T5参数庞大且训练复杂。近年来，研究者试图融合多目标、多任务策略，但缺乏一种统一高效的架构。GLM的出现，旨在解决这一难题，通过创新的空白填充目标，结合多任务训练和位置编码技术，推动模型向更通用、更高效的方向发展。
Core Problem
现有预训练模型在多任务、多场景应用中存在性能瓶颈。BERT在NLU任务中表现优异，但在生成任务中受限；GPT擅长生成，但理解能力不足；T5虽兼顾两者，但参数庞大且训练复杂。如何设计一个既能理解又能生成的统一模型，满足不同任务的需求，是当前的核心难题。此外，模型在长文本理解、跨片段依赖捕获方面仍有不足，限制了其在复杂场景中的应用。解决这些问题，要求模型具备更高的灵活性和泛化能力。
Innovation
本文提出的GLM具有三大创新：• 引入二维位置编码，区分片段内外位置，增强长文本理解能力；• 采用随机打乱的空白片段策略，支持任意顺序预测，提升模型灵活性；• 结合多任务预训练，支持NLU、条件和无条件生成，打破单一任务限制。这些创新使模型在保持参数效率的同时，实现了多场景、多任务的优异表现。特别是，支持多任务训练的设计，使得模型可以在同一架构下同时处理理解和生成任务，极大提高了模型的实用性和适应性。
Methodology
- �� 采样连续文本片段作为空白区域，长度由Poisson分布控制，随机打乱顺序，形成多任务训练样本；• 利用二维位置编码，分别表示片段内位置和片段间位置，增强模型对长文本和跨片段依赖的理解；• 设计多任务目标，包括短片段填充、长文本生成和句子级空白填充，支持多场景应用；• 改良Transformer架构，调整层归一化和残差连接顺序，提升训练稳定性；• 在训练中，根据任务类型调整空白比例和目标任务的损失权重，确保模型兼顾多任务性能。
Experiments
模型在BookCorpus和Wikipedia上预训练，参数规模从110M到515M。评估包括SuperGLUE、CNN/DailyMail、XSum、SQuAD等。对比基线为BERT、T5、RoBERTa，采用标准微调策略。实验设计涵盖单任务和多任务训练，调节空白片段比例，验证不同目标对性能的影响。模型在多个任务上微调，评估其泛化能力和多场景适应性。训练250,000步，确保模型充分学习语义和结构信息。
Results
GLM在SuperGLUE上平均提升4.6%-5.0%，在NLU和生成任务中均优于BERT、T5等。参数为BERT Large的1.25倍，训练数据相当，表现出极强的泛化能力。多任务预训练后，模型在问答、摘要、文本生成等任务中表现优异，验证了其多场景适应性。随机片段打乱和二维位置编码显著改善模型对长文本和跨片段依赖的理解，验证了设计的有效性。
Applications
模型可广泛应用于智能问答、内容生成、机器翻译、语义理解等场景。其多任务能力使开发者能在单一模型上实现多种应用，减少模型部署和维护成本。未来，结合多模态信息，模型有望在图像、视频等多媒体内容理解中发挥作用，推动多模态AI的发展。
Limitations & Outlook
尽管模型具有高度灵活性，但在极长文本或复杂依赖关系场景下仍存在理解不足的问题，受限于位置编码和片段打乱策略的表达能力。训练成本较高，参数规模大时对计算资源需求高。多任务调节仍需优化，部分任务表现不稳定。未来需在模型压缩和效率提升方面持续努力。
Plain Language Accessible to non-experts
想象你在一家工厂工作，工厂里有很多不同的机器，每台机器负责不同任务。有的机器负责组装零件，有的负责包装。以前的模型就像这些机器，只能做单一任务，要么理解内容（像理解文章），要么写作（像生成文本）。但现在，GLM就像一台多功能的超级机器，既能理解复杂的指令，也能创造新内容。它用一种特别的“调度系统”——类似于在工厂里随机安排任务，确保每个部分都能灵活应对不同工作需求。它还能记住长长的生产线上的所有细节，确保每个零件都能准确装配。这样一来，无论是理解文章、回答问题，还是写故事，它都能胜任。它的秘密武器是用一种特殊的“地图”——二维位置编码，让它知道每个零件在工厂里的具体位置，确保每个任务都能准确完成。这就像一台聪明的工厂机器人，能应对各种复杂任务，变得越来越智能和高效。
ELI14 Explained like you're 14
想象你在学校里，有一台超级聪明的机器人老师。这个机器人可以帮你理解难懂的课文，也可以帮你写作文，还能回答你的问题。以前的机器人老师要么只会理解问题（像BERT那样），要么只会写东西（像GPT那样），但不能同时做好两件事。现在，这个新机器人叫GLM，它就像一个万能的老师。它有一个特别的“记忆系统”，让它知道每个词在句子里的位置，就像你在学校里记住每个同学的座位一样。它还可以随机打乱任务的顺序，就像老师随机抽题，训练它更聪明。这样一来，它不仅能理解复杂的问题，还能写出流畅的文章。它在很多测试中都表现得比以前的机器人老师更好，比如理解问题的准确率提高了5%，写作的质量也更高。未来，这个机器人还能帮忙翻译、写故事，甚至帮你做作业。虽然它还不是完美的，但已经是非常厉害的老师了！
Abstract
There have been various types of pretraining architectures including autoencoding models (e.g., BERT), autoregressive models (e.g., GPT), and encoder-decoder models (e.g., T5). However, none of the pretraining frameworks performs the best for all tasks of three main categories including natural language understanding (NLU), unconditional generation, and conditional generation. We propose a General Language Model (GLM) based on autoregressive blank infilling to address this challenge. GLM improves blank filling pretraining by adding 2D positional encodings and allowing an arbitrary order to predict spans, which results in performance gains over BERT and T5 on NLU tasks. Meanwhile, GLM can be pretrained for different types of tasks by varying the number and lengths of blanks. On a wide range of tasks across NLU, conditional and unconditional generation, GLM outperforms BERT, T5, and GPT given the same model sizes and data, and achieves the best performance from a single pretrained model with 1.25x parameters of BERT Large , demonstrating its generalizability to different downstream tasks.