Multitask Prompted Training Enables Zero-Shot Task Generalization
Multitask prompted training (T0) achieves state-of-the-art zero-shot performance, outperforming models up to 16× larger on diverse NLP tasks.
Key Findings
Methodology
This work introduces a systematic approach to convert numerous NLP tasks into human-readable prompts, using structured templates. By aggregating diverse datasets into a multi-task prompt-based training set, the model learns to generalize to unseen tasks. The core architecture is based on the T5 encoder-decoder model, fine-tuned on this mixture. Emphasizing prompt diversity and task variety, the model demonstrates remarkable zero-shot capabilities, surpassing larger models like GPT-3 in several benchmarks. The training involves multiple prompts per dataset, enhancing robustness to prompt phrasing variations.
Key Results
- The model outperforms baseline T5+LM on 9 out of 11 held-out datasets, with an average improvement of over 15%. It surpasses GPT-3 175B on several tasks, notably achieving 85% accuracy on NLI datasets.
- On BIG-bench, T0 variants outperform baseline models, with T0++ exceeding 6× model size in some tasks, demonstrating the efficacy of prompt diversity and multi-task training.
- Robustness to prompt wording improves with more prompt variations, reducing performance variability and increasing stability across different prompts.
Significance
This research shifts the paradigm from scale-dependent models to prompt-based generalization, enabling smaller models to perform competitively on diverse tasks. It addresses core challenges in NLP, such as data scarcity and task adaptability, by leveraging prompt diversity. The approach reduces reliance on massive parameters, lowering computational costs and broadening accessibility. It paves the way for more flexible, efficient, and scalable NLP systems capable of handling real-world, multi-faceted tasks with minimal supervision.
Technical Contribution
The main technical innovation is the development of a structured prompt format that captures task diversity, combined with multi-task training on a large, diverse prompt dataset. The method enhances the model’s ability to interpret and adapt to different task instructions, leading to superior zero-shot performance. The approach integrates prompt templates, multiple prompts per dataset, and a robust training regimen, setting a new standard for prompt-based multi-task learning. It also demonstrates that prompt diversity can compensate for model size, offering a scalable solution for generalization.
Novelty
This is the first comprehensive systematization of multi-task prompted training, explicitly leveraging prompt diversity to improve zero-shot generalization. Unlike prior work that used fixed prompts or limited task sets, this approach employs a broad, diverse prompt collection and trains on multiple tasks simultaneously. It shifts the focus from sheer model size to prompt and task diversity, establishing a new paradigm in NLP model training. The experimental results across multiple benchmarks validate its effectiveness, marking a significant advancement over existing methods.
Limitations
- Despite improvements, the model remains sensitive to prompt phrasing, and poorly designed prompts can degrade performance. Automating prompt optimization remains an open challenge.
- Training on large, diverse prompt sets increases data collection and engineering complexity, potentially limiting rapid deployment.
- The approach’s effectiveness in low-resource or highly specialized domains is yet to be fully explored, and performance gaps may persist in such areas.
Future Work
Future research will focus on automating prompt generation and optimization, reducing manual effort. Combining prompt diversity with few-shot learning techniques could further improve performance in low-resource settings. Additionally, exploring the semantic understanding of prompts and extending the framework to multilingual scenarios are promising directions. Developing methods to automatically generate effective prompts and adapt them to new domains will be critical for broader applicability.
AI Executive Summary
The rapid evolution of large language models has demonstrated impressive capabilities in zero-shot learning, yet scaling models remains costly and impractical for widespread deployment. Addressing this, the present work introduces a novel multi-task prompted training framework, termed T0, which leverages diverse natural language prompts to significantly enhance zero-shot generalization. The core idea is to convert a wide array of NLP tasks into human-readable prompts using structured templates, enabling the model to learn task-agnostic representations that generalize to unseen tasks.
This approach builds upon the T5 encoder-decoder architecture, fine-tuned on a large, diverse set of prompt-formatted datasets. By incorporating multiple prompts per task, the model becomes robust to variations in prompt phrasing, which is a common challenge in prompt-based NLP. Extensive experiments demonstrate that T0 matches or exceeds the performance of much larger models like GPT-3 on several benchmarks, including 9 out of 11 held-out datasets and multiple BIG-bench tasks. Notably, T0 outperforms GPT-3 175B on several tasks despite being significantly smaller.
The significance of this work lies in its potential to democratize NLP capabilities, reducing the need for enormous models and expensive training. It offers a scalable, flexible, and efficient paradigm that can adapt to new tasks with minimal additional data or tuning. The findings suggest that prompt diversity and multi-task training are key to unlocking the full potential of language models, paving the way for more accessible AI systems.
Looking ahead, future work will explore automated prompt generation, domain adaptation, and multilingual extensions, aiming to further improve robustness and applicability. This research marks a pivotal step toward versatile, low-cost, high-performance NLP systems capable of understanding and executing a broad spectrum of tasks with minimal supervision.
Deep Analysis
Background
近年来，预训练语言模型如BERT、GPT系列和T5在NLP领域引领了技术革新。它们通过大规模无监督预训练，掌握了丰富的语言知识，显著提升了多项任务性能。尤其是GPT-3等超大模型，展现出惊人的零样本和少样本能力，推动模型规模与性能同步增长。然而，规模的不断扩大带来了高昂的计算成本和部署难题。近年来，提示工程和多任务学习逐渐成为研究热点，旨在通过设计提示（prompt）引导模型完成新任务。早期工作如GPT-3的few-shot学习和T5的多任务训练已验证提示在泛化中的潜力，但仍存在提示敏感和任务多样性不足的问题。本文在此基础上，提出多任务提示训练策略，旨在系统性增强模型的零样本泛化能力，解决提示设计的鲁棒性和任务适应性难题。
Core Problem
尽管大规模预训练模型在零样本任务中表现优异，但其性能高度依赖模型规模，且对提示措辞敏感，限制了其广泛应用。如何在不依赖极大参数的情况下，提升模型对新任务的泛化能力，成为亟待解决的问题。传统多任务学习虽能改善泛化，但缺乏系统化的提示设计，难以应对任务多样性和提示变化带来的挑战。此外，现有方法多依赖静态提示或少样本调优，缺乏对提示多样性和鲁棒性的系统研究。因此，本文试图通过丰富多样的提示模板和多任务训练，提升模型在未见任务上的表现，并增强对提示变化的适应能力。
Innovation
第一，提出多任务提示（prompt）训练框架，将多样化任务转化为人类可理解的提示模板，增强模型的任务理解能力；第二，设计丰富的提示模板集合，涵盖多种任务类型和措辞变体，提升模型对提示措辞变化的鲁棒性；第三，利用预训练的T5模型，通过在多任务提示数据上微调，显著提升模型的零样本泛化能力，超越参数规模远大于它的模型。此方法结合了提示工程和多任务学习的优势，突破了以往只依赖模型规模的局限，为低成本、多任务泛化提供新路径。
Methodology
- �� 数据准备：收集62个公开任务数据集，设计多样化提示模板，确保任务多样性和提示多样性。• 模型架构：采用预训练的T5编码-解码模型，利用其生成能力。• 多任务微调：将所有任务数据按比例混合，利用多样化提示模板进行训练，目标是最大似然估计。• 提示多样性：每个任务使用多个不同的提示模板，训练模型对措辞变化的鲁棒性。• 训练细节：采用Adafactor优化器，最大序列长度1024，批次大小1024，学习率1e-3。• 评估：在未见任务和BIG-bench子集上进行零样本测试，使用准确率指标，比较GPT-3等模型。
Experiments
实验设计包括在4个未见任务（自然语言推理、共指消解、词义消歧、句子补全）上进行零样本评估，使用多样化提示集，比较模型在不同提示下的表现。还在BIG-bench上测试模型的泛化能力，评估不同提示数量和多样性对性能的影响。对比基线包括T5+LM和GPT-3模型，重点验证多任务提示训练的效果。采用中位数和四分位范围衡量模型鲁棒性，确保评估的公平性和代表性。
Results
模型在9个未见任务中表现优异，平均提升15%以上，尤其在NLI任务中，准确率达85%，超越GPT-3 175B模型。多样化提示显著增强模型鲁棒性，提示数量增加时，性能波动减小。BIG-bench测试中，T0系列模型在大多数任务中优于对比模型，T0++表现尤为突出，达6倍模型规模的优势。实验验证了提示多样性和多任务训练对零样本泛化的关键作用。
Applications
该方法适用于需要快速适应新任务的智能助手、自动问答系统和内容生成平台。无需大量标注数据，只需设计多样化提示，即可实现多任务泛化，降低部署门槛。未来，结合自动提示生成和少样本学习，有望在专业领域和低资源场景中实现更广泛应用。
Limitations & Outlook
模型对提示设计仍较敏感，提示质量影响显著。训练过程依赖大量多样化提示，增加工程复杂度。在极端少样本或专业任务中仍存在性能瓶颈，未来需结合少样本学习策略，提升泛化能力。
Plain Language Accessible to non-experts
想象你在一家工厂工作，工厂里有很多不同的任务，比如组装、包装、检验。以前，我们需要专门训练每个工人做每个任务，但现在，我们设计了一套通用的说明书（提示），告诉工人怎么做不同的任务。工人通过学习这些说明书，变得聪明起来，能自己理解新任务，不用每次都重新训练。这就像给模型写不同的提示，让它理解各种任务，从而在遇到新任务时也能轻松应对。这样，工厂的效率大大提高，工人（模型）变得更聪明、更灵活。
ELI14 Explained like you're 14
你知道吗？就像你在学校学不同的科目，比如数学、英语、科学，老师有时候会给你一些提示，比如“用数学的方法解决这个问题”。如果你能理解这些提示，就能用学过的知识解决新问题。科学家们也在做类似的事情，他们训练一个超级聪明的机器人（模型），让它通过不同的提示理解各种任务，比如回答问题、写故事、判断句子意思。最厉害的是，这个机器人不用专门学每个任务，只要给它不同的提示，它就能自己理解并完成任务，就像你用不同的线索解谜一样。这让机器人变得更聪明，也更能帮你做事！
Glossary
Prompt（提示）
一种用自然语言描述任务的方式，指导模型完成特定任务。技术上是模型输入的文本指令，帮助模型理解任务目标。
本文中用多样化提示模板引导模型学习多任务能力。
Zero-Shot（零样本）
模型在未见过特定任务或数据集上，凭借训练中学到的知识直接完成任务。技术上是模型无需额外训练即可推断。
本文重点在于提升模型的零样本泛化能力。
Multitask Prompted Training（多任务提示训练）
在多任务、多提示条件下训练模型，使其能在未见任务上进行推断。结合多样提示增强泛化。
本文提出的核心训练策略。
T5模型
一种基于Transformer的文本到文本预训练模型，将所有任务统一转化为文本到文本的形式。
本文基础模型。
BIG-bench（大规模基准测试）
一套旨在测试大模型多样能力的挑战性任务集合。
评估模型泛化能力的重要基准。
Open Questions Unanswered questions from this research
- 1 如何自动生成更具语义理解的提示，减少人工设计负担，仍是未来研究的难点。
- 2 模型在极端少样本或专业领域任务中的表现仍有限，需结合少样本学习策略。
- 3 提示设计的最佳实践和理论基础尚不充分，亟待系统研究。
Applications
Immediate Applications
智能问答系统
利用多任务提示训练的模型，快速适应不同领域的问答任务，无需大量标注数据，提升效率和准确率。
内容生成平台
通过丰富提示实现多样化内容创作，如文章摘要、对话生成，降低内容生产门槛。
Long-term Vision
通用人工智能
实现具备多任务、多领域能力的AI系统，能自主理解新任务，减少人工干预，推动智能化普及。
Abstract
Large language models have recently been shown to attain reasonable zero-shot generalization on a diverse set of tasks (Brown et al., 2020). It has been hypothesized that this is a consequence of implicit multitask learning in language models' pretraining (Radford et al., 2019). Can zero-shot generalization instead be directly induced by explicit multitask learning? To test this question at scale, we develop a system for easily mapping any natural language tasks into a human-readable prompted form. We convert a large set of supervised datasets, each with multiple prompts with diverse wording. These prompted datasets allow for benchmarking the ability of a model to perform completely held-out tasks. We fine-tune a pretrained encoder-decoder model (Raffel et al., 2020; Lester et al., 2021) on this multitask mixture covering a wide variety of tasks. The model attains strong zero-shot performance on several standard datasets, often outperforming models up to 16x its size. Further, our approach attains strong performance on a subset of tasks from the BIG-bench benchmark, outperforming models up to 6x its size. All trained models are available at https://github.com/bigscience-workshop/t-zero and all prompts are available at https://github.com/bigscience-workshop/promptsource.