Prefix-Tuning: Optimizing Continuous Prompts for Generation

Recent advances in pre-trained language models (e.g., GPT-2, BART) have significantly improved NLP performance across tasks such as text summarization, machine translation, and table-to-text generation. These models leverage massive datasets and complex architectures, but their adaptation to downstream tasks typically involves full parameter fine-tuning, which is costly in storage and computation. Lightweight methods like adapters and prompting have emerged to mitigate this, yet they still require additional parameters or manual prompt design. The challenge remains to develop a universal, parameter-efficient tuning method that maintains high performance and generalizes well, especially in low-data and out-of-domain scenarios. This paper builds upon prompting ideas, integrating trainable continuous prompts to optimize task-specific guidance without altering the core model weights.

Existing fine-tuning approaches are parameter-heavy, requiring full model updates and storage per task, which limits scalability. In multi-task or personalized settings, this becomes infeasible. Moreover, models often struggle to generalize to unseen topics or domains when fine-tuned on limited data. Prompting methods, while parameter-efficient, lack the expressive power of learned prompts. The core problem is to design a method that combines the efficiency of prompting with the adaptability of fine-tuning, enabling models to be quickly and effectively adapted to new tasks or domains with minimal parameter updates, while maintaining or improving performance.

The paper introduces Prefix-Tuning, which inserts a trainable continuous prefix into the model input, influencing multi-layer activations as virtual tokens. This approach preserves the pre-trained weights, reducing the number of trainable parameters to 0.1%. It employs a re-parameterization strategy (MLP) to stabilize training. Unlike discrete prompts, continuous prompts are more expressive, allowing better task steering. The method is compatible with autoregressive and encoder-decoder models, enabling broad applicability. It also supports modular multi-task learning, as each task can have its own prefix stored separately, facilitating efficient deployment.

• �� Initialize a fixed pre-trained model (GPT-2, BART).• Create a trainable prefix matrix Pθ of size |Pidx| × dim(hi).• The prefix acts as virtual tokens, influencing activations hi at each layer.• During training, only Pθ is optimized; model weights remain frozen.• The prefix is re-parameterized via an MLP for stability.• The training objective minimizes task-specific loss (e.g., BLEU, ROUGE).• For multi-task settings, each task has its own prefix, stored separately.• The approach is compatible with autoregressive and encoder-decoder architectures, influencing activations throughout the network.

The authors evaluate on three table-to-text datasets (E2E, WebNLG, DART) and one summarization dataset (XSUM). They compare prefix tuning with full fine-tuning, partial tuning, and adapter tuning, using metrics like BLEU, METEOR, and ROUGE. Hyperparameters include prefix length, learning rate, and training epochs. Experiments test in low-data regimes and out-of-domain generalization, with ablation studies on prefix size and re-parameterization. Results are validated across different model sizes (GPT-2 Medium/Large, BART Large).

Prefix-Tuning with only 0.1% parameters consistently outperforms adapter tuning and approaches full fine-tuning in performance. On table datasets, it improves BLEU scores by 4.1 points on average. In low-data scenarios, it surpasses full fine-tuning, especially with fewer than 200 samples. External topic generalization tests show superior robustness, with significant metric improvements over traditional methods, confirming its effectiveness and efficiency for scalable NLP deployment.

This method enables rapid, storage-efficient adaptation of large models for multi-task NLP applications, including personalized assistants, content generation, and low-resource language tasks. It simplifies deployment pipelines by requiring only task-specific prefix storage, facilitating scalable, multi-user systems. Future integration with automated prefix generation could further enhance adaptability and performance.

While highly parameter-efficient, Prefix-Tuning may have limited capacity for extremely complex tasks or very small datasets, where prefix size constrains expressiveness. Optimization stability can be sensitive to hyperparameters such as learning rate. Its effectiveness on larger or different architectures beyond GPT-2 and BART remains to be validated. Further, performance gaps in some tasks suggest room for combining with other adaptation strategies.