Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models

Recent years have seen rapid growth in pretrained language models like BERT, RoBERTa, and ELECTRA, which have significantly advanced sentence embedding techniques. Early methods such as InferSent and S-BERT used supervised or fine-tuned approaches to generate sentence vectors, achieving high accuracy in semantic similarity and transfer tasks. The advent of contrastive learning, exemplified by SimCSE, further improved embedding quality by optimizing distance properties. Meanwhile, T5 introduced a unified text-to-text framework trained on multi-task objectives, excelling in generative tasks and broad NLP benchmarks. However, its potential for sentence-level representation remained underexplored. This study bridges this gap by systematically evaluating T5's capacity to produce high-quality sentence embeddings, leveraging large-scale pretraining and contrastive fine-tuning.

Despite T5’s success in various NLP tasks, extracting effective sentence embeddings from its architecture poses challenges. The absence of a dedicated [CLS] token and the anisotropy of embeddings hinder direct use. Existing approaches often rely on task-specific fine-tuning, which limits generality and efficiency. Furthermore, scaling models to billions of parameters raises questions about the transferability and robustness of the resulting embeddings. Addressing these issues requires developing universal extraction methods, understanding the impact of model size, and designing training strategies that mitigate embedding collapse, all while maintaining computational feasibility.

The core innovations include: 1) proposing three extraction methods—encoder-first token, mean pooling, and decoder-first output—tailored for T5’s architecture; 2) establishing SentGLUE, a comprehensive benchmark for sentence transfer tasks; 3) employing multi-stage contrastive learning on QA and NLI datasets to enhance embedding quality; 4) systematically scaling models from hundreds of millions to tens of billions of parameters, demonstrating performance gains. These innovations collectively enable the first large-scale application of T5 for sentence embeddings, surpassing prior encoder-only models and setting new benchmarks.

• �� Design three extraction strategies: encoder-only first token, mean pooling, decoder-first output. • Initialize dual encoders from pre-trained T5, add projection and normalization layers. • Use contrastive loss with positive and negative pairs, employing in-batch sampling and multi-stage training on QA and NLI data. • Fine-tune models with large batch sizes (512-2048), optimizer Adafactor, learning rate 0.001, temperature τ=0.01. • Evaluate embeddings on SentEval and SentGLUE, focusing on transfer accuracy and Spearman correlation on STS. • Scale models from 110M to 11B parameters, analyzing the impact of size on performance.

The experiments utilize web question-answer pairs and SNLI datasets for multi-stage training, with TPU-v8 hardware. Baselines include BERT, RoBERTa, and existing SOTA models. Hyperparameters such as learning rate, batch size, and temperature are carefully tuned. The evaluation covers transfer tasks (classification accuracy) and semantic similarity (Spearman’s correlation). Ablation studies compare extraction methods, model sizes, and fine-tuning strategies. Results show that larger models and contrastive training significantly improve performance, validating the approach’s scalability and robustness.

Unsupervised mean pooling on 11B models achieves 90.46 on STS, outperforming previous models. Fine-tuned models with contrastive loss reach 84.94, establishing new SOTA. Encoder-only models scale better for transfer tasks, while encoder-decoder models excel in similarity metrics. The results confirm that larger models and multi-stage training synergistically enhance embedding quality, with performance improvements consistent across datasets. These findings demonstrate the effectiveness of combining scale, architecture, and contrastive learning for high-quality sentence representations.

The high-quality embeddings can be directly applied to semantic search engines, question-answering systems, and content recommendation platforms. They enable fast retrieval and accurate semantic matching across large corpora. In industry, such embeddings facilitate intelligent information retrieval, content filtering, and multilingual understanding. Future integration with multimodal data could further expand their utility, supporting applications like cross-modal retrieval and multimedia understanding, transforming how machines interpret human language.

Despite scaling, embeddings still face anisotropy and collapse issues, especially in raw form, requiring additional regularization. Large models demand substantial computational resources, limiting deployment in resource-constrained environments. Fine-tuning relies on labeled data, which can be costly and domain-specific, restricting generalization. Further research is needed to develop more efficient training, better regularization techniques, and methods to extend applicability to low-resource languages and specialized domains.