A Contrastive Framework for Neural Text Generation
Contrastive training (SimCTG) and contrastive search improve diversity and coherence in neural text generation, outperforming SOTA methods.
Key Findings
Methodology
This work introduces SimCTG, a contrastive training framework that incorporates a contrastive loss (LCL) to enhance the discriminability and isotropy of token representations. The training process pulls apart similar token embeddings by maximizing cosine distances, leading to a more balanced representation space. During decoding, the contrastive search algorithm combines model probability with a similarity-based penalty, selecting tokens that are both probable and semantically discriminative. Extensive experiments on datasets like Wikitext-103, LCCC, and DailyDialog demonstrate that this approach surpasses traditional decoding methods such as greedy, beam, and nucleus sampling in metrics like perplexity, diversity, and semantic coherence. The method is architecture-agnostic and effective across multiple languages and tasks, including dialogue generation. Results show significant improvements in intrinsic language modeling quality and human judgments, with the large-scale models approaching human-level fluency and coherence. The work addresses fundamental issues of representation anisotropy and degeneration, offering a new paradigm for controllable, diverse, and coherent text generation.
Key Results
- On Wikitext-103, SimCTG reduces perplexity from 24.32 to 23.82, and improves token prediction accuracy from 39.63% to 40.91%. Contrastive search yields over 28% increase in diversity metrics and raises MAUVE scores to 0.61, indicating more human-like distributions.
- Human evaluations show that SimCTG + contrastive search scores above 3.5/5 in coherence and fluency, outperforming baseline methods significantly. The large model (SimCTG-large) even matches human fluency scores, demonstrating strong generalization.
- Analysis reveals that contrastive training reduces token representation similarity, especially at the output layer, leading to better discriminability. Tuning the margin ρ at 0.5 optimizes performance, balancing isotropy and expressiveness.
Significance
This research fundamentally advances neural text generation by addressing the core issue of representation anisotropy, which causes degeneration and repetition. By integrating contrastive learning into training and decoding, it enables models to produce more diverse, coherent, and human-like texts. The approach is versatile, applicable across languages and tasks, and compatible with large-scale pretraining. It paves the way for more natural language generation systems, impacting applications from content creation to conversational AI. The method also opens new avenues for research into representation calibration and decoding algorithms, fostering further innovation in the field.
Technical Contribution
The key technical innovation is the design of the contrastive loss LCL, which enforces a discriminative, isotropic token embedding space. This contrasts with traditional likelihood-based training, which often results in anisotropic, clustered representations. The contrastive search algorithm further leverages these improved representations by balancing model confidence with a similarity-based penalty, effectively promoting diversity without sacrificing coherence. The approach is compatible with existing transformer-based models like GPT-2, and experimental results demonstrate its superiority in multiple benchmarks. The framework also provides theoretical insights into how representation geometry influences generation quality, offering a new perspective on model calibration.
Novelty
This work is the first to systematically incorporate contrastive learning into neural text generation training, directly targeting the representation space's geometry. Unlike prior methods that rely solely on sampling or modified training objectives, it emphasizes the importance of representation discriminability. The contrastive search decoding strategy innovatively combines probability and similarity metrics, setting a new standard for balancing diversity and coherence. This dual approach—representation calibration plus similarity-aware decoding—constitutes a significant departure from existing paradigms, offering a comprehensive solution to degeneration issues.
Limitations
- 虽然在多语种、多任务场景中表现优异，但在极长文本生成或低资源环境下，模型的表示空间仍可能不足以完全避免重复和不连贯。
- 调节对比损失边界参数ρ需要在不同任务和数据集上进行细致调优，增加了使用复杂度。
- 大规模模型的训练和推理成本较高，实际部署时存在效率瓶颈，需优化算法以提升效率。
Future Work
未来将探索自适应调节对比损失边界的方法，提升模型在长文本和多模态任务中的表现。还计划结合预训练大模型（如GPT-3）进行迁移学习，扩展对比搜索策略的适用范围，进一步增强生成内容的多样性和语义一致性。
AI Executive Summary
Neural text generation has become a cornerstone of modern natural language processing, powering applications from storytelling to dialogue systems. However, existing decoding methods such as greedy search, beam search, and nucleus sampling often produce dull, repetitive, and incoherent outputs, limiting their practical utility. These issues stem from fundamental problems in the models’ internal representations, which tend to be clustered and anisotropic, causing the models to generate monotonous and degenerate content. Addressing this challenge requires a paradigm shift in both training and decoding strategies.
This paper introduces SimCTG, a contrastive training framework that incorporates a novel contrastive loss to calibrate the token embedding space. By maximizing the cosine distance between different token representations, SimCTG enhances the discriminability and isotropy of the embeddings, leading to more balanced and meaningful representations. This calibration reduces the tendency of models to produce repetitive tokens and improves their overall understanding of semantic distinctions.
Complementing the training approach, the authors propose contrastive search, a decoding strategy that combines the model’s predicted probabilities with a similarity-based penalty. This method encourages the selection of tokens that are both probable and semantically discriminative, thereby fostering diversity while maintaining coherence. Extensive experiments on datasets like Wikitext-103, LCCC, and DailyDialog demonstrate that their approach outperforms traditional methods, achieving lower perplexity, higher diversity scores, and superior human evaluation results.
Results show that models trained with SimCTG and decoded with contrastive search produce more varied, fluent, and contextually appropriate texts. For instance, on Wikitext-103, the perplexity drops from 24.32 to 23.82, and the diversity increases by over 28%. Human judges rated the outputs significantly better in coherence and fluency, with scores approaching those of human-generated texts, especially in larger models.
This work addresses a core issue in neural language modeling—representation anisotropy—and offers a practical, scalable solution that enhances the quality of generated language. Its implications extend to improving dialogue systems, content creation, and AI storytelling, marking a significant step toward more natural and engaging machine-generated language. Future directions include scaling to larger models like GPT-3, optimizing computational efficiency, and exploring multi-modal extensions.
Deep Analysis
Background
自然语言生成技术经历了从基于规则的系统到深度学习模型的飞跃。Transformer架构（如GPT系列、BERT）极大提升了生成质量，但仍存在内容重复、单调的问题。传统训练方法（最大似然估计）结合贪婪或束搜索，保证语法正确，但缺乏多样性。采样策略（如核采样）引入多样性，但容易偏离语义，导致内容不连贯。近年来，研究者开始关注模型表示空间的结构，尝试通过正则化或对比学习改善词元判别性，以提升生成质量。本文在此基础上，提出对比训练和解码新策略，旨在根本改善模型的表示判别性和生成多样性，推动生成技术的进一步发展。
Core Problem
神经文本生成中，模型的词元表示空间往往表现出各向异性，导致生成内容重复、缺乏多样性和语义连贯性。传统的最大似然训练未能充分校准词元的判别能力，导致模型在解码时偏向于生成相似或重复的内容。此外，现有采样方法虽然增加多样性，但容易引入语义偏差，影响内容的连贯性。这些问题限制了生成模型在实际应用中的表现，亟需新的训练和解码机制，以提升生成内容的丰富性和语义一致性。
Innovation
本研究的创新点在于引入对比学习机制，校准模型的词元表示空间，使其更具判别性和各向同性。具体包括：1) 设计对比损失LCL，通过最大化不同词元间的余弦距离，增强词元区分能力；2) 提出对比搜索解码策略，将模型概率与表示相似性结合，优先选择多样且连贯的词元；3) 在多语种、多任务场景中验证方法的有效性。该方案突破了传统仅依赖概率最大化的局限，为生成模型提供了更深层次的优化路径，显著改善了内容多样性和语义连贯性。
Methodology
- �� 训练阶段：
- 输入：文本序列x
- 过程：引入对比损失LCL，计算每个词元的表示h_xi，最大化不同词元的余弦距离，减少表示的相似性，从而提升判别性。
- 输出：校准后的模型参数，使词元表示更具判别性和各向同性。
- �� 解码阶段：
- 输入：前缀x<t
- 过程：在候选集V(k)中，结合模型预测概率和最大表示相似性，计算得分：
- 1) 模型置信度：p_θ(v|x<t)
- 2) 表示判别惩罚：max{s(h_v, h_xj)}
- 3) 选择得分最高的v作为输出。
- 输出：多样且连贯的文本。
- �� 结合训练与解码：
- 训练：最大似然与对比损失联合优化
- 解码：对比搜索策略，平衡多样性与语义一致性，提升生成质量。
Experiments
采用Wikitext-103、LCCC和DailyDialog数据集，比较基线包括MLE、无概率训练和核采样。训练参数：40k步，批次128，最大长度256。解码方法：贪婪、束搜索、核采样（p=0.95）和对比搜索（k=8，α=0.6）。评估指标涵盖困惑度、预测准确率、重复率、多样性、MAUVE、语义连贯性和人类评分。调节超参数，验证模型在多任务、多语种环境中的适应性和效果。
Results
SimCTG在困惑度和预测准确率方面优于基线，困惑度从24.32降至23.82，准确率提升至40.91%。对比搜索显著改善生成多样性（提升28%）和语义连贯性（得分0.95），在MAUVE和人类评估中表现优异。模型表示的判别性增强，词元间的相似性降低，验证了对比训练的有效性。调节参数ρ，发现0.5最优，进一步优化模型性能。
Applications
该方法适用于自动内容生成、对话系统、智能写作等场景，尤其在需要多样性和语义一致性的应用中表现出色。模型训练依赖大规模文本数据，解码策略可结合不同生成任务调整参数，提升生成内容的丰富性和连贯性。未来，结合预训练大模型，有望实现更高水平的自然语言理解与生成。
Limitations & Outlook
在极长文本生成或低资源环境下，模型表示空间仍可能不足以完全避免重复和不连贯。调节对比损失参数增加调优复杂度，训练和推理成本较高，限制实际部署效率。未来需优化算法，提升效率，增强模型鲁棒性。
Plain Language Accessible to non-experts
想象你在一家厨房做菜，所有食材都放在一个大箱子里。传统做法就像随便拿食材，可能会拿到重复或不搭配的材料，做出来的菜也就单调或奇怪。现在，厨师用一种特别的方法，把不同食材分类得更清楚，确保每次拿到的材料都新鲜且搭配得当。这个方法就像给模型的词元做“整理”，让它们更有区分度，避免重复。然后，厨师在挑选下一道菜时，不仅看食材的受欢迎程度，还会考虑它们是否搭配得好。这样做出来的菜既丰富又美味，也不会重复或跑题。这个厨房的秘诀，就是用“对比”让每个食材都变得独一无二，菜肴也变得更加多样和有趣。
ELI14 Explained like you're 14
想象你在玩拼图游戏，目标是拼出一幅漂亮的画。以前的方法就像随便拼拼，可能会拼出重复的部分，画面看起来单调。现在，有个聪明的朋友教你用一种新技巧：每次拼下一块时，不仅看它和之前的拼块的匹配程度，还会考虑这块是不是和其他部分差别大。这样，你拼出来的画既丰富又不重复，也更有趣。这就像这篇论文里的新方法，用“对比”让每个词都更特别，避免重复，让生成的文本既多样又连贯。通过这个技巧，计算机写作变得更像人类，能讲出更丰富、更自然的故事和对话。
Abstract
Text generation is of great importance to many natural language processing applications. However, maximization-based decoding methods (e.g. beam search) of neural language models often lead to degenerate solutions -- the generated text is unnatural and contains undesirable repetitions. Existing approaches introduce stochasticity via sampling or modify training objectives to decrease probabilities of certain tokens (e.g., unlikelihood training). However, they often lead to solutions that lack coherence. In this work, we show that an underlying reason for model degeneration is the anisotropic distribution of token representations. We present a contrastive solution: (i) SimCTG, a contrastive training objective to calibrate the model's representation space, and (ii) a decoding method -- contrastive search -- to encourage diversity while maintaining coherence in the generated text. Extensive experiments and analyses on three benchmarks from two languages demonstrate that our proposed approach significantly outperforms current state-of-the-art text generation methods as evaluated by both human and automatic metrics.