Trading Off Diversity and Quality in Natural Language Generation

The evolution of natural language generation (NLG) has transitioned from rule-based systems to deep learning models like Transformers (Vaswani et al., 2017). Early decoding strategies such as greedy search and beam search prioritized likelihood maximization but often resulted in repetitive or dull outputs. Recent methods like top-k and nucleus sampling (Holtzman et al., 2019) aimed to enhance diversity, yet lacked systematic evaluation frameworks. Despite advances in model architectures and training techniques, decoding remains a bottleneck, especially for open-ended tasks like dialogue and storytelling, where balancing quality and diversity is crucial. Prior work focused on improving models or heuristics, but few studies quantitatively assessed decoding strategies across the entire spectrum of output quality and diversity, leaving a gap in understanding how best to optimize these tradeoffs.

The core challenge in NLG decoding is achieving an optimal balance between response quality and diversity. Existing algorithms often excel in one at the expense of the other—greedy decoding produces high-quality but monotonous outputs, while sampling methods generate diverse but sometimes incoherent responses. Moreover, the reliance on maximum likelihood estimates leads to the 'likelihood trap,' where sequences with the highest probabilities are surprisingly low in human-perceived quality. This disconnect hampers practical deployment in user-facing applications. The absence of a unified evaluation framework further complicates fair comparison and systematic improvement of decoding algorithms. Addressing these issues requires a principled approach that explicitly models the tradeoff and provides scalable, effective solutions.

This paper introduces a multi-objective optimization framework for decoding, combining human-judged quality and Shannon entropy as core metrics. It proposes a globally-normalized temperature sampling method—selective sampling—using rejection sampling to approximate the optimal distribution without explicit partition function calculation. The key innovations include: 1) formalizing the quality-diversity tradeoff; 2) revealing the 'likelihood trap' phenomenon; 3) designing a rejection-based sampling algorithm that balances global and local probabilities; 4) empirically validating these methods on large-scale human ratings. These contributions provide a new theoretical foundation and practical tools for decoding, addressing longstanding issues in open-ended language generation.

• �� Define a multi-objective goal G(p)=Q(p)+λH(p), where Q(p) is human-judged quality and H(p) Shannon entropy.
• �� Collect large-scale human ratings (38,000 samples) across various decoding strategies (temperature, top-k, nucleus).
• �� Quantify the relationship between model likelihood and human quality, identifying the 'likelihood trap' at log p ≈ -58.09.
• �� Develop a globally-normalized temperature sampling algorithm, leveraging rejection sampling to approximate the distribution p ∝ pmodel^{1/τ} without computing the partition function.
• �� Introduce selective sampling, setting a likelihood cutoff α, rejecting samples exceeding this threshold.
• �� Evaluate the methods through extensive human ratings, comparing performance across different parameters and strategies.
• �� Analyze the impact of hyperparameters, especially α and λ, on quality and diversity tradeoffs.

• �� Use GPT-2 774M to generate 30-token responses conditioned on 48 prompts, ensuring diversity in context.
• �� Human evaluators rate responses on a 5-point scale, focusing on overall quality.
• �� Compare multiple decoding algorithms—temperature, top-k, nucleus, and selective sampling—across hyperparameter ranges.
• �� Measure entropy and quality scores, analyzing their correlation.
• �� Validate the 'likelihood trap' by examining the relationship between likelihood and human judgment.
• �� Test selective sampling’s effectiveness, adjusting cutoff α based on observed inflection points.
• �� Conduct ablation studies to understand the influence of parameters and the impact of rejection sampling on output quality.

• �� Nucleus sampling consistently outperforms other methods in quality when λ is high, with human ratings improving by approximately 15%. In high-diversity regimes, all methods perform similarly.
• �� The 'likelihood trap' is empirically confirmed: sequences with the highest likelihood (log p ≈ -58.09) tend to be rated lower in quality, revealing a nonlinear relationship.
• �� Selective sampling, despite theoretical appeal, underperforms in some scenarios due to parameter sensitivity and prior biases.
• �� The framework provides a rigorous basis for decoding evaluation, highlighting the importance of balancing global and local probabilities.

• �� Enhances dialogue systems by generating more diverse and coherent responses.
• �� Improves storytelling and content creation tools, enabling richer and more engaging outputs.
• �� Can be integrated into user-facing applications requiring controllable diversity and quality, such as chatbots and virtual assistants.
• �� Future integration with reinforcement learning could allow dynamic adjustment of tradeoff parameters for real-time optimization.

• �� Human ratings are subjective, potentially biasing results; objective metrics remain challenging.
• �� Computational costs of global normalization and rejection sampling limit real-time deployment.
• �� Fixed cutoff α may not adapt well across different prompts or tasks, requiring further tuning.
• �� The framework’s generality across diverse models and languages needs further validation.