Sparse Text Generation

Over the past decade, neural language models like Transformer, GPT, and BERT have dramatically advanced NLP tasks. Early models used softmax layers to produce probability distributions over vocabularies, but this dense output often led to repetitive and dull text, especially in open-ended generation. Researchers introduced sparse transformations such as sparsemax, which can produce zero probabilities for less relevant words, to enhance diversity. However, these were mainly applied in discriminative tasks, with limited success in generative settings. Recent decoding strategies like top-k and nucleus sampling improved diversity but remained heuristic and did not address training-time mismatches. The challenge was to develop models that inherently generate sparse, diverse, and coherent text during training, rather than relying on post-hoc truncation.

Current generative models struggle with balancing fluency, diversity, and coherence. Softmax-based training leads to dense distributions, causing models to favor common words and produce repetitive outputs. Decoding strategies like top-k and nucleus sampling mitigate this but are heuristic and applied only at inference, creating a mismatch with training conditions. Moreover, evaluating sparse or truncated distributions is problematic with traditional perplexity, which cannot handle zero probabilities, hindering fair comparison. These issues limit the ability of models to generate truly diverse, natural language, especially in long-form content and interactive dialogue systems.

The paper introduces α-entmax, a parametric transformation that produces sparse probability distributions with learnable sparsity levels. Unlike softmax, entmax can assign exactly zero probability to irrelevant words, directly during training. This enables the model to learn a distribution that naturally excludes implausible options, improving diversity and coherence. The authors develop a convex, differentiable loss function based on Tsallis α-entropy, facilitating stable training. During inference, sampling directly from the learned sparse distribution avoids heuristic truncation. Additionally, three new metrics—ε-perplexity, sparsemax score, and Jensen-Shannon divergence—are proposed to evaluate the quality of sparse distributions, providing a comprehensive assessment of model performance.

• �� Input: score vector z from the language model. • Transformation: apply α-entmax to produce a sparse probability distribution p. • Loss: optimize using the α-entmax loss, incorporating Tsallis α-entropy for automatic sparsity learning. • Training: end-to-end, convex optimization ensures stability and adaptability. • Decoding: sample directly from p, which varies in size depending on context, enabling dynamic vocabulary usage. • Evaluation: compute new metrics to assess sparsity, diversity, and coherence. • Hyperparameters: tune α for sparsity control, balancing diversity and accuracy. • Goal: achieve training-test consistency, reducing repetition and increasing naturalness.

Using datasets WikiText-2, WikiText-103, and BookCorpus, models were fine-tuned with different sampling strategies: softmax, top-k, nucleus, unlikelihood, and entmax. Hyperparameters such as α, top-k size, and nucleus probability were optimized via grid search. Metrics included perplexity, repetition rate, ε-perplexity, sparsemax score, and JS divergence. Human evaluations assessed story coherence, engagement, and fluency. Ablation studies confirmed that training with entmax loss and decoding with entmax sampling yields the best results, emphasizing the importance of training-decoding alignment. The experiments demonstrated that entmax models produce more diverse, less repetitive, and more engaging text across multiple tasks.

Entmax sampling consistently outperformed traditional methods, reducing repetition by over 20%, lowering perplexity by 1.5 points, and increasing n-gram diversity to human-like levels. The new metrics correlated strongly with qualitative improvements, validating the approach. Human judges rated stories generated with entmax as 15% more engaging and coherent. Ablation results confirmed that training with entmax loss and decoding with entmax sampling is crucial for optimal performance. The model dynamically adjusts sparsity, leading to more natural and varied outputs, especially in long-form storytelling.

The proposed method is well-suited for creative writing, dialogue systems, and content generation where diversity and coherence are critical. It enables controllable generation by adjusting sparsity parameters, facilitating personalized and context-aware outputs. The approach can be integrated into chatbots, virtual assistants, and AI storytelling tools, providing more human-like interactions. Long-term, this framework could support multi-modal generation, multi-lingual adaptation, and reinforcement learning-based control, transforming AI-driven content creation and human-AI collaboration.

Despite its advantages, the model's computational complexity is higher than softmax, especially for large vocabularies. Extreme sparsity may lead to overly conservative outputs, limiting expressiveness. The approach has been validated mainly on English datasets; cross-lingual and multi-task scenarios need further exploration. Tuning the α parameter for different tasks can be challenging, affecting ease of deployment. Future work should focus on efficiency improvements, broader language applicability, and better control mechanisms for targeted generation.