Triviality Corrected Endogenous Reward
Proposes TCER (Triviality Corrected Endogenous Reward), leveraging relative information gain to mitigate bias in open-ended text generation, improving diversity and quality.
Key Findings
Methodology
This work draws inspiration from unsupervised reinforcement learning in mathematical reasoning, introducing a reward based on the relative information gain between a specialist policy (fine-tuned on high-quality data) and a generalist reference policy. The core component, TCER, computes the token-level log-likelihood ratio ϕ(yi|si) = log πs(yi|si) - log πb(yi|si), representing the information gain. To address the triviality bias—where models favor high-confidence, low-diversity outputs—a probability-dependent gating weight w(yi|si) = (1 - πs(yi|si))^λ is applied, suppressing overly confident tokens. The reward combines the log probability with the weighted information gain, ensuring exploration of less predictable, domain-specific tokens. Optimization employs GRPO (Reference Augmented Group Relative Policy Optimization), which samples multiple outputs, incorporates reference answers, and balances exploration and exploitation. Extensive experiments across writing and mathematical reasoning benchmarks validate the effectiveness of TCER in enhancing content diversity and quality without external supervision.
Key Results
- Across multiple writing benchmarks (LongBench-Write, HelloBench, WritingBench), TCER consistently outperforms SFT and endogenous reward methods, with improvements of 4-6 points on average, reaching scores up to 82.2 on DeepWriting-8B. In mathematical reasoning tasks (AIME, AMC), TCER yields 3-4% higher accuracy than baselines, demonstrating robust cross-domain generalization. Ablation studies confirm the importance of the probability-dependent gating and reference augmentation components. Sentence-level analysis shows TCER better rewards high-quality, diverse content, reducing trivial, high-confidence outputs. Results are consistent across models like Qwen and Llama, indicating broad applicability.
- Result 2: The method effectively balances exploration and exploitation, leading to richer, more diverse outputs. The ablation studies reveal that removing the gating weight or reference augmentation significantly degrades performance, underscoring their critical roles. The cross-model experiments demonstrate that TCER's benefits are not model-specific, confirming its generality. Quantitative metrics, such as recall of high-quality sentences and average reward scores, further support its capacity to promote content richness while maintaining coherence.
- Result 3: The approach also improves downstream tasks like creative writing and complex reasoning, with notable gains in content diversity metrics. The sentence-level reward analysis shows TCER assigns higher scores to high-quality sentences identified by human judges, confirming its alignment with content quality. Overall, TCER advances the state-of-the-art in unsupervised reward design for open-ended generation, offering a scalable, cost-effective alternative to external evaluators.
Significance
This research addresses a fundamental challenge in AI content generation—balancing content quality and diversity without relying on costly external evaluators. By framing reward design through information theory, it provides a rigorous, theoretically grounded approach that enhances exploration and mitigates bias. The method's model-agnostic nature facilitates broad adoption across various architectures and tasks, including creative writing and mathematical reasoning. It paves the way for scalable, self-supervised reinforcement learning, reducing dependency on annotated data or proprietary judge models. The broader impact lies in enabling AI systems to generate more creative, reliable, and diverse content, which is crucial for applications in education, entertainment, and scientific research. Overall, TCER offers a significant step toward autonomous, high-quality AI content creation.
Technical Contribution
The core technical innovation is the formulation of a reward based on the relative information gain between a specialist and a generalist policy, modulated by a probability-dependent gating weight. This approach effectively counters the triviality bias by encouraging the model to explore less probable, domain-specific tokens, thus maintaining diversity. The theoretical underpinning leverages information theory, providing bounds and stability guarantees for the reward function. The integration of reference augmentation within GRPO enhances exploration stability and aligns model outputs with high-quality references. Compared to prior methods relying solely on confidence scores or external evaluators, TCER offers a principled, scalable solution with rigorous theoretical support, enabling unsupervised reinforcement learning in complex, open-ended tasks.
Novelty
This work is the first to systematically incorporate relative information gain as a reward signal in open-ended text generation, explicitly addressing the bias towards high-confidence, low-diversity outputs. Unlike existing confidence-based or external evaluation methods, TCER introduces a probability-dependent gating mechanism that dynamically suppresses trivial outputs while rewarding domain-specific, informative tokens. The theoretical analysis grounded in information theory provides new insights into reward design, establishing bounds and stability. Its successful application across diverse tasks and models demonstrates a novel, generalizable framework that advances unsupervised reinforcement learning in creative AI tasks.
Limitations
- The method relies on the quality of the initial expert policy; poor initial policies may limit exploration effectiveness.
- Parameter tuning for λ and k remains manual, lacking an adaptive mechanism, which could affect performance in different tasks.
- In highly out-of-distribution scenarios, the bias mitigation might be insufficient, requiring further robustness improvements.
Future Work
Future research will focus on developing adaptive parameter tuning strategies, possibly via meta-learning. Extending the framework to multi-modal and multi-task settings can broaden its applicability. Investigating automatic mechanisms for policy initialization and reference selection will enhance robustness. Additionally, integrating this approach with large-scale pretraining pipelines could further reduce costs and improve generalization, pushing toward more autonomous, creative AI systems.
AI Executive Summary
Deep Dive
Plain Language Accessible to non-experts
想象你在教一个学生画画，老师希望他画出丰富多彩的作品，而不是只用几种颜色。传统的方法可能只鼓励他用最常用的颜色，结果画面变得单调。现在，老师设计了一套新规则：每当学生尝试用少见的颜色或不同的画法时，就会得到奖励。这样，学生会更愿意尝试新东西，而不是只用熟悉的套路。这个规则就像给AI写作或推理设定的奖励机制，让它在生成内容时，不仅关注高概率的“安全答案”，还会探索那些少见但更有趣、更丰富的表达。通过这种方式，内容变得更有创意、更具多样性，就像画作变得色彩斑斓一样。它帮助AI变得更聪明、更有趣，也能写出更精彩的故事或解决更复杂的问题。
ELI14 Explained like you're 14
想象你在写作文或者玩游戏，很多时候你会选择用最简单、最安全的办法，因为这样不容易出错。但这样写出来的内容就会很无聊，没有新意。这个研究就像给你设计了一套新规则，让你在写作时，不仅看你用的词多不多，还会奖励那些用新词、新句子的人。它会比较你写的内容和一些高手写的内容，鼓励你尝试不同的表达方式，而不是只用最常见的句子。这样，你的故事会变得更有趣、更丰富，也更有创意。用这个方法，AI在写作或者推理时，也会变得更聪明，能写出更精彩、更有新意的内容，就像你变成了一个更厉害的作家一样！
Abstract
Reinforcement learning for open-ended text generation is constrained by the lack of verifiable rewards, necessitating reliance on judge models that require either annotated data or powerful closed-source models. Inspired by recent work on unsupervised reinforcement learning for mathematical reasoning using confidence-based endogenous rewards, we investigate whether this principle can be adapted to open-ended writing tasks. We find that directly applying confidence rewards leads to Triviality Bias: the policy collapses toward high-probability outputs, reducing diversity and meaningful content. We propose TCER (Triviality Corrected Endogenous Reward), which addresses this bias by rewarding the relative information gain between a specialist policy and a generalist reference policy, modulated by a probability-dependent correction mechanism. Across multiple writing benchmarks and model architectures, TCER achieves consistent improvements without external supervision. Furthermore, TCER also transfers effectively to mathematical reasoning, validating the generality of our approach across different generation tasks.