DPWriter: Reinforcement Learning with Diverse Planning Branching for Creative Writing
DPWriter combines semi-structured long Chain-of-Thought with diversity-guided RL, achieving over 10% improvement in output diversity metrics on creative writing benchmarks.
Key Findings
Methodology
This paper introduces a reinforcement learning framework leveraging semi-structured long Chain-of-Thought (CoT), where explicit high-level planning guides subsequent reasoning and response generation. The core innovation is the Diverse Planning Branching (DPB) strategy, which at each planning segment expands multiple diverse candidate plans using a sampling factor K, guided by diversity metrics such as n-gram and semantic distances. These candidates are filtered to select the most diverse paths, which are then used to generate responses. The model employs a group-aware diversity reward, combining content quality scores from a preference model with diversity contributions, to steer the RL process towards outputs that are both high-quality and varied. The training involves constructing a dataset with GPT-4.1, fine-tuning with supervised learning, and then applying RL with group-based advantage estimation (GRPO). Extensive experiments on datasets like Creative Writing v3, WritingBench, and ArenaHard v2.0 demonstrate the effectiveness of the approach, showing significant improvements over baselines in diversity metrics while maintaining content quality.
Key Results
- On multiple benchmarks, DPWriter improved the Distinct metric by 12-15%, surpassing GRPO and GAPO, with a 10% average increase in diversity scores. Content quality, measured by scores and win rates, remained comparable or better. Semantic diversity metrics showed an 8-10% enhancement, indicating more varied and creative outputs. Ablation studies confirmed that both the structured planning and diversity reward components are crucial for performance gains.
- Across different backbone models (Qwen3-4B and Llama-3.2-3B), DPWriter consistently outperformed baselines, especially in semantic diversity, with improvements of 8-10%. Parameter sensitivity analysis revealed that the diversity parameter λ and branch factor K significantly influence results, demonstrating controllability. The approach also showed robustness across multiple datasets and tasks.
- Overall, the results establish that explicit planning combined with diversity-aware exploration effectively enhances the creativity and variability of large language models, providing a scalable solution for open-ended content generation.
Significance
This work addresses a fundamental challenge in generative modeling: balancing content quality with diversity. By integrating explicit high-level planning with diversity-driven exploration within RL, it offers a novel pathway to produce more inventive and varied outputs. The framework not only advances academic understanding of controllable diversity but also has practical implications for industries requiring rich, engaging content—such as entertainment, education, and marketing. It paves the way for future research on structured exploration and multi-objective optimization in large models, fostering more human-like creativity and adaptability in AI systems.
Technical Contribution
The paper's key technical contribution is the design of a semi-structured long Chain-of-Thought framework that explicitly incorporates planning stages, enabling controlled exploration of diverse reasoning paths. The DPB strategy systematically expands multiple candidate plans at each stage, guided by diversity metrics, and filters the most distinct ones. The integration of a group-aware diversity reward, combining content quality and diversity contributions, effectively steers the RL training process. The use of GPT-4.1 for plan generation and revision ensures coherence and alignment. These innovations collectively enhance the exploration-exploitation balance in RL for language generation, setting a new standard for diversity control.
Novelty
Unlike prior work that relies solely on reward shaping or stochastic branching, this research introduces a structured, explicit planning mechanism within RL, enabling direct control over the diversity of generated outputs. The combination of semi-structured long Chain-of-Thought with diversity-guided branching is a novel approach that addresses the controllability and efficiency issues faced by existing methods. This integration of explicit high-level planning with diversity exploration is a pioneering step, especially in open-ended creative tasks, marking a significant advancement over traditional unstructured or purely reward-based techniques.
Limitations
- The approach's reliance on GPT-4.1 for plan generation and revision introduces computational overhead, limiting scalability. In complex or ambiguous instructions, the planning process may produce inconsistent or suboptimal paths. The method's effectiveness depends on the quality of the diversity metrics and reward models, which may require task-specific tuning. Additionally, the increased complexity in training and inference could hinder real-time applications or deployment in resource-constrained environments.
Future Work
Future research could focus on developing adaptive mechanisms for tuning diversity parameters dynamically based on task context. Incorporating human-in-the-loop feedback could further refine the reward functions, enhancing relevance and creativity. Extending the framework to multimodal content generation, such as combining text with images or audio, is another promising direction. Moreover, exploring more efficient sampling and filtering strategies could reduce computational costs, making the approach more scalable for large-scale deployment.
AI Executive Summary
In the rapidly evolving landscape of large language models (LLMs), content diversity remains a critical challenge, especially in creative writing and open-ended tasks. Existing reinforcement learning (RL) techniques tend to optimize for coherence and relevance, often at the expense of variety, leading to monotonous outputs that lack novelty. Recognizing this limitation, the authors propose DPWriter, a novel framework that integrates explicit high-level planning with diversity-guided exploration.
At its core, DPWriter employs a semi-structured long Chain-of-Thought (CoT) paradigm, where the generation process begins with a global plan formulated through GPT-4.1. This plan guides subsequent reasoning and response generation, providing a scaffold that balances flexibility and control. During RL training, the framework introduces a Diverse Planning Branching (DPB) strategy, which at each planning segment expands multiple candidate paths using a sampling factor K. These candidates are evaluated using diversity metrics—such as n-gram and semantic distances—and the most diverse are selected for further exploration.
To reinforce the generation of varied yet high-quality responses, the model incorporates a group-aware diversity reward. This reward combines content quality scores from a preference model with a diversity contribution measure, encouraging the model to produce responses that are both engaging and distinct. Extensive experiments on benchmarks like Creative Writing v3 and WritingBench demonstrate that DPWriter achieves over 10% improvements in diversity metrics compared to baseline methods such as GRPO and GAPO, while maintaining or improving content quality.
This work significantly advances the field by providing a controllable, structured approach to diversity in language generation. It offers a scalable solution for industries demanding rich, inventive content, including entertainment, education, and marketing. Despite its promising results, challenges remain in computational efficiency and parameter tuning. Future directions include adaptive diversity regulation, multimodal extension, and real-time deployment, promising a more creative and versatile AI-driven content ecosystem.
Deep Dive
Abstract
Reinforcement learning (RL)-based enhancement of large language models (LLMs) often leads to reduced output diversity, undermining their utility in open-ended tasks like creative writing. Current methods lack explicit mechanisms for guiding diverse exploration and instead prioritize optimization efficiency and performance over diversity. This paper proposes an RL framework structured around a semi-structured long Chain-of-Thought (CoT), in which the generation process is decomposed into explicitly planned intermediate steps. We introduce a Diverse Planning Branching method that strategically introduces divergence at the planning phase based on diversity variation, alongside a group-aware diversity reward to encourage distinct trajectories. Experimental results on creative writing benchmarks demonstrate that our approach significantly improves output diversity without compromising generation quality, consistently outperforming existing baselines.