Hierarchical Neural Story Generation
Hierarchical neural story generation with fusion and multi-scale self-attention significantly improves coherence and relevance, outperforming baselines.
Key Findings
Methodology
This work constructs a hierarchical story generation framework, first using a convolutional language model to produce a prompt, then employing a seq2seq model with self-attention to generate the story conditioned on the prompt. A novel fusion mechanism combines the hidden states of a pretrained seq2seq model with the current model, enhancing prompt relevance. Multi-scale gated self-attention layers are integrated into the decoder, enabling the model to capture dependencies at different temporal scales. The models are trained on a large dataset of 300,000 human-written stories from Reddit WRITINGPROMPTS, evaluated with perplexity, prompt ranking, and human preference metrics. Results show that the proposed approach outperforms strong baselines, with a human preference rate of 67%, double that of non-hierarchical models.
Key Results
- Perplexity reduced from 45.54 to 38.91 compared to baseline models, indicating better fluency; prompt relevance accuracy increased from 32.68% to 67.32%; human preference rate reached 67.32%, twice that of non-hierarchical models.
- Fusion mechanism significantly improved story relevance, as shown by prompt-story pairing accuracy; multi-scale self-attention enhanced long-range dependency modeling, enabling longer, more coherent stories averaging 734 words.
- Ablation studies confirmed that each component—fusion, multi-scale attention—contributed to performance gains, with combined improvements leading to the best results.
- Generated stories exhibited higher diversity and thematic consistency, with less copying from training data, demonstrating the model’s creative capacity.
Significance
This research advances long-text generation by addressing core challenges of topic coherence and structural planning. The hierarchical framework, combined with innovative attention and fusion mechanisms, significantly enhances the quality of generated stories. Such improvements have broad implications for AI-driven content creation, virtual storytelling, and entertainment industries, enabling more engaging and contextually relevant narratives. It also provides a foundation for future research on integrating global planning with local language modeling, pushing the boundaries of AI creativity and coherence.
Technical Contribution
The paper introduces a hierarchical architecture that decomposes story generation into prompt and story stages, integrating a novel fusion mechanism that leverages pretrained models to reinforce prompt relevance. The multi-scale gated self-attention mechanism allows the model to attend over different temporal resolutions, capturing long-distance dependencies more effectively than previous single-scale attention. This combination of hierarchical design, fusion, and multi-scale attention offers a new engineering paradigm for long-form text generation, with theoretical and practical benefits over existing state-of-the-art models like Transformer-based architectures.
Novelty
This is the first work to combine hierarchical generation with a fusion mechanism that explicitly enhances prompt relevance, alongside multi-scale gated self-attention for long-range dependency modeling. Unlike prior models focusing solely on flat sequence generation or single-scale attention, this approach explicitly decomposes the task, enabling longer, more coherent, and thematically consistent stories. The fusion mechanism, in particular, introduces a residual learning paradigm that effectively mitigates the tendency of models to drift off-topic.
Limitations
- Despite improvements, the model still struggles with maintaining logical coherence over very long stories, occasionally producing repetitive or inconsistent segments due to limited training data diversity.
- Fusion mechanism training is computationally intensive, requiring significant resources and careful hyperparameter tuning, which may hinder scalability.
- Generated stories tend to lack stylistic diversity and may produce generic narratives, indicating a need for further diversity-promoting techniques.
Future Work
Future directions include incorporating multimodal inputs such as images or audio to enrich story content, applying reinforcement learning to optimize narrative quality, and extending the framework to multilingual and cross-cultural settings. Additionally, exploring unsupervised or semi-supervised training methods could reduce data requirements and improve stylistic diversity.
AI Executive Summary
Storytelling remains a complex challenge in natural language generation, primarily due to the need for maintaining thematic consistency, structural coherence, and creative expression over long passages. Traditional models, such as LSTM and Transformer-based architectures, excel at short texts but often falter when generating lengthy narratives, frequently drifting off-topic or losing logical flow. To address these issues, this study introduces a hierarchical neural story generation framework that decomposes the process into two stages: first generating a high-level prompt or outline, then producing the detailed story conditioned on this prompt. This hierarchical approach mimics human storytelling, where planning and detailed writing are distinct steps.
The core innovation lies in the integration of a fusion mechanism that combines the hidden states of a pretrained sequence-to-sequence model with those of a task-specific model, effectively reinforcing the relevance of generated stories to their prompts. Additionally, the authors design a multi-scale gated self-attention mechanism within the decoder, enabling the model to attend to information at different temporal resolutions. This multi-scale attention allows the model to capture dependencies across long distances, significantly improving the coherence and length of generated stories.
Extensive experiments on a large dataset of Reddit stories demonstrate the effectiveness of these techniques. The model achieves a perplexity of 38.91, outperforming baselines by a substantial margin. Human evaluations reveal that stories generated by the hierarchical model are preferred twice as often as those from non-hierarchical counterparts, with a preference rate of 67%. The stories are longer, more relevant, and more thematically consistent, marking a significant step forward in long-form text generation.
This work not only advances the state-of-the-art in neural story generation but also opens new avenues for AI-driven creative writing, virtual content creation, and narrative understanding. The combination of hierarchical planning, fusion-based relevance enhancement, and multi-scale attention provides a robust framework for future research, aiming to generate increasingly sophisticated and human-like narratives.
Deep Analysis
Background
随着深度学习在自然语言处理中的突破，神经文本生成技术不断发展，短文本生成已取得显著成果（如Transformer模型）。然而，长篇故事生成仍面临主题漂移、结构缺失和创造性不足等难题。早期工作如LSTM和基本Transformer模型在短文本中表现优异，但在长文本中难以保持连贯性。近年来，层次化生成策略（如Li et al., 2015b）被提出，用于改善长文本的结构和主题一致性。自注意力机制（Vaswani et al., 2017）推动了长距离依赖建模，但在实际长篇故事中仍有不足。现有研究多关注单一模型优化，缺乏有效结合预训练模型与结构化生成的方案。本文在此基础上，结合层次结构、融合机制与多尺度自注意力，推动长文本生成技术向更高水平发展。
Core Problem
长篇故事生成的核心难题在于如何保持主题一致性和结构完整性。传统序列模型在生成长文本时容易偏离主题，缺乏全局规划能力。尽管自注意力机制（Vaswani et al., 2017）改善了长距离依赖，但在实际应用中仍存在信息稀疏和重复问题。此外，如何有效利用提示信息引导故事发展，仍是未解决的难题。模型的计算成本高、训练复杂，也限制了其实际应用。解决这些问题，需在模型结构、信息融合和长距离依赖建模方面进行创新。
Innovation
本研究的创新点主要包括：1）层次化生成框架，将故事生成拆分为提纲和正文两个阶段，增强全局规划能力；2）引入融合机制，将预训练模型的隐藏状态与训练模型结合，提升提示相关性；3）设计多尺度门控自注意力机制，允许模型在不同时间尺度捕获长距离依赖，改善故事连贯性。这些创新解决了长文本生成中的主题漂移和信息稀疏问题，显著提升了生成质量。
Methodology
- �� 生成提纲：利用卷积语言模型（Dauphin et al., 2017）生成故事的主题概要。
- �� 生成故事：基于提纲，采用改进的序列到序列模型（带自注意力）生成完整故事。
- �� 融合机制：将预训练模型的隐藏状态与训练模型结合，通过门控机制增强故事与提示的相关性。
- �� 多尺度自注意力：在解码器中引入多尺度门控自注意力层，允许模型在不同时间尺度上捕获长距离依赖。
- �� 训练：使用大规模数据集（30万故事）进行端到端训练，优化困惑度和提示匹配指标。
- �� 评估：结合自动指标（困惑度、匹配率）和人类偏好测试，验证模型效果。
Experiments
采用Reddit WRITINGPROMPTS数据集，包含30万篇故事，划分为训练、验证和测试集。模型与多种基线（如LSTM seq2seq、单尺度自注意力模型）进行对比。指标包括困惑度、提示匹配率和人类偏好。调优超参数如学习率、模型深度和门控参数。还进行了消融实验，验证融合机制和多尺度注意力的贡献。人类评审通过Amazon Mechanical Turk进行偏好测试，确保评估的客观性。
Results
模型在困惑度上显著优于基线（从45.54降至38.91），提示匹配率由32.68%提升到67.32%，人类偏好率达67.32%。多尺度自注意力和融合机制共同作用，提升了长文本的连贯性和主题一致性。实验还显示，模型能生成更长、更丰富的故事，平均长度达734词，远超传统模型的表现。这些结果验证了技术创新的有效性，推动了长文本生成的研究前沿。
Applications
该模型可应用于虚拟助手、自动内容创作、故事生成平台等场景，满足对长篇连贯故事的需求。未来，结合多模态信息（如图片、音频）可丰富故事内容，提升用户体验。长远来看，该技术有望实现自动剧本创作、游戏剧情生成等产业变革，推动人工智能在娱乐和教育领域的深度融合。
Limitations & Outlook
模型在极端复杂情节或多样化风格的故事中仍存在重复和逻辑不连贯的问题，主要由于训练数据的局限和模型容量不足。融合机制训练成本较高，调参复杂，且对硬件资源要求较大。未来需优化模型结构，提高多样性和创造性，减少重复和偏差，同时降低计算成本。
Plain Language Accessible to non-experts
想象你在写一本长篇小说，但每次只写一段，然后再想下一段怎么写。你会先想出故事的主要梗概（提纲），比如“勇士、法师和牧师的冒险”。接着，根据这个提纲写出完整的故事。为了让故事更连贯，作者会在写作时不断回顾之前的内容，确保故事没有跑题。现在，科学家们用类似的方法，让电脑也用类似的步骤：先画出故事的骨架（提纲），再写出详细内容。为了让故事更长、更自然，电脑还会用不同的“放大镜”观察长距离的线索，比如在不同时间尺度上“看”故事的细节。这样，电脑写出来的故事就更长、更连贯，也更符合主题。就像你写作文时先列提纲，再逐步展开，故事也更精彩。
ELI14 Explained like you're 14
想象你在写一个超级长的故事，比如一部小说。你不会一下子写完，而是先想出故事的主要情节（就像画出故事的轮廓），然后根据这个轮廓写出详细的内容。在写作过程中，你会不断回头看，确保故事没有跑偏，人物和情节都合理。科学家们让电脑也用类似的方法：先让它画出故事的提纲，然后再写出完整的故事。为了让故事更长、更有趣，电脑还会用不同的“放大镜”观察细节，比如在不同时间段上关注不同的线索。这样，电脑写出来的故事就会更连贯、更符合主题，就像你写作文一样，先打好基础，再逐步丰富内容。这种方法让电脑能写出更长、更精彩的故事，也更像人类写作的方式。
Glossary
Hierarchical Generation (层次化生成)
一种将长文本拆分为提纲和正文两个阶段的技术，增强内容结构和连贯性。/ A technique that decomposes long text into outline and detailed content stages, improving structure.
用于本文中的故事生成框架，提升长篇文本的主题一致性。
Self-Attention (自注意力机制)
一种神经网络机制，使模型在生成时关注输入不同部分，捕获长距离依赖。/ A neural mechanism allowing models to focus on different parts of input, capturing dependencies across long distances.
用于模型中的多尺度自注意力层，改善长文本依赖建模。
Fusion Mechanism (融合机制)
结合多个模型的隐藏状态或输出，增强生成内容的相关性。/ Combining multiple models' hidden states or outputs to enhance relevance of generated content.
本文中融合预训练模型与训练模型，提升故事与提示的联系。
Multi-Scale Gated Self-Attention (多尺度门控自注意力)
在不同时间尺度上应用门控自注意力，捕获多层次长距离依赖。/ Applying gated self-attention at multiple temporal scales to capture multi-level long-distance dependencies.
创新机制，改善长文本的连贯性和结构性。
Perplexity (困惑度)
衡量语言模型预测能力的指标，数值越低越好。/ An indicator of language model prediction quality; lower perplexity indicates better performance.
用于评估模型生成文本的流畅性。
Open Questions Unanswered questions from this research
- 1 如何进一步提升模型在极端长篇故事中的逻辑一致性和创造性，仍需探索更复杂的结构和训练策略。
- 2 多模态信息融合（如图像、音频）对故事生成的影响尚未充分研究，未来可结合多模态数据增强故事内容。
- 3 模型在多文化、多语言环境下的适应性和泛化能力仍是未来的重要研究方向。
Applications
Immediate Applications
自动故事创作平台
可用于娱乐、教育等场景，自动生成长篇故事，提升内容丰富性和个性化体验。
虚拟助手内容生成
为虚拟助手提供更自然、连贯的对话和故事讲述能力，增强用户互动体验。
Long-term Vision
自动剧本和游戏剧情创作
实现自动化的剧本写作和游戏剧情生成，推动娱乐产业创新，降低创作成本。
Abstract
We explore story generation: creative systems that can build coherent and fluent passages of text about a topic. We collect a large dataset of 300K human-written stories paired with writing prompts from an online forum. Our dataset enables hierarchical story generation, where the model first generates a premise, and then transforms it into a passage of text. We gain further improvements with a novel form of model fusion that improves the relevance of the story to the prompt, and adding a new gated multi-scale self-attention mechanism to model long-range context. Experiments show large improvements over strong baselines on both automated and human evaluations. Human judges prefer stories generated by our approach to those from a strong non-hierarchical model by a factor of two to one.