StoryAlign: Evaluating and Training Reward Models for Story Generation
StoryAlign evaluates reward models for story generation using StoryRMB; StoryReward achieves 66.3% accuracy on StoryRMB.
Key Findings
Methodology
The paper introduces StoryRMB, a benchmark for evaluating story preference reward models, containing 1,133 high-quality instances. By constructing 100,000 story preference pairs, the StoryReward model is trained to significantly improve human preference capture.
Key Results
- StoryReward achieves 66.3% accuracy on StoryRMB, surpassing larger models.
- In downstream applications, StoryReward excels in BoN selection, choosing stories better aligned with human preferences.
- Experiments show existing reward models perform poorly in selecting human-preferred stories, with the best achieving only 66.3% accuracy.
Significance
This research fills the gap in modeling human preferences in story generation, providing new directions and tools for future research with the introduction of StoryRMB and StoryReward, advancing automated story generation.
Technical Contribution
Introduced the first benchmark focused on story preferences, StoryRMB, and trained StoryReward on a large dataset, significantly enhancing model performance in story preference selection.
Novelty
StoryRMB is the first benchmark focused on story preferences; StoryReward excels in capturing human preferences, outperforming existing larger models.
Limitations
- StoryReward still struggles with complex narrative structures.
- The model's performance in handling multidimensional preferences needs improvement.
Future Work
Future research could explore more complex preference modeling and integrate multimodal data to enhance story generation quality and diversity.
AI Executive Summary
Story generation aims to automatically produce coherent, structured, and engaging narratives. Despite significant advances in text generation by large language models, generated stories still fall short of human-authored works in terms of complex narrative structure and human-aligned preferences. This paper introduces StoryRMB, the first benchmark for assessing reward models on story preferences, containing 1,133 high-quality, human-verified instances. The study finds existing reward models struggle to select human-preferred stories, with the best model achieving only 66.3% accuracy. To address this limitation, the researchers constructed approximately 100,000 high-quality story preference pairs and developed StoryReward, an advanced reward model for story preference trained on this dataset. StoryReward achieves state-of-the-art performance on StoryRMB, outperforming much larger models. The study also adopts StoryReward in downstream test-time scaling applications for BoN story selection and finds that it generally chooses stories better aligned with human preferences. The research team will release their dataset, model, and code to facilitate future research.
Deep Analysis
Background
The field of story generation has advanced significantly with the development of large language models, yet generated stories still lack complexity and creativity compared to human works. Existing reward models focus on general domains, neglecting story preference modeling.
Core Problem
Current large language models lack effective modeling of human story preferences during training, leading to generated stories that diverge from human-authored works in narrative structure and preferences.
Innovation
The paper introduces StoryRMB, the first benchmark focused on story preferences, and trains StoryReward on a large dataset, significantly enhancing model performance in story preference selection.
Methodology
- �� Constructed the StoryRMB benchmark with 1,133 high-quality instances.
- �� Collected 100,000 story preference pairs across diverse domains.
- �� Trained the StoryReward model to improve human preference capture.
Experiments
Experiments used StoryRMB to evaluate existing reward models, finding poor performance in selecting human-preferred stories. BoN experiments validated StoryReward's effectiveness in downstream applications.
Results
StoryReward achieves 66.3% accuracy on StoryRMB, surpassing larger models, and excels in BoN selection, choosing stories better aligned with human preferences.
Applications
StoryReward can be used for automated story generation, improving the quality and alignment of generated stories with human preferences.
Limitations & Outlook
StoryReward still has room for improvement in handling complex narrative structures and multidimensional preferences; future research could explore more complex preference modeling.
Plain Language Accessible to non-experts
Imagine you're in a kitchen cooking. Large language models are like chefs who can make delicious dishes (stories) based on recipes (prompts). However, sometimes these dishes don't match your taste (human preferences). StoryReward acts like a food critic, helping chefs understand your taste preferences to create dishes that better suit your palate. Through continuous tasting and feedback, chefs can improve the flavor of their dishes, ensuring you enjoy a delightful meal every time.
ELI14 Explained like you're 14
Imagine you're playing a game where you need to create a story. Large language models are like characters in the game that can generate stories based on your instructions. But sometimes, these stories don't match your ideas. StoryReward is like a game assistant that helps characters understand your ideas to create stories that better match what you have in mind. Through trial and error, characters can improve the story content, ensuring you enjoy an exciting story adventure every time!
Glossary
Large Language Model (LLM)
An AI model capable of generating natural language text, typically based on deep learning techniques.
Used for generating story text.
Reward Model (RM)
A model used to evaluate the quality of generated content, typically trained with human feedback.
Used to select stories aligned with human preferences.
StoryRMB
A benchmark for evaluating story preference reward models, containing 1,133 high-quality instances.
Used to evaluate existing reward models' performance.
StoryReward
An advanced reward model trained on a large dataset of story preference pairs to enhance human preference capture.
Achieves excellent performance on StoryRMB.
BoN Selection
A method for selecting the best result from multiple generated outcomes, typically used in test-time scaling applications.
Used to validate StoryReward's effectiveness.
Open Questions Unanswered questions from this research
- 1 How to improve StoryReward's performance on complex narrative structures remains an open question.
- 2 Existing models need improvement in multidimensional preference modeling.
Applications
Immediate Applications
Automated Story Generation
StoryReward can be used to enhance the quality and alignment of generated stories with human preferences, suitable for content creation platforms.
Long-term Vision
Multimodal Story Generation
Integrating multimodal data to enhance the diversity and complexity of story generation, advancing the creative industry.
Abstract
Story generation aims to automatically produce coherent, structured, and engaging narratives. Although large language models (LLMs) have significantly advanced text generation, stories generated by LLMs still diverge from human-authored works regarding complex narrative structure and human-aligned preferences. A key reason is the absence of effective modeling of human story preferences, which are inherently subjective and under-explored. In this work, we systematically evaluate the modeling of human story preferences and introduce StoryRMB, the first benchmark for assessing reward models on story preferences. StoryRMB contains $1,133$ high-quality, human-verified instances, each consisting of a prompt, one chosen story, and three rejected stories. We find existing reward models struggle to select human-preferred stories, with the best model achieving only $66.3\%$ accuracy. To address this limitation, we construct roughly $100,000$ high-quality story preference pairs across diverse domains and develop StoryReward, an advanced reward model for story preference trained on this dataset. StoryReward achieves state-of-the-art (SoTA) performance on StoryRMB, outperforming much larger models. We also adopt StoryReward in downstream test-time scaling applications for best-of-n (BoN) story selection and find that it generally chooses stories better aligned with human preferences. We will release our dataset, model, and code to facilitate future research. Related code and data are available at https://github.com/THU-KEG/StoryReward.