Can LLM Agents Stick to the Script? A Benchmark for Long-Horizon Consistency in Interactive Narratives
Introduces NCP-Bench, evaluating LLMs' long-horizon story consistency; GPT-5.2 achieves only 42% survival after 20 turns.
Key Findings
Methodology
This paper proposes the Narrative Commitment Preservation (NCP) framework, integrating 100 movie synopses to build NCP-Bench. It employs structured story specifications—trajectory, commitments, and initial facts—and automated conflict detection, fact updating, and commitment verification to monitor model responses across multiple turns. The system simulates adversarial interventions to test model robustness, ensuring responses adhere to logical constraints. The evaluation pipeline automates conflict checks, fact updates, trajectory progress, and commitment satisfaction, providing a comprehensive measure of long-term story coherence.
Key Results
- GPT-5.2 maintains only 42% survival rate after 20 turns, with rapid decline as interaction deepens, approaching zero before 100 turns. Most models fail to satisfy all narrative commitments within 100 turns, with fact conflict rates between 40% and 68%, highlighting significant gaps in current long-horizon consistency.
- Model performance varies widely under adversarial interventions; despite high linguistic fluency, models struggle to preserve story logic, revealing a fundamental limitation in current LLM architectures for sustained narrative coherence.
- The structured evaluation framework quantifies long-term commitment preservation, setting a new standard for assessing interactive storytelling systems and guiding future model improvements.
Significance
This work underscores the importance of long-horizon logical consistency in interactive narrative systems, revealing that current LLMs, despite impressive language capabilities, often fail to maintain story integrity over extended interactions. The NCP-Bench provides a standardized, reproducible platform for benchmarking and advancing research in this domain, addressing a critical bottleneck in deploying AI for games, virtual characters, and educational tools. By focusing on commitment preservation, the study shifts the paradigm from superficial fluency to deep logical coherence, fostering more reliable and immersive AI-driven storytelling.
Technical Contribution
The paper introduces the NCP framework, combining structured story specifications with automated conflict detection and validation mechanisms. It formalizes the problem of long-horizon story consistency as a measurable task, integrating trajectory, commitments, and facts into a unified evaluation pipeline. The creation of NCP-Bench, a diverse dataset derived from movie synopses, enables standardized benchmarking. The approach also incorporates adversarial testing to evaluate model robustness, providing insights into failure modes and guiding future architectural enhancements.
Novelty
This is the first systematic effort to formalize and quantify narrative commitment preservation in multi-turn interactive systems. Unlike prior works focusing solely on language fluency or superficial coherence, this study emphasizes logical consistency and adherence to story constraints. The conversion of movie synopses into structured, testable environments and the automation of conflict and commitment checks represent significant innovations, establishing a new benchmark for long-term narrative evaluation.
Limitations
- Models exhibit fragility under adversarial interventions, indicating that current architectures lack robustness for complex, unpredictable user behaviors.
- The evaluation relies on predefined story specifications, which may limit creative flexibility and adaptation to real-world scenarios.
- The dataset is based on movie synopses, which may not fully capture the diversity and complexity of real interactive narratives. Further research is needed to extend the framework to more dynamic and multimodal environments.
Future Work
Future directions include developing more resilient model architectures capable of maintaining long-term coherence, integrating multimodal data (images, audio), and expanding the dataset to include more diverse and complex narratives. Additionally, refining automatic conflict detection and exploring learning-based methods for dynamic story management will be crucial. The goal is to build AI systems that can reliably generate and sustain coherent stories over extended interactions, ultimately enabling more immersive and trustworthy virtual environments.
AI Executive Summary
The rapid development of large language models (LLMs) has opened new horizons for interactive storytelling, yet maintaining story coherence over long interactions remains a significant challenge. Existing models excel at generating fluent text but often falter when it comes to preserving logical consistency and narrative commitments across multiple turns, especially under adversarial user interventions. This gap hampers the deployment of AI-driven narrative systems in real-world applications like games, virtual assistants, and educational platforms.
To address this, the authors introduce the Narrative Commitment Preservation (NCP) framework, which formalizes the problem of long-horizon story consistency as a structured, measurable task. Central to this framework is NCP-Bench, a benchmark composed of 100 movie synopses transformed into interactive environments with explicit story specifications, including trajectories, commitments, and initial facts. The benchmark employs automated conflict detection, fact updating, and commitment verification mechanisms to evaluate model responses during multi-turn interactions.
Experimental results reveal a stark performance gap: even the most advanced GPT-5.2 model achieves only a 42% survival rate after 20 turns, with most models failing to satisfy all narrative commitments within 100 turns. The high fact conflict rates (40-68%) highlight the difficulty current LLMs face in maintaining story coherence under adversarial conditions. These findings underscore the need for more robust architectures and evaluation standards focused on logical consistency.
This research provides a standardized platform for benchmarking long-term story coherence, offering valuable insights for both academia and industry. It emphasizes that fluent language generation alone is insufficient for reliable interactive narratives; logical integrity and commitment adherence are equally crucial. Future work will explore integrating multimodal data, enhancing model robustness, and expanding narrative diversity, aiming to create AI systems capable of sustaining coherent, engaging stories over extended interactions. Despite current limitations, this work marks a significant step toward trustworthy, long-horizon AI storytelling.
Deep Dive
Abstract
The rapid advancement of Large Language Models (LLMs) is revolutionizing AI for Games by enabling open-ended and fluid interactive storytelling. However, existing research has largely overlooked the critical challenge of maintaining long-horizon logical consistency and narrative integrity against unconstrained user interventions. To address this, we formulate this challenge as Narrative Commitment Preservation (NCP), and take interactive narrative as our testbed. We introduce NCP-Bench, a benchmark of 100 narrative environments derived from movie synopses. Each environment includes a structured narrative specification (trajectory, commitments, and initial facts) that we can automatically check throughout the interaction between the player agent and the narrator agent. Experiments across state-of-the-art LLMs reveal a substantial long-horizon consistency gap: high linguistic quality does not guarantee commitment preservation; even strong models frequently generate logically conflicting content under adversarial interventions, with the best-performing model (GPT-5.2) achieving only 42% survival rate after 20 turns and fact conflict rates ranging from 40% to 68% across models, and only isolated runs satisfying all achievement commitments within the 100-turn limit.