OneStory: Coherent Multi-Shot Video Generation with Adaptive Memory
OneStory uses adaptive memory for multi-shot video, reaching 0.5813 T2MSV coherence.
Key Findings
Methodology
OneStory reformulates multi-shot video generation as next-shot prediction and fine-tunes the pretrained Wan2.1 I2V model. Frame Selection queries latent frames from all previous shots using the current caption, scores relevance, and selects Top-K frames. Adaptive Conditioner then applies importance-guided patchification with different compression ratios, concatenating compact context tokens with noisy tokens for direct DiT conditioning.
Key Results
- On T2MSV, OneStory reaches 0.5813 average inter-shot coherence, above StoryDiffusion+Wan2.1 at 0.5657 and Mask2DiT at 0.5446. Its character and environment consistency scores are 0.5874 and 0.5752.
- On I2MSV, OneStory achieves 0.5784 average inter-shot coherence, with character consistency 0.5851 and environment consistency 0.5716, exceeding Mask2DiT's 0.5449 average.
- Fine-tuned on approximately 60K videos, the model generates minute-long ten-shot stories. The corpus contains 50K two-shot and 10K three-shot examples, supported by shot inflation and progressive coupling.
Significance
The work addresses a persistent bottleneck in narrative video generation: preserving characters, environments, and story continuity across discontinuous shots. It extends a strong I2V model into text- and image-conditioned multi-shot synthesis without requiring a new foundation model. This has immediate relevance to storyboarding, advertising, education, interactive media, and cinematic previsualization.
Technical Contribution
The central contribution is content-relevant rather than temporally local memory. Frame Selection uses caption-conditioned queries, visual-memory attention, and frame-level scoring to retrieve evidence from arbitrarily earlier shots. Adaptive Conditioner controls token cost through importance-guided patchifiers and injects context directly into the DiT's joint attention stream. Unlike Mask2DiT's fixed window or keyframe pipelines, the design combines long-range access, compact conditioning, and end-to-end optimization.
Novelty
The novelty lies in the selection–compression–injection pipeline, not simply in enlarging an attention window. A later shot can retrieve the first shot when it contains the relevant character or setting, while ignoring an unrelated intervening shot. This content-driven memory differs fundamentally from fixed temporal ordering and single-keyframe conditioning.
Limitations
- The corpus is predominantly human-centric. Although the paper shows out-of-domain examples, reliable performance on animals, objects, fantasy settings, and unusual narrative structures is not systematically established.
- Training is dominated by two- and three-shot sequences. Ten-shot inference relies on autoregressive accumulation, so long stories may suffer from error propagation, selector mistakes, and increasing memory computation.
Future Work
Future work should scale shot-level data, improve memory compression and error recovery, and add editable story controls. Important extensions include audio and dialogue, causal action consistency, cinematic planning, long-context evaluation, and efficient online memory for much longer narratives.
AI Executive Summary
Real-world stories are assembled from discontinuous shots. A protagonist may leave the frame, locations may change, and camera viewpoints may jump, yet identity and narrative logic must persist. Existing systems typically use fixed-window attention or one keyframe per shot: the former forgets early context, while the latter cannot transmit rich narrative evidence.
OneStory reframes generation as autoregressive next-shot synthesis. Frame Selection uses the current caption to retrieve semantically relevant frames from the entire history. Adaptive Conditioner then assigns finer patchification to important frames, compresses the selected memory, and concatenates its tokens with noisy DiT tokens. Initialized from Wan2.1 I2V, it is trained on a curated approximately 60K-shot-sequence corpus.
The model achieves 0.5813 T2MSV inter-shot coherence versus 0.5657 for StoryDiffusion+Wan2.1, and 0.5784 in I2MSV versus 0.5449 for Mask2DiT. It produces minute-long ten-shot videos from text or images. Remaining concerns include human-centric training data, autoregressive error accumulation, and the absence of systematic audio, dialogue, and causal-story evaluation.
Deep Analysis
Background
Single-shot T2V and I2V have advanced rapidly through DiT systems such as Wan, HunyuanVideo, CogVideoX, and Mochi. However, real storytelling requires persistent identities, environments, and event structure across cuts. Fixed-window methods such as Mask2DiT and LCT model only local context, while keyframe pipelines using StoryDiffusion plus I2V provide only one image-level reference per shot.
Core Problem
Given shot-level captions and an optional initial image, the model must generate a coherent sequence while preserving identity and setting but allowing viewpoint, time, location, and action to evolve. Historical relevance is not monotonic with time: an early shot may matter more than the immediately preceding one. Feeding every historical visual token is also computationally expensive.
Innovation
- �� Next-shot formulation reuses pretrained I2V generation.
- �� Frame Selection retrieves Top-K semantically relevant frames from global history.
- �� Adaptive Conditioner applies finer patchifiers to important frames and stronger compression to secondary frames.
- �� Dataset curation uses TransNetV2, two-stage VLM captioning, CLIP, SigLIP2, and DINOv2 filtering.
- �� Training uses three-shot inflation and progressive coupling for stable end-to-end learning.
Methodology
- �� Encoding: a 3D VAE maps each shot to latent features, concatenated into historical memory M.
- �� Querying: learnable queries first attend to caption tokens and then to projected visual memory: Q′=Attn(Q,φT(t),φT(t)); Q″=Attn(Q′,φV(M),φV(M)).
- �� Scoring: frame relevance is S=Mean(φP(M1)Q″ᵀ), followed by Top-K selection.
- �� Compression: selected frames are partitioned by importance and processed by patchifiers Pℓ.
- �� Injection: context C is concatenated with current noise tokens N and processed jointly by the DiT.
- �� Training: two-shot samples are inflated to triplets; warm-up uses uniform conditioning before selector-driven conditioning.
Experiments
The model is initialized from Wan2.1 I2V and trained with AdamW at learning rate 0.0005 and weight decay 0.01 for one epoch on 128 NVIDIA A100 GPUs. Videos are center-cropped to 480×832. The approximately 60K corpus contains 50K two-shot and 10K three-shot videos. Baselines include Mask2DiT, StoryDiffusion+Wan2.1, StoryDiffusion+LTX-Video, and Flux+Wan2.1/LTX-Video. Metrics cover semantic alignment, subject, background, character, environment, aesthetic quality, and dynamics.
Results
For T2MSV, OneStory obtains semantic alignment 0.2389, character consistency 0.5874, environment consistency 0.5752, and average inter-shot coherence 0.5813; StoryDiffusion+Wan2.1 reaches 0.5657. For I2MSV, OneStory reaches 0.5784 average coherence, versus Mask2DiT's 0.5449, with character and environment scores of 0.5851 and 0.5716. Qualitative examples show stronger reappearance, composition, and caption adherence.
Applications
Film and advertising teams can generate coherent storyboards from shot captions or a starting image. Educators, journalists, and social creators can prototype minute-scale narratives without training a new video foundation model. Deployment requires GPU resources, reliable shot-level captions, and content-safety filtering. The unified text/image interface also supports interactive stories and game cinematics.
Limitations & Outlook
The dataset contains mainly short, human-centric sequences, with 50K two-shot and only 10K three-shot examples; longer coherence is obtained by autoregressive rollout. CLIP and DINOv2 pseudo-labels measure visual relevance but may miss causality, intention, or dialogue. The paper does not systematically evaluate audio, speech, temporal planning, or very long contexts. Future systems should add long-range supervision, robust revision, broader domains, and lower-cost memory attention.
Plain Language Accessible to non-experts
Imagine a television director preparing the next scene. A director who can see only the last few scenes may forget the protagonist introduced at the beginning. A director given only one photograph per scene knows what things look like, but not what happened. OneStory keeps a searchable archive of all previous scenes and reads the script for the next one. It then retrieves the most relevant images—for example, the first appearance of a character who must return—rather than blindly using the latest scene.
The archive is compressed intelligently. Important images receive a detailed version, while less important ones become smaller summaries, much like a producer spending more rehearsal time on a crucial actor and less on background extras. The system starts with Wan2.1, which already turns images into videos, and generates scenes one after another. This lets it begin from text or an image and build a coherent ten-scene, roughly one-minute story while keeping memory manageable.
ELI14 Explained like you're 14
Suppose you are making a social-media mini-series. In episode one, a cat floats on a board; in episode three it runs beside the water; later it returns to the beach. A forgetful video AI might accidentally change the cat or scenery because it remembers only the latest episode—or because it uses one screenshot instead of the whole story.
OneStory acts like a smart editor. Before filming the next scene, it reads the scene description and searches the old footage for the most useful evidence. If the script says “the same cat returns,” it looks for the cat and the beach, even if those appeared several scenes ago. Why should memory follow the calendar when the story does not?
It also stores important images in more detail and unimportant ones more compactly, like studying the hard questions carefully and skimming the easy ones. That saves time while preserving the clues that matter.
In the paper, OneStory beats systems such as Mask2DiT in both text-started and image-started generation, reaching 0.5813 and 0.5784 coherence scores. It can create ten shots lasting about a minute. But super-long stories may still accumulate mistakes, and the model needs broader training beyond mostly human videos. Pretty impressive—and definitely not the final boss!
Glossary
Multi-shot Video Generation
Generating multiple discontinuous but semantically connected shots as one narrative video. The objective combines visual quality with identity, environment, and story continuity.
This is the paper's central task, evaluated as T2MSV and I2MSV.
Next-shot Generation
Autoregressively generating the next shot from previous shots and the current caption. It decomposes long-form synthesis into repeated conditional predictions.
OneStory's task reformulation.
Frame Selection
Retrieving historical latent frames that are most relevant to the current caption. Relevance is estimated through caption-conditioned queries and frame-level interactions.
It constructs global historical memory.
Adaptive Conditioner
A module that converts selected frames into compact context tokens using importance-dependent patchifiers. Important frames retain finer spatial detail.
It directly conditions the DiT generator.
Diffusion Transformer
A diffusion denoiser whose backbone is a Transformer. It predicts clean video latents by processing noisy and conditioning tokens with attention.
The OneStory generation backbone.
Shot Inflation
Transforming two-shot examples into three-shot training cases by inserting another shot or augmenting the first shot. It creates a more uniform training format.
Used for stable unified three-shot training.
Open Questions Unanswered questions from this research
- 1 How can a system revise early mistakes across dozens or hundreds of shots? Current rollout lacks explicit backtracking and story-level correction, motivating reversible memory and long-range narrative supervision.
- 2 CLIP and DINOv2 capture visual similarity but not necessarily causality, identity logic, or dialogue relations. Better narrative relevance labels and human-centered evaluation remain necessary.
Applications
Immediate Applications
Film storyboarding
Directors and writers can provide shot captions and an initial frame to prototype coherent visual sequences. The output can expose identity, composition, and transition problems before expensive shooting, while retaining compatibility with existing I2V infrastructure.
Social-media narrative production
Creators can turn a text outline or character image into a multi-shot short, advertisement, or travel story. Practical deployment requires GPU capacity, high-quality shot captions, and safety review, but not a newly trained foundation model.
Long-term Vision
Interactive narrative worlds
Future systems could maintain character, environment, and player-state memory to generate game cinematics, virtual-character stories, and personalized films in real time. Major obstacles are causal consistency, latency, cost, and controllable revision.
Abstract
Storytelling in real-world videos often unfolds through multiple shots -- discontinuous yet semantically connected clips that together convey a coherent narrative. However, existing multi-shot video generation (MSV) methods struggle to effectively model long-range cross-shot context, as they rely on limited temporal windows or single keyframe conditioning, leading to degraded performance under complex narratives. In this work, we propose OneStory, enabling global yet compact cross-shot context modeling for consistent and scalable narrative generation. OneStory reformulates MSV as a next-shot generation task, enabling autoregressive shot synthesis while leveraging pretrained image-to-video (I2V) models for strong visual conditioning. We introduce two key modules: a Frame Selection module that constructs a semantically-relevant global memory based on informative frames from prior shots, and an Adaptive Conditioner that performs importance-guided patchification to generate compact context for direct conditioning. We further curate a high-quality multi-shot dataset with referential captions to mirror real-world storytelling patterns, and design effective training strategies under the next-shot paradigm. Finetuned from a pretrained I2V model on our curated 60K dataset, OneStory achieves state-of-the-art narrative coherence across diverse and complex scenes in both text- and image-conditioned settings, enabling controllable and immersive long-form video storytelling.