DirectorBench: Diagnosing Long-Form Video Generation with Personalized Multi-Agent Evaluation
DirectorBench uses personalized multi-agent diagnosis and finds cross-shot transitions average only 0.256.
Key Findings
Methodology
DirectorBench combines 80 structured metadata entries, seven user profiles, and 40 checkpoints spanning script, visual, audio, cross-modal, and stability dimensions. A vision-language model builds a content profile and activates only applicable criteria. A LangGraph DAG orchestrates ScriptEvalAgent, VideoEvalAgent, AudioEvalAgent, StabilityEvalAgent, and CrossModalEvalAgent, grounded by ffmpeg, PySceneDetect, OpenCV, MobileViCLIP-Small, Librosa, and ASR evidence.
Key Results
- The shared bottleneck is between-unit quality: transition quality averages 0.256 and reaches only 0.356 for the best workflow, while temporal coherence is 0.405 and video–audio consistency 0.416. In contrast, user-demand fulfillment reaches 0.71, suggesting that local fidelity and prompt compliance have advanced further than long-range continuity.
- Dreamina is strong in prompt compliance and structural control but scores 0.22 on transitions. Kling achieves 0.71 text–video consistency through a shared canvas. ViMax reaches only 0.18 temporal coherence, while MovieAgent reaches only 0.08 user-requirement consistency, revealing distinct architectural failure signatures.
- Across video types, Action is hardest at 0.455 overall, whereas Cinematic/Shot-Design reaches 0.509. Human evaluation with 14 annotators indicates that DirectorBench captures perceptible dimension-level differences rather than merely producing opaque aggregate scores.
Significance
The work changes long-form evaluation from ranking systems by one number to identifying which checkpoint fails and where intervention is needed. This addresses persistent problems in minute-scale generation: cross-shot continuity, audiovisual coordination, narrative structure, and user-dependent trade-offs. For industry, the reports support workflow selection, quality gates, and human review. For research, they establish between-shot and profile-aware evaluation as first-class concerns.
Technical Contribution
DirectorBench contributes dynamic checkpoint routing, auditable tool traces, confidence-aware aggregation, and profile-weighted scoring. Normalized checkpoint scores produce Sd=Σ(srcr)/Σcr, while user-aware quality is Soverall=ΣwdSd/Σwd. The LangGraph DAG executes specialist agents in parallel, performs sequential cross-modal verification, and synthesizes bottlenecks, low-confidence flags, radar data, explanations, and actionable recommendations.
Novelty
Unlike benchmarks centered on short clips, frame quality, temporal smoothness, or generic prompt alignment, DirectorBench explicitly separates objective creative intent from subjective user weighting. Its fundamental novelty is the integration of dynamic applicability, tool-grounded evidence, confidence calibration, and personalized diagnosis for agentic long-form production workflows.
Limitations
- The evaluator uses gpt-4o as its default backend, creating potential single-model bias. Proxy measures, such as mouth-motion/audio-energy correlation for lip synchronization, may not fully represent perceptual synchronization.
- The reported study covers four workflows, six base LLMs, and seven profiles; the supplied text does not include complete RQ2 results or broad ablations, limiting claims about model replacement and generalization.
Future Work
Future work should expand languages, cultures, genres, and user populations; calibrate multiple evaluator models against larger human panels; and study explicit global-state propagation, post-generation self-revision, and transition-specific training. It should also quantify cost, latency, tool failures, and whether diagnostic feedback can be automatically consumed by generation agents.
AI Executive Summary
Long-form video generation is moving from attractive short clips to minute-scale, multi-shot narratives. Yet existing benchmarks mostly measure local visual quality, short-horizon smoothness, or generic prompt alignment. They rarely explain why a sequence breaks between shots, how audio interacts with images, or why the same output may satisfy one user and disappoint another. DirectorBench addresses this gap by separating objective production intent from user-dependent priorities.
The benchmark uses 80 structured metadata specifications, seven user profiles, and 40 checkpoints across script, visual, audio, cross-modal, and stability dimensions. A vision-language model first constructs an 18-attribute content profile and activates relevant criteria. A LangGraph DAG then runs specialist agents in parallel, followed by CrossModalEvalAgent verification. Evidence from ffmpeg, PySceneDetect, OpenCV, MobileViCLIP-Small, Librosa, and ASR is logged for auditability. Scores are confidence-weighted and profile-weighted, while reports identify bottlenecks instead of hiding them in one aggregate number.
Across MovieAgent, ViMax, Dreamina, and Kling, transition quality averages only 0.256 and peaks at 0.356; temporal coherence is 0.405 and video–audio consistency 0.416. User-demand fulfillment is substantially higher at 0.71. Dreamina scores 0.22 on transitions, ViMax 0.18 on temporal coherence, MovieAgent 0.08 on user-requirement consistency, and Kling 0.71 on text–video consistency. Results across Action and Cinematic/Shot-Design videos, plus validation by 14 annotators, show that long-form generation’s frontier is no longer merely realistic frames: it is sustained global coherence under heterogeneous user expectations. The framework’s main promise is an evaluation loop that can tell developers what to repair, not just which system wins.
Deep Analysis
Background
Text-to-video models have improved short-clip fidelity, motion realism, and controllability. Agentic systems such as MovieAgent and ViMax add planning, storyboarding, and composition, while Dreamina and Kling represent production-oriented workflows. However, common benchmarks remain dominated by frame-level quality, short temporal consistency, and generic alignment, underrepresenting narrative continuity, audio, cross-modal timing, and personalization.
Core Problem
A long video is not a sum of independent good shots. Characters, settings, lighting, motion, dialogue, and music may become inconsistent across boundaries. Moreover, users weight narrative, cinematography, emotion, and synchronization differently. A single universal score therefore both obscures distributed workflow failures and imposes an implicit average user.
Innovation
- �� Metadata specifies verifiable creative intent; profiles specify weights, hard constraints, taste, expertise, and tone.
- �� An 18-attribute content profile dynamically activates applicable criteria among 40 checkpoints.
- �� Specialist agents, tool traces, confidence calibration, and cross-modal verification produce auditable diagnosis.
- �� Profile-aware aggregation exposes trade-offs that generic scoring collapses.
Methodology
- �� Input: 80 JSON specifications encode three-act structure, shot scripts, camera directions, audio requirements, and consistency constraints; seven profiles define priorities and hard constraints.
- �� Prompting: an LLM maps (m,u) to a natural personalized creation directive while preserving information content.
- �� Preprocessing: ffprobe/ffmpeg, PySceneDetect, OpenCV boundary measures, optical flow, audio separation, and ASR extract evidence.
- �� Evaluation: four specialist agents run in parallel; CrossModalEvalAgent checks text–video and video–audio relations. Ordinal 1–5 scores are normalized to [0,1].
- �� Synthesis: confidence-weighted dimension scores and profile-weighted overall scores yield bottleneck lists and recommendations.
Experiments
The study evaluates MovieAgent, ViMax, Dreamina Creation Agent, and Kling Canvas Agent, plus six base LLMs and seven profiles. The assessor is fixed, uses gpt-4o and LangGraph, and incorporates MobileViCLIP-Small, Sentence-BERT, Librosa, OpenCV, and a lip-sync proxy. Research questions isolate workflow design, base-LLM choice, and personalized evaluation. Fourteen annotators provide human validation.
Results
The lowest pooled checkpoints are transitions at 0.256, temporal coherence at 0.405, and video–audio consistency at 0.416. Higher scores include user-demand fulfillment at 0.71, text–video consistency at 0.63, and stability at 0.57. Action videos score 0.455 overall, compared with 0.509 for Cinematic/Shot-Design. Workflows exhibit complementary strengths rather than a universal winner.
Applications
Production platforms can use activated checkpoints for pre-release quality control, human-review routing, and automated rework. Developers can identify whether to improve global state, constraint preservation, audio post-processing, or transitions. The framework is relevant to narrative films, advertising, education, action content, and multilingual voiceover.
Limitations & Outlook
The conclusions depend partly on gpt-4o and proxy tools whose outputs are not identical to human perception. Coverage is limited to four workflows and a small profile set, and the supplied paper text does not show complete RQ2 results or systematic ablations. Future studies need larger multilingual samples, independent evaluator calibration, cost and latency analysis, and closed-loop use of diagnosis for generation repair.
Plain Language Accessible to non-experts
Imagine a film factory. A customer brings a detailed order: what the story should contain, what each shot should look like, when voices and music should appear, and which details must never change. Different inspectors represent different customers—one cares most about plot, another about camera style, and another about sound. DirectorBench does not simply stamp the finished film with one grade. It first checks which inspections make sense, then sends the film to several specialists.
The specialists also examine factory records: where shots were cut, how much colors changed, whether movement continued smoothly, and whether speech matched the soundtrack. A final supervisor combines these findings and lists the most urgent repairs. Across four production lines, individual shots were usually more successful than the handoffs between them. Transitions averaged 0.256, with the best line reaching 0.356, while fulfillment of requested content reached 0.71.
The lesson is simple: making several good scenes is not the same as making one coherent movie. DirectorBench is valuable because it identifies the broken handoff instead of merely saying that the whole factory performed poorly.
ELI14 Explained like you're 14
Suppose you ask a game AI to make a one-minute adventure: a cat loses a ball, runs through a park, and finds it. The AI might make three gorgeous scenes, but the cat suddenly changes color, the park moves, and the music ignores the action. Each clip looks fine alone—together, the story feels glitchy!
DirectorBench is like a review squad with different superpowers. It has 80 detailed missions, seven kinds of viewers, and 40 questions. One reviewer checks the story, another the visuals, another sound, and another whether everything matches. It first skips questions that do not fit; a silent landscape should not be judged on lip-sync.
The squad also checks evidence from the video, audio, cuts, movement, and speech. The results are surprising: across four systems, connecting scenes scored only 0.256 on average, and the best system scored 0.356. But following the user’s requested content scored 0.71. So AI is getting better at making a scene and still learning how to connect many scenes into one adventure.
That is like passing every level separately but having the game freeze between levels. The benchmark tells engineers which part to fix—and lets different viewers explain why they prefer different versions. Pretty useful, right?
Glossary
DirectorBench
A diagnostic benchmark for long-form video generation. It combines structured checkpoints, tool evidence, and user profiles rather than relying only on one score.
The paper’s central framework.
Dynamic checkpoint rubric
A rubric that activates only evaluation criteria applicable to a video’s content. It prevents irrelevant tests, such as lip-sync for a character-free landscape.
Driven by an 18-attribute content profile and 40 checkpoints.
Agentic workflow
A multi-stage creation process in which an LLM coordinates planning, storyboarding, generation, tools, and post-processing. Failures can therefore be distributed across components.
Used to compare MovieAgent, ViMax, Dreamina, and Kling.
Cross-modal consistency
The degree to which text, visuals, audio, and their timing agree. It includes semantic correspondence and audiovisual synchronization.
Evaluated by CrossModalEvalAgent.
Confidence-weighted aggregation
An aggregation method that gives greater influence to checkpoint judgments supported by more reliable and consistent evidence. The paper uses Sd=Σ(srcr)/Σcr.
Used to compute dimension scores.
Open Questions Unanswered questions from this research
- 1 It remains unclear whether different evaluator LLMs would reproduce the alignment observed with 14 human annotators at scale.
- 2 The causal benefit of global-state propagation and post-generation self-revision for improving transitions has not been established.
- 3 Seven profiles may not represent cultural, linguistic, professional, and accessibility-related diversity in real users.
Applications
Immediate Applications
Pre-release video quality gates
Platforms can run dynamic checkpoints before publication, flagging broken transitions, audiovisual misalignment, and omitted requirements. Low-confidence cases can be routed to human reviewers, provided the video is accessible to the preprocessing toolchain.
Workflow debugging
Engineering teams can use failure signatures to target repairs: temporal coherence for ViMax, transition continuity for Dreamina, or requirement preservation for MovieAgent, rather than optimizing an opaque aggregate score.
Long-term Vision
Closed-loop filmmaking agents
Future systems could feed diagnosis reports directly back to planners and generators, rewrite storyboards, propagate global state, and regenerate defective boundaries. This would shift long-form production from one-shot synthesis toward controlled iterative creation.
Abstract
Long-form video generation is rapidly moving from short, single-scene synthesis toward minute-long, multi-shot creation with narrative structure, cinematic control, audio, and cross-modal synchronization. However, evaluating such videos remains challenging, since existing benchmarks largely focus on local visual quality, short-horizon temporal consistency, or generic prompt alignment, and provide limited diagnosis of workflow failures and user-dependent preferences. We introduce DirectorBench, a personalized multi-agent diagnostic benchmark for long-form video generation. DirectorBench evaluates generated videos with respect to 80 structured metadata entries, 7 user profiles, and 40 checkpoint criteria across 5 dimensions: script, visual, audio, cross-modal, and stability. Instead of reducing quality to a single aggregate score, DirectorBench localizes checkpoint-level bottlenecks and supports profile-aware evaluation. We evaluate 4 long-form video generation workflows, 6 base LLMs, and 7 user profiles. Across workflows, DirectorBench reveals a between-unit bottleneck: transition quality averages only 0.256 and reaches 0.356 for the best workflow, while prompt-level user demand fulfillment averages 0.71. We further conduct human evaluation with 14 annotators to validate the alignment between DirectorBench and human judgment. The results show that DirectorBench captures human-perceptible quality differences and reveals workflow- and profile-dependent failure modes that are hidden by aggregate scoring. These findings highlight the importance of diagnostic and profile-aware benchmarking for long-form video generation.