CoAgent: Collaborative Planning and Consistency Agent for Coherent Video Generation
CoAgent employs a plan-synthesize-verify-edit loop with explicit entity memory, achieving significant improvements in long video coherence and visual consistency, with Subject Consistency at 94.70%.
Key Findings
Methodology
CoAgent comprises four core modules: a Storyboard Planner decomposes high-level prompts into structured shot plans with explicit entities, spatial relations, and pacing cues; a Global Context Manager (GCM) maintains entity-level visual representations across shots to ensure identity consistency; a Synthesis Module generates each shot conditioned on the shot plan, entity memory, and previous frame, guided by a Visual Consistency Controller; a Verifier Agent evaluates generated frames using a vision-language model, providing feedback and triggering selective regeneration when inconsistencies are detected. The entire process forms a closed loop, enabling dynamic planning, generation, verification, and refinement, effectively integrating multi-modal reasoning and multi-agent collaboration.
Key Results
- On VBench and VideoScore benchmarks, CoAgent surpasses SOTA models, achieving Subject Consistency of 94.70%, Background Consistency of 96.50%, Temporal Flickering of 99.20%, and Motion Smoothness of 99.40%. Quantitative improvements include a 5.9% increase in Visual Quality and a 7.1% boost in Text-Video Alignment over baseline models like Wan2.1. Ablation studies confirm that adding GCM improves entity consistency by 3.9 points, while the verifier enhances temporal stability by 4.5 points, demonstrating the effectiveness of each component.
- These results indicate that the integration of explicit entity memory and feedback-driven verification effectively reduces identity drift and scene discontinuity, leading to more coherent, style-aligned, and narratively consistent long videos. The framework's ability to dynamically detect and correct inconsistencies significantly advances the state of automated video synthesis.
- Overall, CoAgent establishes a new paradigm for long-form video generation by combining structured planning, persistent entity memory, and closed-loop quality control, setting a foundation for scalable, controllable, and high-fidelity content creation.
Significance
This work addresses longstanding challenges in open-domain long video synthesis, notably identity preservation, scene continuity, and rhythm stability. By integrating structured narrative planning with explicit entity memory and a verification feedback loop, CoAgent offers a robust, scalable solution that bridges the gap between static prompt-based methods and dynamic, multi-agent systems. Its ability to produce coherent, style-consistent videos with minimal manual intervention has profound implications for entertainment, virtual production, and content creation industries. The framework's modular design facilitates future extensions, such as incorporating more sophisticated reasoning modules or real-time editing, paving the way for fully autonomous, high-quality video generation systems. This represents a significant leap toward AI-driven creative workflows, reducing costs and expanding creative possibilities.
Technical Contribution
The core technical innovation of CoAgent lies in its integration of a structured storyboarding process, an explicit entity-level global memory (MGCM), and a verification-driven feedback mechanism within a multi-agent, closed-loop framework. Unlike prior methods that treat video synthesis as a stateless, open-loop process, CoAgent maintains persistent entity representations across shots, enabling consistent identity preservation. The verification module employs a vision-language model to assess semantic and stylistic fidelity, triggering targeted regeneration to correct errors. This combination of explicit memory, dynamic planning, and feedback control introduces a new paradigm for controllable, high-quality long video generation, with theoretical guarantees of improved coherence and stability.
Novelty
This research is the first to unify structured narrative planning, explicit entity memory, and a verification-based closed-loop control into a single, stateful multi-agent system for long video synthesis. Unlike existing approaches that rely solely on prompt conditioning or passive memory, CoAgent actively reasons about content consistency and narrative coherence, enabling dynamic correction and refinement. Its innovative interface design allows seamless information exchange among planning, synthesis, and verification modules, setting a new standard for controllable, scalable, and robust video generation frameworks.
Limitations
- Despite significant progress, the system struggles with highly complex scenes involving multiple interacting entities and rapid scene changes, which can still cause inconsistencies or artifacts.
- The verification relies on pre-trained vision-language models, which may introduce biases or miss subtle semantic errors, limiting robustness in certain scenarios.
- Computational costs remain high, especially for long videos, due to multiple rounds of regeneration and extensive memory management, hindering real-time applications. Future work should focus on efficiency improvements and broader scene understanding.
Future Work
Future research will aim to enhance the scalability and efficiency of CoAgent, including model compression and faster inference strategies. Integrating more advanced reasoning modules, such as multi-modal transformers with better scene understanding, could improve handling of complex interactions. Additionally, incorporating user-in-the-loop controls for personalized content editing and expanding the framework to support real-time generation are promising directions. Exploring unsupervised or weakly supervised training paradigms for the verifier could further improve robustness and reduce reliance on large annotated datasets.
AI Executive Summary
Long-form video generation remains a formidable challenge in AI, primarily due to difficulties in maintaining narrative coherence, visual consistency, and temporal stability over extended sequences. Existing models, often based on diffusion or GAN architectures, generate short clips effectively but falter when scaling to longer videos with recurring characters and evolving scenes. These limitations manifest as identity drift, scene discontinuities, and rhythm inconsistencies, undermining the storytelling experience.
In response, Qinglin Zeng and colleagues introduce CoAgent, a novel framework that redefines video synthesis as a closed-loop, multi-agent process. The system integrates four key modules: a Storyboard Planner that decomposes high-level prompts into structured shot plans, a Global Context Manager (GCM) that maintains entity-level visual representations across shots, a Synthesis Module guided by a Visual Consistency Controller that generates each shot conditioned on memory and previous frames, and a Verifier Agent that evaluates intermediate outputs using vision-language models. This feedback-driven mechanism enables dynamic correction of errors, ensuring both local fidelity and global coherence.
The technical core of CoAgent lies in its explicit entity memory and verification loop. The GCM stores visual embeddings of key entities, allowing the synthesizer to retrieve and preserve identities across shots. The verifier assesses semantic and stylistic consistency, triggering targeted regeneration when discrepancies are detected. This interplay creates a robust, adaptive system capable of producing long videos with high narrative fidelity.
Experimental results on benchmarks like VBench and VideoScore demonstrate that CoAgent outperforms state-of-the-art models, achieving Subject Consistency of 94.70% and Background Consistency of 96.50%. It also significantly improves temporal flickering and motion smoothness, validating the effectiveness of the feedback loop. Ablation studies confirm that each component—GCM and verifier—contributes critically to performance gains.
This work marks a pivotal step toward autonomous, controllable, and high-quality long video generation. Its modular design and innovative integration of planning, memory, and verification mechanisms open new avenues for research and industrial applications, from entertainment to virtual reality. Challenges remain in scaling efficiency and handling highly complex scenes, but the framework sets a strong foundation for future advancements in AI-driven content creation.
Deep Dive
Glossary
实体级记忆 (Entity-Level Memory)
一种存储关键实体视觉表示的机制，用于跨镜头保持身份一致性。技术上为存储实体的视觉特征向量，支持检索和更新。
在论文中，GCM用以确保人物和场景在长视频中的连续性。
闭环验证 (Closed-Loop Verification)
利用模型评估生成内容的语义和风格一致性，并根据反馈调整生成过程的机制。技术上为基于视觉-语言模型的动态质量评估。
用于检测和修正生成中的错误，确保视频连贯性。
结构化故事板 (Structured Storyboard)
将高层叙事拆解为明确的镜头计划，包括文本提示、实体、空间关系和节奏参数。技术上为高层次内容规划工具。
作为生成的基础输入，确保内容的可控性和逻辑性。
视觉一致性控制 (Visual Consistency Controller)
引导生成模型保持风格和动作连贯的机制，通过特征空间相似性调节扩散引导。技术上为辅助生成的调控模块。
确保每个镜头在视觉风格和动作上保持一致。
多智能体系统 (Multi-Agent System)
由多个协作的智能体组成，分别负责规划、生成、验证等任务，实现动态协作。技术上为分布式协同架构。
实现视频生成中的多任务协同与反馈控制。
Open Questions Unanswered questions from this research
- 1 当前模型在极端复杂场景、多角色交互和细节丰富的内容中仍存在生成不稳定的问题，未来需增强多模态理解和场景推理能力。
- 2 验证器的评估依赖预训练视觉-语言模型，可能存在偏差，限制了对微妙语义和风格错误的检测能力。
- 3 系统在大规模长视频生成时计算成本较高，需优化推理速度和效率，以实现实时或近实时应用。
Applications
Immediate Applications
虚拟内容创作
可用于自动生成虚拟主播、虚拟偶像或虚拟场景，减少人工制作成本，提升内容多样性和个性化水平。
影视动画辅助
辅助动画师快速生成长篇动画或影视片段，提升制作效率，支持多场景、多角色的复杂叙事。
Long-term Vision
自主内容生产平台
未来可构建全自动化的视频内容生产系统，实现个性化、实时化的影视、游戏等多媒体内容创作，推动数字娱乐产业变革。
Abstract
Maintaining narrative coherence and visual consistency remains a central challenge in open-domain video generation. Existing text-to-video models often treat each shot independently, resulting in identity drift, scene inconsistency, and unstable temporal structure. We propose CoAgent, a collaborative and closed-loop framework for coherent video generation that formulates the process as a plan-synthesize-verify pipeline. Given a user prompt, style reference, and pacing constraints, a Storyboard Planner decomposes the input into structured shot-level plans with explicit entities, spatial relations, and temporal cues. A Global Context Manager maintains entity-level memory to preserve appearance and identity consistency across shots. Each shot is then generated by a Synthesis Module under the guidance of a Visual Consistency Controller, while a Verifier Agent evaluates intermediate results using vision-language reasoning and triggers selective regeneration when inconsistencies are detected. Finally, a pacing-aware editor refines temporal rhythm and transitions to match the desired narrative flow. Extensive experiments demonstrate that CoAgent significantly improves coherence, visual consistency, and narrative quality in long-form video generation.