AgenticVAU: Multi-Agent Explore-Verify Reasoning for Video Anomaly Understanding
AgenticVAU employs multi-agent explore-verify reasoning, surpassing zero-shot and RL baselines in video anomaly understanding with significant accuracy gains.
Key Findings
Methodology
This paper introduces AgenticVAU, a training-free multi-agent system with four specialized agents—RuleBuilder, SearchPlanner, Observer, and Summarizer—that collaborate via a shared evidence memory (anchor registry). The system begins with global scene analysis to generate scene-aware visual rules, then iteratively explores the video, planning observations based on current evidence. During each cycle, agents select targeted video segments, perform dense or cross-interval observations, and update the evidence store. This process continues until sufficient evidence supports or rejects a candidate anomaly. The final step aggregates all evidence into a comprehensive explanation. The framework relies solely on inference-driven reasoning, avoiding training dependencies, and significantly improves coverage and interpretability of video anomaly understanding.
Key Results
- On VAU-Bench datasets (ECVA, UCF-Crime, MSAD), AgenticVAU outperforms zero-shot and reinforcement fine-tuned baselines across multiple tasks, especially in binary and multi-class anomaly classification. For instance, on UCF-Crime, binary accuracy increased from 62.55% to 94.42%, and multi-class from 57.77% to 85.66%. In VAU-Eval, overall scores improved by over 30%, indicating enhanced evidence organization and reasoning capabilities.
- The multi-round exploration and verification enable cross-temporal evidence comparison and fine-grained validation, addressing limitations of single-agent approaches. Ablation studies confirm role specialization and shared evidence as key factors for performance gains.
- This approach's novelty lies in role-specific agents, shared evidence management, and training-free inference, setting a new paradigm for explainable, flexible video understanding.
Significance
This work advances video anomaly understanding from detection to deep semantic reasoning and causal inference. By distributing perception and reasoning among specialized agents, it overcomes the bottleneck of single-model systems, offering better generalization and interpretability. The training-free nature reduces data dependency, making it suitable for real-world deployment in surveillance, security, and traffic monitoring. Long-term, this framework paves the way for AI systems capable of high-level cognitive video analysis, transforming automated safety and analysis workflows.
Technical Contribution
The core innovation is a role-divided multi-agent architecture with a shared evidence store (anchor registry), enabling iterative exploration and evidence verification without training. The system integrates visual rule generation, multi-scale observation tools, cross-interval comparison, and dynamic planning, resulting in a robust, explainable reasoning process. This design significantly enhances the coverage and depth of anomaly understanding, setting a new standard for inference-driven video analysis.
Novelty
This is the first application of a multi-agent explore-verify framework to video anomaly understanding, emphasizing explicit role separation, shared structured evidence, and inference-based reasoning. Unlike prior single-model or RL-based methods, it leverages multi-agent collaboration for dynamic evidence collection and verification, providing a novel, interpretable approach that generalizes well across unseen scenarios.
Limitations
- The current system may face efficiency challenges in very long videos or multiple simultaneous anomalies, as multiple exploration cycles increase computational load.
- Observation tool parameters and planning strategies are sensitive, requiring further adaptive optimization.
- Multi-modality integration (audio, text) is not yet incorporated, which could further improve robustness and interpretability in complex scenes.
Future Work
Future directions include integrating multi-modal data (audio, text) to enrich evidence and reasoning, developing adaptive observation policies to reduce exploration steps, and enabling autonomous learning of observation strategies. Extending the framework to handle multiple concurrent anomalies and real-time deployment are also promising avenues. Ultimately, the goal is to create a comprehensive, explainable AI system capable of high-level video cognition across diverse real-world scenarios.
AI Executive Summary
Video anomaly understanding (VAU) has traditionally focused on detecting irregular events, often relying on supervised training and limited interpretability. While early methods like reconstruction-based models and weakly supervised techniques improved localization, they lacked semantic depth and causal reasoning. Recent advances incorporated language models and large multimodal systems, but these often still depended on training data and lacked structured reasoning frameworks.
This paper introduces AgenticVAU, a novel, training-free multi-agent framework that models VAU as an explore-verify process. Four specialized agents—RuleBuilder, SearchPlanner, Observer, and Summarizer—collaborate via a shared evidence memory (anchor registry) to dynamically explore video content, generate visual rules, plan targeted observations, and verify anomalies. The system begins with global scene analysis to generate scene-aware rules, then iteratively refines its understanding through dense local and cross-interval observations, updating evidence in a structured manner.
Experimental results on VAU-Bench datasets (ECVA, UCF-Crime, MSAD) demonstrate that AgenticVAU significantly outperforms zero-shot and reinforcement learning baselines. For example, on UCF-Crime, binary accuracy jumps from 62.55% to 94.42%, and multi-class accuracy from 57.77% to 85.66%. The approach excels in anomaly classification, reasoning, and temporal grounding, validating the effectiveness of role-specific agents and shared evidence management.
This work marks a shift towards more interpretable, flexible, and generalizable video understanding systems. By avoiding training dependencies and leveraging structured inference, it addresses key challenges in anomaly comprehension, with promising applications in security, traffic monitoring, and automated surveillance. Future work will focus on multi-modal integration, adaptive observation strategies, and real-time deployment, aiming to realize fully autonomous, high-level video cognition systems.
Deep Dive
Plain Language Accessible to non-experts
想象你在看一场侦探电影，里面有很多线索散布在不同场景。你不会一次性看完所有内容，而是像侦探一样，反复回看某些片段，寻找线索。你还会根据场景背景判断某个行为是否正常，比如在学校跑步很正常，但在图书馆就不合适。你的小伙伴们各自负责不同任务：一个找线索，一个计划去哪里看，一个仔细观察某个片段，最后一个总结所有线索告诉你答案。每次他们都把发现的线索写在一本共享的笔记本里，反复验证，直到找到真相。这个过程不用提前学习，只靠他们的合作和推理，就能像侦探一样逐步揭示视频中的异常事件。
ELI14 Explained like you're 14
想象你在玩一个侦探游戏，你要找出视频中发生了什么奇怪的事情。你可以让不同的朋友帮忙：一个找线索，一个计划去哪里看，一个看具体的片段，最后一个总结所有线索告诉你答案。每次他们都把发现的线索写在一本共享的笔记本里，然后再一起讨论。这样反复找证据、验证线索，直到确定事情的真相。这就像我们用多个“侦探”合作，逐步拼凑出视频中的“秘密”。这个方法不用提前学习，只靠他们的合作和推理，就能找到异常事件，就像侦探们逐步解开谜题一样。
Glossary
Explore-Verify (探索-验证)
一种多智能体合作框架，通过探索和验证逐步确认事件，避免单一模型的局限。
作为核心推理流程。
Anchor Registry (共享证据存储)
存放观察证据的共享空间，组织支持或反对证据，帮助智能体动态推理。
用于证据管理和推理决策。
Visual Rules (视觉规则)
基于场景描述生成的支持或反对事件的视觉条件，用于验证潜在异常。
由规则智能体生成，用于证据验证。
Open Questions Unanswered questions from this research
- 1 如何进一步提升多模态信息融合的效率和效果，尤其在多异常同时出现时的证据组织和推理能力仍需探索。
- 2 未来需要研究智能体自主学习观察策略，减少轮次，提高推理速度。
Abstract
Video anomaly understanding (VAU) focuses on comprehensively interpreting abnormal events in videos, requiring models to identify anomalous occurrences, discover their supporting evidence, and explain the underlying causes beyond simple anomaly detection. Existing VAU methods often rely on specialized training or limited observations, restricting generalization or evidence coverage. Although single-agent alternatives support adaptive video observation, they still integrate exploration, observation, and decision-making within a unified reasoning process, offering limited role specialization and structured evidence coordination. To address these limitations, we present AgenticVAU, a training-free multi-agent framework that casts VAU as an explore--verify process, where the system first discovers potential anomalies and then verifies them through targeted observations. To achieve this, four specialized agents are introduced to handle visual-rule construction, search planning, video observation, and final decision, respectively. These agents communicate through an anchor registry, a shared evidence memory that binds each observation. Guided by this agent framework, AgenticVAU interleaves broad temporal exploration, dense local verification, and cross-interval comparison until sufficient evidence is collected. We conduct extensive experiments on the ECVA, UCF-Crime, and MSAD subsets of VAU-Bench, the results show that AgenticVAU outperforms zero-shot inference and reinforcement learning-based baselines, demonstrating the value of multi-agent collaboration for video anomaly understanding.