Caption-once, Frames-on-Demand: Visual-Need Routing for Budget-Aware Agentic Long Video Understanding
Proposes CFD framework using Visual-Need Routing to optimize long video understanding, reducing visual processing.
Key Findings
Methodology
The study introduces the Caption-once, Frames-on-Demand (CFD) framework, combining edge devices and cloud collaboration. Edge devices perform a one-time offline captioning to create an event-level story skeleton and a clip-level micro-log. The cloud uses a Visual-Need Router to determine if keyframes need to be retrieved, only accessing visuals for perceptual questions.
Key Results
- On the Video-MME benchmark, CFD uses an average of 5.8 frames per question with an overall accuracy of 67.5%, significantly reducing visual processing.
- On InfiniBench, CFD excels in chronological understanding and global appearance, using an average of 14.8 frames per question.
- Experiments show CFD maintains high accuracy while reducing visual processing, especially in long video scenarios.
Significance
The CFD framework is significant in the field of long video understanding, particularly on resource-constrained edge devices. By reducing the need for visual processing, it enhances system efficiency and responsiveness while maintaining accurate video content understanding. This method offers new insights for multimodal applications, especially in scenarios requiring real-time processing and low latency.
Technical Contribution
The technical contribution of the CFD framework lies in its innovative Visual-Need Routing mechanism, which for the first time treats visual access as a query-conditioned cost. This approach not only reduces the overhead of visual processing but also provides a flexible framework to handle different types of questions with high adaptability.
Novelty
The novelty of the CFD framework is in transforming visual access into a tunable query-conditioned cost, a first in video QA systems. This innovation allows visual processing to be selectively allocated based on question type, rather than a one-size-fits-all approach.
Limitations
- CFD may struggle with extremely complex visual scenes as it relies on initial text indexing.
- In scenarios requiring high-precision visual analysis, accuracy may decrease due to reduced visual processing.
Future Work
Future research directions include optimizing the decision-making mechanism of the Visual-Need Router to improve performance in complex scenes. Additionally, exploring better integration of multimodal information to further enhance system accuracy and efficiency.
AI Executive Summary
Long video understanding on edge devices faces constraints in computation and bandwidth. Traditional methods either lose temporal structure or fine-grained visual attributes. To address this, researchers propose the Caption-once, Frames-on-Demand (CFD) framework. This framework performs one-time offline captioning to create an event-level story skeleton and a clip-level micro-log, cached and reused during queries. A cloud-side multimodal large model reasons over the index in a story-first loop, using a Visual-Need Router to determine if keyframes need retrieval. Experiments show CFD maintains high accuracy while reducing visual processing, especially in long video scenarios. The CFD framework is significant in the field of long video understanding, particularly on resource-constrained edge devices. By reducing the need for visual processing, it enhances system efficiency and responsiveness while maintaining accurate video content understanding. This method offers new insights for multimodal applications, especially in scenarios requiring real-time processing and low latency. Future research directions include optimizing the decision-making mechanism of the Visual-Need Router to improve performance in complex scenes. Additionally, exploring better integration of multimodal information to further enhance system accuracy and efficiency.
Deep Analysis
Background
Long video understanding underlies multimodal applications like interactive video assistants and wearable visual aids. Existing methods mainly use visual compression and textual translation to reduce computational costs, but both have limitations. Visual compression may lead to insufficient coverage, while textual translation may lose fine-grained visual evidence.
Core Problem
Long video understanding requires processing vast content under limited computation and bandwidth. Existing methods face trade-offs in handling temporal structure and visual attributes, making it challenging to achieve both efficiency and accuracy.
Innovation
The core innovation of the CFD framework is its Visual-Need Routing mechanism. By transforming visual access into a query-conditioned cost, CFD can selectively allocate visual processing resources based on question type. This approach not only reduces the overhead of visual processing but also provides a flexible framework to handle different types of questions.
Methodology
- �� Edge devices perform one-time offline captioning to create an event-level story skeleton and a clip-level micro-log.
- �� The cloud uses a Visual-Need Router to determine if keyframes need retrieval, only accessing visuals for perceptual questions.
- �� A multimodal large model reasons over the index in a story-first loop.
Experiments
Experiments were conducted on Video-MME and InfiniBench benchmarks using Qwen3-VL-8B-Instruct and Qwen3-VL-32B-Instruct models. By comparing frame usage and accuracy across methods, CFD's efficiency and accuracy were validated.
Results
On the Video-MME benchmark, CFD uses an average of 5.8 frames per question with an overall accuracy of 67.5%. On InfiniBench, CFD excels in chronological understanding and global appearance, using an average of 14.8 frames per question.
Applications
The CFD framework is suitable for multimodal applications requiring real-time processing and low latency, such as interactive video assistants and wearable devices. Its efficient visual processing mechanism makes it well-suited for resource-constrained edge devices.
Limitations & Outlook
CFD may struggle with extremely complex visual scenes as it relies on initial text indexing. In scenarios requiring high-precision visual analysis, accuracy may decrease due to reduced visual processing.
Plain Language Accessible to non-experts
Imagine watching a long movie with limited battery on your phone. CFD is like a smart assistant that first notes down the movie's storyline and only opens the screen when you need to see a detail. This way, you can understand the whole movie without draining your battery.
ELI14 Explained like you're 14
Imagine playing a long game with limited battery on your device. CFD is like a smart assistant that first notes down the game's storyline and only opens the screen when you need to see a detail. This way, you can understand the whole game without draining your battery.
Glossary
Visual-Need Routing
A mechanism that determines whether to retrieve visual information based on question type.
Used to decide if keyframes need retrieval during queries.
Multimodal Large Model (MLLM)
A machine learning model capable of handling multiple data modalities like text and images.
Used for reasoning and decision-making in the cloud.
Event-level Story Skeleton
A simplified textual representation of video content, including main events and temporal order.
Used to create initial indexing on edge devices.
Clip-level Micro-log
Detailed textual descriptions of video clips, including local temporal details.
Used to provide fine-grained temporal information during queries.
Edge Device
A small computing device close to the data source, usually with limited computational power.
Used for initial offline captioning.
Open Questions Unanswered questions from this research
- 1 How to improve CFD's performance in complex visual scenes without increasing computational costs?
- 2 How to optimize the decision-making mechanism of the Visual-Need Router to enhance accuracy?
Applications
Immediate Applications
Interactive Video Assistants
CFD can enhance the responsiveness and accuracy of video assistants, especially on resource-constrained devices.
Long-term Vision
Smart Surveillance Systems
CFD can be used to develop efficient surveillance systems capable of real-time analysis without consuming excessive resources.
Abstract
Long-video understanding on edge devices must reason over hours of content under tight compute and bandwidth budgets. Subsampling visual tokens loses temporal structure, while text-only video memories lose fine-grained visual attributes. We observe a visual-textual duality: language memories carry long-range temporal structure better than dense frames, while pixels remain decisive for attribute-level perception. Building on this insight, we propose Caption-once, Frames-onDemand (CFD), a budget-aware edge-cloud agentic framework. The edge runs a single offline captioning pass that builds a dual-track narrative index, an event-level story skeleton plus a clip-level micro-log, cached and reused across queries without re-captioning. At query time, a cloud-side MLLM reasons over the index in a story-first loop centered on a lightweight Visual-Need Router: a per-query gating module that triggers bounded keyframe retrieval only for perceptual questions (appearance, on-screen text, attribute disambiguation) and keeps temporal-structural questions in language space. The router turns visual access into a first-class, query-conditioned cost, capping per-query frame consumption regardless of video length. Experiments on long-video benchmarks demonstrate strong accuracy-efficiency trade-offs while substantially reducing online visual processing.