CoFiE: Coarse-to-Fine Evidence Selection for Efficient Streaming Video Understanding
CoFiE achieves a state-of-the-art accuracy-efficiency trade-off in video understanding benchmarks, reaching 78.86% accuracy with coarse-to-fine evidence selection.
Key Findings
Methodology
CoFiE framework divides evidence selection into two stages: coarse filtering before visual encoding and fine selection during LLM prefill. Coarse filtering uses Novelty-Guided Frame Filtering to retain visually distinctive frames, while fine selection employs Query-Specific Evidence Refinement to choose frames most relevant to the user query.
Key Results
- On StreamingBench, CoFiE achieves 78.86% accuracy with 50% frame dropping, outperforming previous methods by 1.60%.
- On OvO-Bench, CoFiE reaches 68.72% average score with 50% frame dropping, surpassing Qwen3-VL-8B-Instruct by 3.69%.
- Even with 80% evidence-frame filtering, CoFiE retains 99.0% of its no-drop performance.
Significance
CoFiE significantly reduces computational redundancy by selecting evidence before visual encoding, enhancing efficiency in streaming video understanding. This approach is crucial for both academia and industry, especially in real-time video analysis requiring rapid responses.
Technical Contribution
CoFiE is the first to perform pre-encoder frame-level pruning for streaming VLLM inference. By introducing Novelty-Guided Frame Filtering and Query-Specific Evidence Refinement, CoFiE offers new theoretical guarantees and engineering possibilities.
Novelty
CoFiE is the first to perform evidence selection before visual encoding, significantly reducing computational costs while maintaining high accuracy. Unlike traditional single-stage compression methods, CoFiE's dual-stage design is more robust in handling non-uniform video evidence.
Limitations
- CoFiE may miss critical frames in extremely complex or rapidly changing video scenes, affecting accuracy.
- Reliance on novelty scoring may lead to misjudgments in certain cases.
Future Work
Future research could explore more sophisticated frame selection mechanisms, integrating deep learning models to improve novelty scoring accuracy. Extending CoFiE to support more types of video data is also an intriguing direction.
AI Executive Summary
Streaming video understanding requires processing growing video streams under tight latency constraints. Existing methods mainly improve efficiency through token pruning after visual encoding, which limits end-to-end latency reduction. CoFiE framework significantly reduces computational redundancy by performing coarse filtering before visual encoding and fine selection during LLM prefill. Experiments show that CoFiE achieves a new accuracy-efficiency trade-off across multiple video understanding benchmarks, reaching 78.86% accuracy and maintaining high performance even with 80% evidence-frame filtering. CoFiE's design is significant for both academia and industry, particularly in real-time video analysis requiring rapid responses. However, CoFiE may miss critical frames in extremely complex or rapidly changing video scenes. Future research could explore more sophisticated frame selection mechanisms to enhance its robustness.
Deep Analysis
Background
Streaming video understanding is a core capability for Vision Language Models (VLLMs). Existing methods like VideoStreaming and TimeChat-Online improve efficiency by reducing visual tokens, but they perform evidence reduction after visual encoding, leading to limited computational savings.
Core Problem
The core problem in streaming video understanding is reducing computational costs without compromising accuracy. Existing methods mainly prune tokens after visual encoding, failing to significantly reduce end-to-end latency.
Innovation
CoFiE's core innovation lies in its dual-stage evidence selection framework: coarse filtering before visual encoding to retain visually distinctive frames, and fine selection during LLM prefill to choose frames most relevant to the user query.
Methodology
- �� Coarse Filtering: Uses Novelty-Guided Frame Filtering, calculating novelty scores based on histogram changes.
- �� Fine Selection: During LLM prefill, uses text-to-visual attention to rank candidate frames and select the most relevant evidence frames.
Experiments
Experiments were conducted on benchmarks like StreamingBench and OvO-Bench, using Qwen3-VL-8B-Instruct as the baseline. Evaluation metrics included accuracy and inference time, showing CoFiE achieves a new accuracy-efficiency trade-off across multiple benchmarks.
Results
CoFiE achieves 78.86% accuracy on StreamingBench, outperforming previous methods by 1.60%. On OvO-Bench, CoFiE reaches 68.72% average score with 50% frame dropping, surpassing Qwen3-VL-8B-Instruct by 3.69%.
Applications
CoFiE can be applied in real-time video analysis, such as surveillance systems and autonomous driving, where rapid response is crucial. Its efficient evidence selection mechanism significantly reduces computational costs, enhancing system responsiveness.
Limitations & Outlook
CoFiE may miss critical frames in extremely complex or rapidly changing video scenes, affecting accuracy. Future research could explore more sophisticated frame selection mechanisms to enhance its robustness.
Plain Language Accessible to non-experts
Imagine watching a long soccer match. You don't need every second's detail, just the key moments like goals and fouls. CoFiE acts like a smart viewer, selecting clips that might contain exciting moments before the match starts, then further refining these based on commentary during the match. This way, you see the highlights without wasting time.
ELI14 Explained like you're 14
Imagine playing a super long game, and you don't want to see every frame, just the coolest moments. CoFiE is like a super smart assistant that picks out frames that might be cool, then when you ask a question, it picks out the most relevant frames. So, you get to see the exciting parts of the game quickly without wasting time!
Glossary
Vision Language Models
Models that combine visual and language processing capabilities for understanding and generating multimodal information.
Used in streaming video understanding to process video streams and user queries.
Token Pruning
A method to reduce model computation by removing unimportant tokens to improve efficiency.
Existing methods mainly prune tokens after visual encoding.
Novelty-Guided Frame Filtering
A mechanism that selects candidate frames based on novelty scores.
Used in CoFiE for coarse filtering before visual encoding.
Query-Specific Evidence Refinement
A mechanism that selects frames most relevant to the user query using text-to-visual attention.
Used in CoFiE for fine selection during LLM prefill.
StreamingBench
A benchmark for evaluating streaming video understanding performance.
CoFiE achieves 78.86% accuracy on this benchmark.
Open Questions Unanswered questions from this research
- 1 How to further reduce computational costs without affecting accuracy? Existing methods prune after visual encoding, limiting latency reduction.
- 2 How to improve the accuracy of novelty scoring? Current methods may misjudge in certain cases.
Applications
Immediate Applications
Real-time Surveillance Systems
Efficient evidence selection reduces computational costs, enhancing system responsiveness and real-time performance.
Autonomous Driving
Quickly identifies and processes critical visual information, improving safety in autonomous driving scenarios.
Long-term Vision
Smart Cities
Efficient video analysis systems enhance public safety and traffic management in smart cities.
Abstract
Streaming video understanding requires Vision Language Models (VLLMs) to process growing video streams and answer user questions under tight latency constraints. Existing methods improve efficiency through token pruning and memory-bank schemes, but mainly reduce visual tokens after visual encoding. Consequently, downstream token pruning alone cannot substantially reduce end-to-end latency because the expensive frame encoding cost has already been incurred. We propose CoFiE, a Coarse-to-Fine Evidence Selection framework that decouples evidence selection into a coarse, query-agnostic filtering stage before the vision encoder and a fine, query-specific refinement stage during LLM prefill. CoFiE introduces Novelty-Guided Frame Filtering to retain visually distinctive candidate frames and Query-Specific Evidence Refinement to select the frames most relevant to the user query. This design removes substantial redundancy before frame encoding while preserving query-specific refinement once semantic information becomes available. Experiments show that CoFiE establishes a new state-of-the-art accuracy-efficiency trade-off across multiple video understanding benchmarks, reaching 78.86% accuracy on StreamingBench and 68.72% on OvO-Bench, with improvements of up to 3.15% over prior methods. Even with up to 80% evidence-frame filtering, CoFiE outperforms strong open-source multimodal models while improving end-to-end inference latency by up to 2.54 times.