VideoHarness-RSI: Recursive Harness Self-Improvement for Long-Video Understanding with Frozen Vision-Language Models
VideoHarness-RSI recursively optimizes context construction for frozen vision-language models to improve long-video understanding.
Key Findings
Methodology
VideoHarness-RSI framework recursively searches for optimized context constructors around a frozen VLM, keeping the model and interface fixed. It explores weak and strong initialization regimes, progressively discovering structured context-construction programs from a weak uniform constructor and advancing a competitive frontier from a strong AKS harness.
Key Results
- From a weak uniform constructor, recursive search improved accuracy to 54.8% on LVBench.
- From a strong AKS harness, the search advanced accuracy to 58.3%.
- Cross-benchmark tests showed 66.9% and 72.0% accuracy on Video-MME and MLVU, respectively.
Significance
This study significantly enhances long-video understanding by recursively optimizing context construction, establishing a new optimization layer and advancing frozen VLM research.
Technical Contribution
Introduces a novel recursive optimization framework that improves system performance without altering model parameters, providing an auditable baseline for studying context construction discovery, transfer, and efficiency.
Novelty
First to treat context construction as an independent optimization target for recursive search, differing from traditional hand-crafted strategies.
Limitations
- The method relies on frozen VLMs, which may not adapt to dynamic model parameter changes.
- Recursive search can increase computational costs in certain scenarios.
Future Work
Future directions include applying the recursive optimization framework to dynamic VLMs and exploring its applicability in other domains.
AI Executive Summary
Understanding long videos remains a challenge for vision-language models, as existing methods often rely on hand-crafted sampling, retrieval, and memory strategies, making context construction difficult to study independently. VideoHarness-RSI offers a recursive optimization framework that keeps the vision-language model and interface fixed, establishing a new optimization layer. Experiments show that this method performs exceptionally well on LVBench and other long-video benchmarks, significantly improving long-video understanding. Future research directions include applying this framework to dynamic vision-language models and exploring its applicability in other fields.
Deep Analysis
Background
Long-video understanding requires effective context construction beyond the capabilities of vision-language models. Existing systems often use compression, retrieval, and memory strategies, but these methods make it difficult to optimize context construction independently.
Core Problem
Relevant evidence in long videos is sparse and scattered, making it challenging for existing methods to construct effective context, impacting vision-language model performance.
Innovation
VideoHarness-RSI optimizes context construction through recursive search, keeping the vision-language model and interface fixed, providing a new optimization layer.
Methodology
- �� Use weak uniform constructor for recursive search, progressively discovering structured context-construction programs.
- �� Start from strong AKS harness to further advance hand-crafted frontier.
- �� Direct transfer to cross-benchmark tests without further search.
Experiments
Experiments conducted on LVBench using 83 videos and 1,232 QA pairs for development set search, reserving 882 QA pairs for final evaluation.
Results
Results show recursive search improved accuracy to 54.8% on LVBench, achieving 66.9% and 72.0% accuracy on cross-benchmark tests.
Applications
Applicable for long-video understanding, suitable for industries dealing with large amounts of video data, such as media and entertainment.
Limitations & Outlook
The method relies on frozen VLMs, which may not adapt to dynamic model parameter changes.
Plain Language Accessible to non-experts
Imagine you're in a library looking for books. There are thousands of books on the shelves, but you only need to find a few relevant ones. VideoHarness-RSI is like a smart librarian who knows how to quickly find the books you need and put them in a basket for you. This process doesn't change the books in the library, it just optimizes the way to find them.
ELI14 Explained like you're 14
Hey buddy! Imagine playing a super long game where you need to find hidden treasures. VideoHarness-RSI is like a super helper that quickly finds those treasures without you having to search the entire game map! Isn't that cool? It helps you find answers faster in the game, saving time and effort.
Glossary
Vision-Language Model
A model combining visual and language processing capabilities, able to understand the relationship between images and text.
Used as the core model in long-video understanding.
Context Construction
Extracting relevant information from long videos and constructing context that the model can process.
Optimization target to enhance model performance.
Recursive Optimization
Optimizing a program or algorithm through repeated search and improvement.
Used to enhance the efficiency of context construction programs.
AKS Harness
A hand-crafted context construction strategy used as a starting point for optimization.
Used in strong initialization regime.
LVBench
A benchmark dataset for long-video understanding used to evaluate model performance.
Main dataset used in experiments.
Open Questions Unanswered questions from this research
- 1 How to apply recursive optimization framework in dynamic vision-language models?
- 2 What is the applicability of this method in other domains?
Applications
Immediate Applications
Media Industry
Helps the media industry quickly process and understand large amounts of video data, improving efficiency.
Long-term Vision
AI Research
Provides a new optimization framework for the AI field, driving technological advancement.
Abstract
Long-video understanding depends not only on the capability of a vision-language model (VLM), but also on how its limited context is constructed from a much longer video. Existing systems typically introduce hand-designed sampling, retrieval, memory, or agentic control strategies, making the context-construction program itself difficult to study as an independent optimization target. We introduce VideoHarness-RSI, a controlled framework that recursively searches executable context constructors around a frozen VLM while keeping the answering model and interface fixed. We study this baseline under complementary weak- and strong-initialization regimes. From a weak uniform constructor, recursive search progressively discovers more structured context-construction programs; from a stronger AKS harness, the same process further advances an already competitive hand-crafted frontier. The resulting harness retains its advantage under a matched cumulative visual-token control and transfers directly to additional long-video benchmarks without further search. Together, these results establish executable context construction as a distinct optimization layer and provide an auditable baseline for studying harness discovery, transfer, and efficiency around frozen VLMs.