ReaSon: Reinforced Causal Search with Information Bottleneck for Video Understanding
ReaSon employs causal information bottleneck and reinforcement learning to select keyframes, significantly improving video reasoning accuracy under limited frame budgets.
Key Findings
Methodology
ReaSon integrates a learnable policy network with a causal information bottleneck (CIB), comprising two modules: predictive sufficiency and causal necessity. The sufficiency module detects question-relevant visual elements and approximates mutual information via variational methods to select informative frames. The necessity module employs counterfactual interventions to assess the causal contribution of each frame, measuring output shifts with KL divergence. Reinforcement learning with a composite reward guides the policy to identify minimal yet causally decisive keyframes, optimizing both information retention and causal relevance. This approach effectively balances efficiency and reasoning robustness, validated across multiple datasets.
Key Results
- On NExT-QA, ReaSon with 8 frames achieved 81.4% accuracy, surpassing previous SOTA AKEYS by 3.3%. On EgoSchema, it reached 72.2%, outperforming T*'s 66.6%. In long video datasets like Video-MME, using GPT-4o, ReaSon achieved 66.4%, a 10% improvement over models without keyframe selection. Ablation studies confirmed the importance of both modules, with significant performance gains when combined. Cross-model evaluations showed consistent generalization, demonstrating the method’s robustness across different vision-language models and tasks.
- Results demonstrate that incorporating causal information bottleneck significantly enhances reasoning accuracy, especially in causal and long-form video tasks. Ablation experiments reveal the individual contributions of sufficiency and necessity modules, with combined use yielding optimal results. The method's adaptability across diverse datasets and models underscores its potential as a universal framework for efficient, causally grounded video understanding.
- These findings highlight the importance of causal reasoning in visual understanding, paving the way for more interpretable and resource-efficient models capable of complex inference with limited data.
Significance
ReaSon's causal information bottleneck approach fundamentally advances video understanding by explicitly modeling causal dependencies, overcoming limitations of correlation-based methods. It enables models to focus on causally essential frames, improving reasoning accuracy and robustness, particularly under constrained input conditions. This paradigm shift fosters more interpretable AI systems that can better mimic human causal reasoning, with broad implications for applications such as autonomous systems, surveillance, and multimedia retrieval. The integration of causal inference with reinforcement learning opens new avenues for developing adaptive, resource-efficient models capable of deep understanding in complex, real-world scenarios. Its success across multiple datasets and models demonstrates its potential as a foundational technique in next-generation visual reasoning systems.
Technical Contribution
This work introduces a novel causal information bottleneck (CIB) framework that extends traditional IB by incorporating causal interventions, enabling explicit modeling of causal necessity. The policy network trained via reinforcement learning optimizes a composite reward aligned with CIB objectives, balancing predictive sufficiency and causal necessity. The method employs variational approximations for mutual information and counterfactual interventions to evaluate causal contributions, providing theoretical guarantees of causal relevance. The approach bridges information-theoretic principles with causal inference, offering a rigorous foundation for causally grounded keyframe selection. Empirically, it outperforms existing methods across multiple benchmarks, demonstrating both effectiveness and generalization, and opens new directions for integrating causality into visual reasoning pipelines.
Novelty
This is the first work to embed causal inference directly into the keyframe selection process via a causal information bottleneck. Unlike prior methods focusing solely on visual relevance or semantic similarity, ReaSon models causal dependencies explicitly, using counterfactual interventions to identify frames that are causally necessary for reasoning. The integration of variational mutual information approximation with reinforcement learning guided by a composite reward is a novel engineering solution that ensures efficiency and interpretability. This paradigm shift from correlation-based to causally grounded frame selection marks a significant advancement in video understanding, providing both theoretical novelty and practical performance gains.
Limitations
- The computational overhead of counterfactual interventions and KL divergence calculations may limit real-time applications, especially on very long videos. The reliance on visual element detection accuracy can introduce errors affecting causal inference. Additionally, the current framework primarily focuses on single-modal visual data, and extending it to multi-modal scenarios remains a challenge. Further research is needed to improve scalability and robustness in diverse, real-world environments.
Future Work
Future directions include developing more efficient causal inference algorithms to reduce computational costs, integrating multi-modal data such as audio and text for richer causal modeling, and exploring self-supervised learning strategies to enhance generalization. Extending the framework to handle streaming data and real-time inference is also a promising avenue. Moreover, applying the causal information bottleneck to other domains like robotics, natural language understanding, and multimodal reasoning could broaden its impact. Theoretical work on formal guarantees and interpretability of causal models will further strengthen its foundation.
AI Executive Summary
Video understanding has long been challenged by the sheer volume of redundant information and the limited capacity of current vision-language models (VLMs) to process lengthy sequences. Traditional keyframe selection methods rely heavily on visual relevance or semantic similarity, often neglecting the underlying causal structures that drive meaningful inference. This shortcoming results in models that are efficient but lack robustness and interpretability, especially when operating under strict input constraints.
ReaSon introduces a groundbreaking framework that leverages the causal information bottleneck (CIB) to address these issues. By explicitly modeling the causal dependencies among frames, questions, and outputs, ReaSon ensures that selected keyframes are not only informative but also causally necessary. The core of the approach involves two modules: a predictive sufficiency module that detects relevant visual elements and approximates mutual information via variational methods, and a causal necessity module that employs counterfactual interventions to evaluate the causal contribution of each frame. These modules are integrated into a reinforcement learning paradigm, where a learnable policy network optimizes a composite reward combining answer correctness, semantic consistency, and causal influence.
Extensive experiments on datasets such as NExT-QA, EgoSchema, and Video-MME demonstrate that ReaSon consistently outperforms existing state-of-the-art methods, especially under limited-frame settings. For instance, on NExT-QA, it achieves 81.4% accuracy with only 8 frames, surpassing previous bests by over 3%. Similarly, on long videos, ReaSon enhances reasoning accuracy by capturing causal structures that traditional relevance-based methods overlook. These results validate the effectiveness of integrating causality into keyframe selection, offering a new paradigm for resource-efficient, interpretable video understanding.
The broader impact of this work lies in its potential to transform how AI systems reason about complex temporal data. By focusing on causally decisive information, models become more robust, explainable, and adaptable across diverse applications—from autonomous driving to multimedia retrieval. Despite its advances, the approach faces challenges such as computational costs for causal inference and scalability to multi-modal data. Future research will aim to develop more efficient algorithms, extend causal modeling to richer data sources, and explore self-supervised training strategies. Overall, ReaSon sets a new standard for causally grounded visual reasoning, opening avenues for more intelligent and trustworthy AI systems.
Deep Dive
Abstract
Keyframe selection has become essential for video understanding with vision-language models (VLMs) due to limited input tokens and the temporal sparsity of relevant information across video frames. Video understanding often relies on effective keyframes that are not only informative but also causally decisive. To this end, we propose Reinforced Causal Search with Information Bottleneck (ReaSon), a framework that formulates keyframe selection as an optimization problem with the help of a novel Causal Information Bottleneck (CIB), which explicitly defines keyframes as those satisfying both predictive sufficiency and causal necessity. Specifically, ReaSon employs a learnable policy network to select keyframes from a visually relevant pool of candidate frames to capture predictive sufficiency, and then assesses causal necessity via counterfactual interventions. Finally, a composite reward aligned with the CIB principle is designed to guide the selection policy through reinforcement learning. Extensive experiments on NExT-QA, EgoSchema, and Video-MME demonstrate that ReaSon consistently outperforms existing state-of-the-art methods under limited-frame settings, validating its effectiveness and generalization ability.