SER: Learning to Ground Video Reasoning with Semantic Evidence Rewards
SER method enhances video reasoning with Semantic Evidence Rewards, achieving a 3.0-point mLGM improvement on V-STAR.
Key Findings
Methodology
The paper introduces the Semantic Evidence Reward (SER) framework, redefining spatio-temporal evidence grounding as a constrained verification task. SER uses a referee VLM as a local checker to evaluate model-generated evidence claims in terms of relevance and localization quality, combined with a temporal penalty. This design reduces reliance on dense box annotations and enables training directly on standard video QA data.
Key Results
- On the V-STAR benchmark, SER achieved 49.6% mLGM, improving by 3.0 points over the strong evidence-grounded baseline Open-o3-Video, demonstrating its potential in enhancing both answer accuracy and evidence grounding.
- Experiments show that SER improves average performance across temporal, spatial, and overall metrics by about 3.0%, significantly outperforming its base Video MLLM.
- On long-form reasoning tasks, SER-7B reached 71.7% accuracy, surpassing VideoR1-7B's 68.9% and Open-o3-Video's 69.4%.
Significance
This research effectively bridges high-level reasoning with low-level visual grounding by introducing a semantic-level verification loop, addressing the limitations of existing geometry-only reward methods. It not only improves answer accuracy but also enhances the interpretability and reliability of the reasoning process.
Technical Contribution
The technical contribution of the SER framework lies in its innovative use of semantic evidence rewards to replace traditional geometry-only rewards, providing a smoother credit assignment mechanism. By introducing a referee VLM for local verification, SER avoids reliance on dense box annotations, opening up new engineering possibilities.
Novelty
SER is the first to introduce semantic verification into spatio-temporal evidence grounding in video reasoning, offering a more robust reward mechanism compared to existing geometry-only methods, better handling the semantic distribution across multiple frames and nearby regions.
Limitations
- In some complex scenarios, SER may fail to capture all relevant spatio-temporal evidence, leading to reduced reasoning accuracy.
- Due to reliance on the referee VLM for verification, SER may incur higher computational costs.
Future Work
Future research could explore further optimization of the referee VLM's performance to reduce computational costs. Additionally, investigating how SER can be applied to other multimodal tasks, such as image reasoning and text generation, would be valuable.
AI Executive Summary
Multimodal Large Language Models (MLLMs) have shown strong performance in open-ended question answering and event understanding but still struggle with fine-grained spatio-temporal reasoning, especially in long videos and complex scenes. Existing reinforcement learning frameworks typically rely on geometry-only (IoU) rewards, which are sensitive to boundary perturbations and overlook semantic alignment.
To address these issues, this paper proposes the Semantic Evidence Reward (SER), which reformulates spatio-temporal evidence grounding as a constrained verification task. SER uses a referee VLM as a local checker to evaluate model-generated evidence claims across two dimensions: relevance and localization quality, combined with a temporal penalty. This design reduces reliance on dense box annotations and enables training directly on standard video QA data.
On the V-STAR benchmark, SER achieved 49.6% mLGM, improving by 3.0 points over the strong evidence-grounded baseline Open-o3-Video, demonstrating its potential in enhancing both answer accuracy and evidence grounding. Experimental results show that SER improves average performance across temporal, spatial, and overall metrics by about 3.0%, significantly outperforming its base Video MLLM. Future research could explore further optimization of the referee VLM's performance to reduce computational costs and investigate how SER can be applied to other multimodal tasks.
Deep Analysis
Background
Multimodal Large Language Models (MLLMs) have made significant progress in open-ended question answering and event understanding. However, they still face challenges in fine-grained spatio-temporal reasoning, particularly in long videos, crowded or interaction-rich scenes. Existing methods often rely on geometry-only (IoU) rewards, which are sensitive to boundary perturbations and overlook semantic alignment.
Core Problem
Existing geometry-only reward methods are often ineffective in video reasoning because video evidence is often semantically distributed: multiple frames or nearby regions may all support the same answer. Geometry-only rewards are highly sensitive to boundary perturbations, object deformation, and scale changes, leading to unstable reward signals.
Innovation
This paper proposes the Semantic Evidence Reward (SER), which reformulates spatio-temporal evidence grounding as a constrained verification task. Unlike geometry-only rewards, SER uses a referee VLM as a local checker to evaluate model-generated evidence claims in terms of relevance and localization quality, combined with a temporal penalty.
Methodology
- �� The SER framework uses a referee VLM as a local checker to evaluate model-generated evidence claims in terms of relevance and localization quality.
- �� By introducing a temporal penalty, SER reduces reliance on dense box annotations.
- �� SER can be trained directly on standard video QA data.
Experiments
Experiments were conducted on the V-STAR benchmark to evaluate the performance of SER in terms of temporal, spatial, and overall metrics. Results show that SER improves average performance across these metrics by about 3.0%, significantly outperforming its base Video MLLM.
Results
On the V-STAR benchmark, SER achieved 49.6% mLGM, improving by 3.0 points over the strong evidence-grounded baseline Open-o3-Video. Experimental results show that SER improves average performance across temporal, spatial, and overall metrics by about 3.0%.
Applications
SER can be applied to video reasoning tasks to enhance answer accuracy and the interpretability of the reasoning process. It can also be extended to other multimodal tasks, such as image reasoning and text generation.
Limitations & Outlook
In some complex scenarios, SER may fail to capture all relevant spatio-temporal evidence, leading to reduced reasoning accuracy. Additionally, due to reliance on the referee VLM for verification, SER may incur higher computational costs.
Plain Language Accessible to non-experts
Imagine a factory where workers need to produce products based on different orders. Traditional methods are like focusing only on the product's appearance, ignoring the details of the production process. SER is like a quality inspector who not only checks the product's appearance but also examines every step of the production process to ensure each step meets the standards. This way, even if there are small issues during production, the inspector can detect and correct them in time, ultimately producing high-quality products.
ELI14 Explained like you're 14
Imagine you're playing a complex game with many levels and tasks. Traditional methods are like only caring if you pass the level, not how you pass it. SER is like a smart assistant that not only helps you pass the level but also gives you tips at each stage, telling you where to pay attention. So, even if you encounter difficulties in a level, the assistant can help you find a solution, allowing you to pass the level faster!
Glossary
Semantic Evidence Reward
A reward mechanism that evaluates the relevance and localization quality of model-generated evidence claims through semantic verification.
Used to replace traditional geometry-only rewards, enhancing the accuracy and interpretability of video reasoning.
Referee VLM
A visual language model used to evaluate the relevance and localization quality of model-generated evidence claims.
Acts as a local checker in the SER framework, providing semantic-level reward signals.
Intersection over Union (IoU)
A geometric metric used to evaluate the overlap between two regions, commonly used in object detection tasks.
Traditionally used to evaluate the validity of evidence grounding in video reasoning.
Spatio-temporal Evidence Grounding
The process of identifying and localizing spatio-temporal evidence that supports the answer in video reasoning.
A key step achieved through semantic verification in the SER framework.
Multimodal Large Language Model
A language model that combines multiple modalities (e.g., text, image, video) for complex reasoning tasks.
Used in video reasoning tasks to generate answers and reasoning traces.
Open Questions Unanswered questions from this research
- 1 How to further optimize the performance of the referee VLM without increasing computational costs?
- 2 How to apply SER to other multimodal tasks, such as image reasoning and text generation?
Applications
Immediate Applications
Video Reasoning
SER can be applied to video reasoning tasks to enhance answer accuracy and the interpretability of the reasoning process.
Long-term Vision
Multimodal Tasks
SER can be extended to other multimodal tasks, such as image reasoning and text generation, enhancing their performance and interpretability.
Abstract
Video MLLMs often struggle with fine-grained spatio-temporal reasoning, sometimes generating correct answers based on irrelevant frames or objects. Although outputting spatio-temporal evidence during reasoning is a promising direction, existing RL frameworks typically rely on geometry-only (IoU) rewards, which can be sensitive to boundary perturbations and overlook semantic alignment. To address this, we propose Semantic Evidence Reward (SER), which reformulates spatio-temporal evidence grounding as a constrained verification task. Instead of computing pixel-level overlap, SER uses a referee VLM as a local checker to evaluate model-generated evidence claims across two dimensions: relevance and localization quality, combined with a temporal penalty. This design reduces the reliance on dense box annotations and enables training directly on standard video QA data. On the V-STAR benchmark, SER achieves 49.6% mLGM, improving by 3.0 points over the strong evidence-grounded baseline Open-o3-Video, demonstrating its potential in enhancing both answer accuracy and evidence grounding.