From Accuracy to Visual Dependence: Auditing and Filtering Modality Collapse in Traffic VideoQA
Introduced Blind Gap and Visual Gain metrics to reveal visual dependence issues in traffic accident VideoQA.
Key Findings
Methodology
The study employs Blind Gap and Visual Gain as dataset-level diagnostics to assess visual dependence. Additionally, it proposes an instance-level Shortcut Score to identify shortcut-prone questions and enable continuous filtering.
Key Results
- On the MM-AU dataset, removing video input improved accuracy, with Blind Gap reaching 40.33 and Visual Gain being negative, indicating video input failed to help.
- VRU-Accident dataset showed positive Visual Gain but still high Blind Gap, indicating text solvability.
- SUTD-TrafficQA dataset exhibited the lowest Blind Gap and highest Visual Gain, indicating the strongest visual dependence.
Significance
The study reveals that models in traffic accident VideoQA may answer questions using textual shortcuts rather than visual evidence, emphasizing the importance of visually grounded evaluation in safety-critical tasks.
Technical Contribution
Introduced Blind Gap and Visual Gain as lightweight diagnostic tools, combined with Shortcut Score for instance-level filtering, significantly reducing shortcut bias and improving visual grounding evaluation quality.
Novelty
First to propose a diagnostic framework combining Blind Gap and Visual Gain, providing an instance-level Shortcut Score to identify and filter shortcut-prone questions.
Limitations
- Current methods still show ineffective visual input in certain datasets, possibly due to dataset design.
- Filtering mechanism may reduce dataset size, affecting coverage.
Future Work
Future research could explore more complex VideoQA scenarios, develop finer visual dependence assessment tools, and improve dataset design to reduce textual shortcuts.
AI Executive Summary
In the field of traffic accident VideoQA, existing models often answer questions using textual shortcuts rather than visual evidence, which is particularly concerning in safety-critical tasks. To address this issue, the study introduces Blind Gap and Visual Gain as dataset-level diagnostics, along with an instance-level Shortcut Score to identify and filter shortcut-prone questions.
The study finds that on the MM-AU dataset, removing video input actually improved accuracy, indicating that video input failed to help. In contrast, the SUTD-TrafficQA dataset exhibited the strongest visual dependence, showing positive Visual Gain and the lowest Blind Gap.
This research emphasizes the importance of visually grounded evaluation in safety-critical tasks and provides new insights for future multimodal dataset design. By reducing textual shortcuts, the study improves the quality of visual dependence evaluation, laying the foundation for further development in the field of traffic accident VideoQA.
Deep Analysis
Background
With the evolution of multimodal learning, the field of VideoQA has expanded to temporal and causal reasoning. However, existing traffic accident VideoQA datasets often fail to fully utilize visual evidence, leading models to answer questions using textual shortcuts.
Core Problem
The core problem in traffic accident VideoQA is whether models truly rely on visual evidence to answer questions, rather than using textual shortcuts. This issue is particularly important in safety-critical tasks.
Innovation
The study introduces Blind Gap and Visual Gain as dataset-level diagnostics, along with an instance-level Shortcut Score to identify and filter shortcut-prone questions. These tools help improve the quality of visual dependence evaluation.
Methodology
- �� Use Blind Gap to assess text solvability
- �� Employ Visual Gain to measure the benefit of video input
- �� Propose Shortcut Score for instance-level filtering, combining text confidence and visual necessity signals
Experiments
The study evaluates four traffic accident VideoQA datasets, including MM-AU, VRU-Accident, SUTD-TrafficQA, and AccidentBench. By comparing accuracy with text and video inputs, it analyzes visual dependence.
Results
On the MM-AU dataset, removing video input improved accuracy, with Blind Gap reaching 40.33 and Visual Gain being negative. The SUTD-TrafficQA dataset exhibited the lowest Blind Gap and highest Visual Gain, indicating the strongest visual dependence.
Applications
The findings can be used to improve the design of traffic accident VideoQA models, enhancing the quality of visual dependence evaluation and reducing the impact of textual shortcuts on model performance.
Limitations & Outlook
Current methods still show ineffective visual input in certain datasets, possibly due to dataset design. Filtering mechanism may reduce dataset size, affecting coverage.
Plain Language Accessible to non-experts
Imagine a kitchen where a chef needs to cook based on a recipe. The recipe is like text input, while the ingredients are the visual input. If the chef only looks at the recipe and ignores the ingredients, he might cook the wrong dish. In traffic accident VideoQA, models are like chefs, needing to rely on both text and visual input to make correct judgments.
ELI14 Explained like you're 14
Imagine you're playing a game where you need to make choices based on the scene. If you only look at the game manual and ignore the screen, you might make the wrong choice. In traffic accident VideoQA, models are like players, needing to rely on both the manual and the screen to make correct judgments.
Glossary
Blind Gap
Measures the accuracy gap where text input exceeds random chance.
Used to assess text solvability.
Visual Gain
Measures the change in accuracy when video input is added.
Used to assess the benefit of video input.
Shortcut Score
An instance-level score combining text confidence and visual necessity signals.
Used to identify and filter shortcut-prone questions.
MM-AU
A traffic accident VideoQA dataset with 58 accident categories.
Used to assess visual dependence.
SUTD-TrafficQA
A video QA dataset with multiple reasoning types.
Used to assess visual dependence.
Open Questions Unanswered questions from this research
- 1 How to design more complex VideoQA scenarios to enhance visual dependence?
- 2 How to optimize dataset design to reduce textual shortcuts?
Applications
Immediate Applications
Traffic Accident Analysis
Improving the accuracy of traffic accident analysis models by enhancing visual dependence evaluation quality.
Long-term Vision
Autonomous Driving Safety
Enhancing the application of VideoQA models in autonomous driving by reducing the impact of textual shortcuts on safety.
Abstract
High benchmark accuracy does not guarantee genuine use of visual evidence. We study this problem in traffic accident Video Question Answering (VideoQA), where correct answers should depend on scene-specific visual evidence but may instead be inferred from textual shortcuts. Through an audit of four public benchmarks, we find that several recent open-weight Vision-Language Models (VLMs) perform competitively, and sometimes better, without video input. On the MM-AU benchmark, removing video consistently improves accuracy, and adding more frames further degrades performance. To quantify visual dependence, we introduce two dataset-level diagnostics: Blind Gap, measuring above-chance text-only performance, and Visual Gain, measuring the marginal benefit of adding video. We further propose an instance-level Shortcut Score that combines text-only confidence with visual necessity signals, enabling continuous, training-free filtering of shortcut-prone questions. The resulting subsets reduce shortcut bias and improve visual grounding. Our findings reveal large differences in grounding quality across benchmarks and show that visually grounded evaluation, not just high accuracy, is essential in safety-critical VideoQA.