AutoEval-Video: An Automatic Benchmark for Assessing Large Vision Language Models in Open-Ended Video Question Answering
AutoEval-Video is a benchmark for evaluating large vision-language models in video QA; GPT-4V excels but lags behind human accuracy of 72.8%.
Key Findings
Methodology
AutoEval-Video constructs open-ended video questions across 9 skill dimensions, assessing perception, comprehension, and generation capabilities. It employs an LLM evaluation approach with unique evaluation rules for each video-question pair, enhanced by an adversarial annotation mechanism.
Key Results
- GPT-4 as an automatic evaluator achieves a stable evaluation accuracy of 97.0%, comparable to human evaluators' 94.9% - 97.5%.
- GPT-4V achieves 32.2% accuracy on AutoEval-Video, outperforming other models but still below human accuracy of 72.8%.
- Case studies reveal GPT-4V's limitations in temporal and dynamic comprehension, and overly general responses.
Significance
AutoEval-Video provides a comprehensive and challenging benchmark for evaluating large vision-language models, addressing gaps in current evaluation methods for open-ended video QA. Its detailed evaluation rules and adversarial annotation mechanism enhance accuracy and reliability, guiding future model improvements.
Technical Contribution
This study introduces a new benchmarking framework combining instance-specific evaluation rules and an adversarial annotation mechanism, significantly improving evaluation accuracy and robustness. Compared to existing methods, AutoEval-Video offers superior comprehensiveness and detail.
Novelty
AutoEval-Video is the first to introduce instance-specific evaluation rules and an adversarial annotation mechanism in open-ended video QA, significantly enhancing evaluation accuracy and robustness.
Limitations
- GPT-4V exhibits poor performance in temporal and dynamic comprehension, leading to imprecise answers.
- The model's responses are overly general, lacking detail.
Future Work
Future research could explore improving models' temporal and dynamic comprehension abilities and developing more detailed response generation mechanisms to enhance overall performance.
AI Executive Summary
AutoEval-Video is a novel benchmark designed to evaluate large vision-language models in open-ended video question answering. Existing methods fall short in comprehensiveness and detail, while AutoEval-Video provides a robust framework by constructing video questions across 9 skill dimensions. The method employs an LLM evaluation approach with unique evaluation rules for each video-question pair, enhanced by an adversarial annotation mechanism. Experimental results show that GPT-4, as an automatic evaluator, achieves a stable evaluation accuracy of 97.0%, comparable to human evaluators' 94.9% - 97.5%. Although GPT-4V outperforms other models on AutoEval-Video, its 32.2% accuracy still lags behind human accuracy of 72.8%. Case studies reveal GPT-4V's limitations in temporal and dynamic comprehension, and overly general responses. Future research could explore improving models' temporal and dynamic comprehension abilities and developing more detailed response generation mechanisms to enhance overall performance.
Deep Analysis
Background
Recent advancements in deep learning and computer vision have led to significant progress in vision-language models for image and video understanding tasks. However, existing evaluation methods focus on closed-ended questions, lacking comprehensive evaluation for open-ended video QA. AutoEval-Video addresses this gap by constructing video questions across 9 skill dimensions.
Core Problem
Existing evaluation methods for vision-language models fall short in the open-ended video QA domain, failing to comprehensively assess models' perception, comprehension, and generation capabilities. AutoEval-Video provides a robust framework by constructing multidimensional question sets.
Innovation
AutoEval-Video's core innovation lies in introducing instance-specific evaluation rules and an adversarial annotation mechanism, significantly enhancing evaluation accuracy and robustness. This innovation enables a more detailed evaluation process, accurately reflecting models' actual capabilities.
Methodology
- �� Construct video questions across 9 skill dimensions
- �� Employ LLM evaluation approach with unique evaluation rules for each video-question pair
- �� Enhance evaluation rules' robustness through adversarial annotation mechanism
- �� Use GPT-4 as an automatic evaluator for accuracy assessment
Experiments
Experiments utilized a newly collected video dataset covering 40 themes, evaluating the performance of eight large vision-language models, including GPT-4V. Evaluation criteria included models' perception, comprehension, and generation capabilities.
Results
Experimental results show that GPT-4, as an automatic evaluator, achieves a stable evaluation accuracy of 97.0%, comparable to human evaluators' 94.9% - 97.5%. GPT-4V outperforms other models on AutoEval-Video, but its 32.2% accuracy still lags behind human accuracy of 72.8%.
Applications
AutoEval-Video can be used to evaluate and improve vision-language models' performance in open-ended video QA, applicable to scenarios requiring high-precision video understanding, such as autonomous driving and intelligent surveillance.
Limitations & Outlook
Despite AutoEval-Video's superior comprehensiveness and detail, models still exhibit poor performance in temporal and dynamic comprehension, leading to imprecise answers. Future research could explore improving these aspects.
Plain Language Accessible to non-experts
Imagine you're watching a movie, and someone asks you questions about the plot. AutoEval-Video is like a super-smart movie critic that not only understands the movie scenes but also answers various questions about the movie. This system sets specific rules for each question, like giving the critic a detailed scoring guide, allowing it to evaluate the movie's quality more accurately.
ELI14 Explained like you're 14
Imagine you're playing a super complex game, and the game asks you lots of questions about the game world. AutoEval-Video is like a super-smart game assistant that helps you answer these questions. It sets specific rules for each question, like giving the game assistant a detailed walkthrough, allowing it to help you win the game more accurately.
Glossary
AutoEval-Video
A benchmark for evaluating large vision-language models in video question answering.
Used to assess models' performance in open-ended video QA.
LLM (Large Language Model)
A deep learning model capable of understanding and generating natural language.
Used to evaluate the accuracy of video question answering.
GPT-4V(ision)
The vision version of GPT-4, focused on processing visual information.
Outperforms other models on AutoEval-Video.
Adversarial Annotation Mechanism
A method to enhance evaluation rules' robustness by introducing adversarial samples.
Used to improve the robustness of evaluation rules.
Open-ended Video QA
A task requiring understanding of video content and generating natural language answers.
The core evaluation task of AutoEval-Video.
Open Questions Unanswered questions from this research
- 1 Improving models' performance in temporal and dynamic comprehension remains an open question.
- 2 Existing models lack detailed response generation, requiring further research.
Applications
Immediate Applications
Autonomous Driving
Enhancing environmental perception and decision-making in autonomous driving systems through improved video understanding.
Long-term Vision
Intelligent Surveillance
Applying in intelligent surveillance systems to enhance anomaly detection and response capabilities.
Abstract
We propose a novel and challenging benchmark, AutoEval-Video, to comprehensively evaluate large vision-language models in open-ended video question answering. The comprehensiveness of AutoEval-Video is demonstrated in two aspects: 1) AutoEval-Video constructs open-ended video-questions across 9 skill dimensions, addressing capabilities of perception, comprehension, and generation. 2) AutoEval-Video contains newly collected videos that cover over 40 distinct themes. To efficiently evaluate responses to the open-ended questions, we employ an LLM-based evaluation approach, but instead of merely providing a reference answer, we annotate unique evaluation rules for every single instance (video-question pair). To maximize the robustness of these rules, we develop a novel adversarial annotation mechanism. By using instance-specific rules as prompt, GPT-4, as an automatic evaluator, can achieve a stable evaluation accuracy of around 97.0%, comparable to the 94.9% - 97.5% accuracy of a human evaluator. Furthermore, we assess the performance of eight large vision-language models on AutoEval-Video. Among them, GPT-4V(ision) significantly outperforms other models, achieving an accuracy of 32.2%. However, there is still substantial room for improvement compared to human accuracy of 72.8%. By conducting an extensive case study, we uncover several drawbacks of GPT-4V, such as limited temporal and dynamic comprehension, and overly general responses. Code is available at https://github.com/Xiuyuan-Chen/AutoEval-Video.