FETV: A Benchmark for Fine-Grained Evaluation of Open-Domain Text-to-Video Generation
FETV benchmark introduces multi-dimensional, temporal-aware fine-grained evaluation for open-source T2V models, revealing poor correlation of existing metrics with human judgment.
Key Findings
Methodology
FETV employs three orthogonal aspects: major content, controllable attributes, and prompt complexity, integrating spatial and temporal categories. Text prompts are automatically categorized using keyword matching and WordNet, then manually refined. Data collection includes 619 prompts from datasets like MSR-VTT and WebVid, plus manually created unusual prompts. Four representative T2V models are manually evaluated across these categories, analyzing strengths and weaknesses. The benchmark assesses automatic metrics (CLIPScore, FVD) against human ratings, revealing poor correlations. Leveraging large vision-language models (e.g., UMT), the authors propose improved metrics—FVD-UMT and UMTScore—that show significantly higher correlation with human judgment.
Key Results
- In spatial categories, videos of 'people' and 'animals' have lower quality, while 'water' and 'scenery' are better; motion and action videos are of worse quality due to high-frequency content. Static quality scores are highest for Text2Video-zero, but temporal coherence lags. Existing automatic metrics correlate poorly with human ratings (Kendall τc ~0.2-0.3); after incorporating UMT, correlation improves to over 0.6. Models struggle with controlling motion direction and event order, especially in complex scenarios. The new metrics outperform traditional ones, providing more reliable automatic evaluation.
- FETV enables detailed performance analysis across content and attribute categories, revealing specific weaknesses in dynamic content and high-frequency details. It offers a comprehensive tool for model comparison, guiding future improvements. The benchmark emphasizes the importance of temporal categories, aligning evaluation more closely with real video generation challenges.
- Overall, FETV bridges the gap between qualitative assessments and quantitative metrics, fostering more nuanced understanding of open-domain T2V models. It encourages development of models with better controllability and temporal coherence, supporting industry adoption and research standardization.
Significance
This work addresses the critical need for fine-grained, multi-aspect evaluation of T2V models, which is essential for advancing realistic and controllable video synthesis. By systematically categorizing prompts and incorporating temporal-aware categories, FETV offers a comprehensive framework that reveals performance gaps overlooked by traditional metrics. The improved automatic metrics based on large vision-language models significantly enhance evaluation reliability, reducing dependence on costly human judgments. This benchmark not only facilitates fair comparison among models but also guides targeted improvements, accelerating progress in the field. Its adoption can lead to more robust, controllable, and high-quality video generation systems, impacting applications in entertainment, virtual reality, and content creation.
Technical Contribution
The key innovation lies in the multi-aspect, multi-level categorization system that combines spatial and temporal content with controllable attributes and complexity levels. The integration of temporal categories addresses a major gap in existing evaluation frameworks. The automatic categorization process, combining keyword matching with manual refinement, ensures data richness and accuracy. Building on large-scale vision-language models like UMT, the authors develop FVD-UMT and UMTScore, which significantly outperform traditional metrics in correlating with human judgments. These contributions establish a new standard for automatic evaluation in open-domain T2V, enabling more precise and reliable performance assessment.
Novelty
This study is the first to systematically incorporate multi-aspect, fine-grained, and temporal-aware categories into T2V evaluation. Unlike prior benchmarks that focus on coarse content or challenge types, FETV captures detailed content, controllability, and dynamic aspects, especially emphasizing temporal content. The automatic matching and manual correction process ensures high-quality, diverse prompt datasets. The development of UMT-based metrics tailored for video-text alignment and quality assessment further distinguishes this work, providing tools that better reflect human perception and control in generated videos.
Limitations
- The current类别体系主要覆盖静态内容和部分动态类别，难以全面描述极端复杂或模糊场景，存在偏差。模型在高复杂度和细粒度控制任务中的表现仍有限，尤其在事件顺序和运动细节的精确控制方面。自动匹配依赖关键词和规则，可能出现偏差。指标与人类评价存在差异，需进一步优化算法和数据标注流程。未来应扩展类别体系，提升自动化水平，增强模型在复杂动态环境中的适应能力。
- 评估过程依赖大量人工校正，成本较高，自动化程度仍有提升空间。模型在处理极端复杂场景时表现不足，未来需引入更丰富的类别和更强的时序理解能力。自动指标虽有改进，但在某些模糊或高复杂度场景中仍不够准确，需持续优化。
Future Work
未来将扩展FETV的类别体系，涵盖更多复杂场景和细粒度属性，增强其适应性。计划引入深度学习自动标注与校正机制，降低人工成本。结合多模态信息，提升模型在复杂动态内容生成中的控制能力。探索端到端的实时自动评估框架，结合模型训练过程实现动态性能监控。最终目标是建立统一、标准化的T2V性能评估体系，推动行业标准化和模型性能持续提升。
AI Executive Summary
近年来，开源文本到视频（T2V）生成技术取得了快速发展，模型在视觉质量和内容对齐方面不断突破。诸如CogVideo、Make-a-Video和ZeroScope等模型在生成逼真、多样化视频方面表现出色，但其性能评估仍主要依赖宏观指标，如FID、FVD和CLIPScore。这些指标在反映模型在不同内容类别和动态场景中的细粒度性能方面存在明显不足，难以指导模型优化。为解决这一问题，本文提出FETV（Fine-grained Evaluation for Text-to-Video），一个多维、多层次的评估体系，结合空间与时间类别，构建了丰富的文本提示分类库。通过自动匹配与人工校正，收集了619个多类别提示，确保数据的多样性和准确性。基于此，研究对四个代表性开源T2V模型进行了细粒度手动评估，揭示了模型在动作、运动、事件顺序等方面的不足，特别是在复杂动态内容和高频细节生成上存在明显差距。同时，作者还评估了自动指标（如CLIPScore和FVD）与人类评价的相关性，发现现有指标相关性偏低，限制了其实际应用价值。为此，作者基于大规模视觉-语言模型（如UMT）开发了改进指标FVD-UMT和UMTScore，显著提升了自动评估的相关性和鲁棒性。这一工作为T2V模型的性能分析提供了系统性工具，有助于推动生成模型在真实场景中的应用与优化。整体而言，FETV不仅丰富了评估体系，也为未来多模态内容生成技术的发展奠定了坚实基础。
Deep Dive
Limitations & Outlook
What gaps remain?
Abstract
Recently, open-domain text-to-video (T2V) generation models have made remarkable progress. However, the promising results are mainly shown by the qualitative cases of generated videos, while the quantitative evaluation of T2V models still faces two critical problems. Firstly, existing studies lack fine-grained evaluation of T2V models on different categories of text prompts. Although some benchmarks have categorized the prompts, their categorization either only focuses on a single aspect or fails to consider the temporal information in video generation. Secondly, it is unclear whether the automatic evaluation metrics are consistent with human standards. To address these problems, we propose FETV, a benchmark for Fine-grained Evaluation of Text-to-Video generation. FETV is multi-aspect, categorizing the prompts based on three orthogonal aspects: the major content, the attributes to control and the prompt complexity. FETV is also temporal-aware, which introduces several temporal categories tailored for video generation. Based on FETV, we conduct comprehensive manual evaluations of four representative T2V models, revealing their pros and cons on different categories of prompts from different aspects. We also extend FETV as a testbed to evaluate the reliability of automatic T2V metrics. The multi-aspect categorization of FETV enables fine-grained analysis of the metrics' reliability in different scenarios. We find that existing automatic metrics (e.g., CLIPScore and FVD) correlate poorly with human evaluation. To address this problem, we explore several solutions to improve CLIPScore and FVD, and develop two automatic metrics that exhibit significant higher correlation with humans than existing metrics. Benchmark page: https://github.com/llyx97/FETV.