E-VAds: An E-commerce Short Videos Understanding Benchmark for MLLMs
E-VAds-R1 achieves 109.2% improvement in commercial intent reasoning using MG-GRPO reward design.
Key Findings
Methodology
Introduced E-VAds benchmark with 3,961 high-quality e-commerce videos and 19,785 open-ended QA pairs, leveraging multimodal density assessment and MG-GRPO reward design in E-VAds-R1.
Key Results
- E-VAds-R1 improves commercial intent reasoning by 109.2% compared to general-purpose baselines with only a few hundred samples.
- Multimodal density metrics show E-VAds' visual dynamic density (Vden) at 60.44, significantly higher than mainstream datasets.
- Cross-modal tasks demonstrate strong reasoning performance combining OCR and ASR cues.
Significance
Addresses the lack of benchmarks for e-commerce short video understanding, advancing multimodal models in dense signal environments and providing new pathways for commercial reasoning.
Technical Contribution
Introduced MG-GRPO reward design combining strict and relaxed scoring standards to optimize sparse rewards; developed multimodal density assessment framework quantifying visual, audio, and textual complexity.
Novelty
First dedicated benchmark for e-commerce short videos, introducing multimodal density metrics and multi-grained reward design to tackle sparse supervision challenges.
Limitations
- Limited robustness in handling extreme noise or redundant signals, potentially affecting reasoning accuracy.
- Dataset primarily sourced from Chinese platforms, introducing regional bias.
Future Work
Future directions include expanding to international platforms for cross-cultural reasoning and improving robustness against extreme noise scenarios.
AI Executive Summary
E-commerce short videos pose unique challenges due to their goal-driven format and dense multimodal signals, yet existing models lack benchmarks tailored for commercial reasoning.
E-VAds fills this gap, offering 3,961 curated videos and 19,785 QA pairs across five task types. Using a multimodal density assessment framework, it quantifies visual, audio, and textual complexity, setting a new standard for model evaluation.
The RL-based E-VAds-R1 model leverages MG-GRPO reward design to achieve a 109.2% improvement in commercial intent reasoning, demonstrating strong performance under sparse supervision. This study opens new avenues for e-commerce video understanding and multimodal AI development.
Deep Analysis
Background
E-commerce short videos have emerged as a major segment in online video, characterized by dense multimodal signals and conversion-driven goals. Existing datasets like ActivityNetQA and AdsQA focus on general video QA or brand ads, leaving e-commerce videos underexplored.
Core Problem
E-commerce videos' multimodal density and open-ended commercial reasoning make them challenging for current models. The absence of dedicated benchmarks hinders systematic evaluation of model performance.
Innovation
Introduced E-VAds benchmark tailored for e-commerce short videos; developed MG-GRPO reward design to address sparse RL rewards; proposed multimodal density metrics to quantify visual, audio, and textual complexity.
Methodology
- �� Data collection: Filtered 3,961 high-quality videos from Taobao using dynamic sampling for balanced categories.
- �� Multimodal processing: ASR transcription with Whisper-v3-large, OCR extraction with Qwen2.5-VL, and temporal alignment.
- �� QA generation: Multi-agent system creates open-ended QA pairs with manual review.
- �� Reinforcement learning: MG-GRPO reward mechanism combines strict and relaxed scoring to optimize learning.
Experiments
Experiments used E-VAds dataset, with 1,980 QA pairs for training and 16,384 for testing. Baselines included Video-LLaVA and AdsQA, evaluated on commercial reasoning accuracy.
Results
E-VAds-R1 achieved 109.2% improvement in commercial reasoning; multimodal density metrics confirmed E-VAds' higher complexity; cross-modal tasks showed strong performance combining OCR and ASR cues.
Applications
Applicable for e-commerce platforms to analyze and optimize short video ads; useful for testing and improving multimodal model performance.
Limitations & Outlook
Limited robustness against extreme noise or redundant signals; dataset sourced from Chinese platforms, introducing regional bias.
Plain Language Accessible to non-experts
Imagine you're browsing Amazon and see a short video ad for a dishwasher. The video shows the machine in action, highlights its features with text on the screen, and includes a voiceover explaining why it's better than others. E-VAds is a tool that helps AI understand such videos, combining visuals, text, and audio to answer questions like 'Who is this product for?' or 'Is the ad compliant with regulations?'.
ELI14 Explained like you're 14
Ever seen a short ad on TikTok for a cool gadget? Like a dishwasher that claims '52,000 Pa water pressure' while showing it cleaning dishes super fast? Scientists made a tool called E-VAds that helps AI understand these ads, figuring out stuff like 'How does this ad convince people to buy?' or 'Is it telling the truth?'. Pretty awesome, right?
Glossary
E-VAds Benchmark
A dataset designed for e-commerce short video understanding, containing multimodal signals and open-ended QA pairs.
Used to evaluate models on commercial reasoning tasks.
MG-GRPO Reward Design
A reinforcement learning reward mechanism combining strict and relaxed scoring standards to optimize sparse rewards.
Applied in training E-VAds-R1 model for reasoning tasks.
Multimodal Density Assessment
Framework quantifying visual, audio, and textual signal complexity in datasets.
Used to compare E-VAds with other benchmarks.
ASR Transcription
Automatic Speech Recognition technology converting audio into text.
Extracts speech content from e-commerce videos.
OCR Text Extraction
Optical Character Recognition technology extracting text from video frames.
Analyzes textual signals in e-commerce videos.
Open Questions Unanswered questions from this research
- 1 How to adapt models for cross-cultural commercial reasoning?
- 2 How to enhance robustness against extreme noise scenarios?
Applications
Immediate Applications
Ad Optimization
Helps platforms analyze short video ads to improve conversion rates.
Multimodal Model Evaluation
Provides dense datasets for testing model performance.
Long-term Vision
Global E-Commerce Adaptation
Expands to international platforms for multilingual, cross-cultural reasoning.
Abstract
E-commerce short videos represent a high-revenue segment of the online video industry characterized by a goal-driven format and dense multi-modal signals. Current models often struggle with these videos because existing benchmarks focus primarily on general-purpose tasks and neglect the reasoning of commercial intent. In this work, we first propose a multi-modal information density assessment framework to quantify the complexity of this domain. Our evaluation reveals that e-commerce content exhibits substantially higher density across visual, audio, and textual modalities compared to mainstream datasets, establishing a more challenging frontier for video understanding. To address this gap, we introduce E-commerce Video Ads Benchmark, which is the first benchmark specifically designed for e-commerce short video understanding. We curated 3,961 high-quality videos from Taobao covering a wide range of product categories and used a multi-agent system to generate 19,785 open-ended Q&A pairs, which consist of five distinct tasks. Finally, we develop E-VAds-R1, an RL-based reasoning model featuring a multi-grained reward design called MG-GRPO. This strategy provides smooth guidance for early exploration while creating a non-linear incentive for expert-level precision. Experimental results demonstrate that E-VAds-R1 achieves a 109.2% performance gain in commercial intent reasoning with only a few hundred training samples. Data is available at https://github.com/TaobaoTmall-AlgorithmProducts/E-VAds_Benchmark.