Dataset Biases and Shortcut Learning in Motion-Based AI-Generated Video Detection
This study evaluates four motion-based AI video detectors, revealing biases and shortcut learning, with frequency-based methods showing better robustness.
Key Findings
Methodology
The paper systematically analyzes four recent motion-based detectors (D3, ReStraV, Over-Coherence, NSG-VD), focusing on biases from data preprocessing, sampling, and dataset composition. By reproducing these methods with original code and datasets, it uncovers how biases—such as frame rate discrepancies, resolution artifacts, and compression—are exploited. The study applies bias removal techniques, data augmentation, and compares with a frequency-based detector, assessing performance shifts to evaluate robustness and generalization.
Key Results
- Results show that on biased datasets, detectors achieve near-perfect performance (AUC > 97%), but performance drops significantly (average AUC decrease of 0.1736) after bias removal, approaching random levels. The frequency-based detector maintains high performance (AUC > 0.96) across all datasets, demonstrating superior robustness.
- Sampling biases like inconsistent frame rates (e.g., 3 fps vs 25-30 fps) cause detectors to rely on non-semantic cues, inflating performance artificially. Removing such biases reduces detection accuracy, exposing overfitting to dataset artifacts.
- Simple spatial augmentations (e.g., slight cropping, translation) severely degrade detection results, indicating models' reliance on superficial cues. In contrast, frequency features are less affected, highlighting their potential for more generalizable detection.
Significance
This work exposes critical vulnerabilities in current motion-based AI video detectors, emphasizing how dataset biases and shortcut learning inflate performance metrics. It advocates for more representative, unbiased datasets and robust evaluation protocols. The findings guide future research toward frequency-based features, which offer promising avenues for developing reliable, real-world applicable detection systems, essential for combating misinformation and digital forgery.
Technical Contribution
The paper provides a comprehensive bias analysis in motion-based detectors, demonstrating how sampling, preprocessing, and dataset biases influence results. It introduces a frequency domain detection approach that outperforms existing methods in robustness. The study also offers a framework for bias identification and mitigation, contributing to the theoretical understanding of model vulnerabilities and practical detection strategies.
Novelty
This is the first systematic investigation into dataset biases and shortcut learning in AI-generated video detection, especially focusing on motion-based methods. It highlights the limitations of current benchmarks and proposes frequency analysis as a more reliable alternative, marking a significant step forward in the field.
Limitations
- Analysis primarily relies on publicly available datasets and specific detectors; real-world scenarios may involve more complex biases and artifacts.
- Frequency-based detection, while robust, may face challenges under extreme compression or noise conditions not covered in this study.
- Further research is needed to integrate multi-modal cues and deep features for comprehensive detection, especially in dynamic, real-world environments.
Future Work
Future directions include developing multi-modal detection frameworks combining visual, audio, and contextual cues, constructing unbiased, diverse datasets, and establishing standardized evaluation protocols. Additionally, exploring deep learning models that incorporate frequency features and robustness against adversarial manipulations will be crucial for real-world deployment.
AI Executive Summary
The rapid evolution of AI-generated videos has created a pressing need for reliable detection methods. Existing state-of-the-art detectors based on motion cues, such as D3, ReStraV, Over-Coherence, and NSG-VD, have demonstrated impressive results on specific datasets. However, this performance often hinges on dataset biases—sampling strategies, resolution artifacts, compression signatures—that detectors exploit as shortcuts. When these biases are removed or datasets are balanced, detection accuracy plummets, revealing that much of the reported success is illusory.
This study conducts a thorough bias analysis, revealing that many detectors rely on superficial cues linked to dataset artifacts rather than genuine motion inconsistencies. For example, sampling videos at different frame rates introduces artificial motion patterns, which detectors learn to identify. Similarly, resolution and compression artifacts serve as unintended signals, misleading models into high accuracy. By applying bias removal techniques and data augmentation, the authors demonstrate a significant performance decline, exposing the overfitting to dataset-specific cues.
In contrast, a frequency-based detection approach, analyzing spectral features of videos, maintains high performance across diverse datasets, indicating better generalization. This suggests a promising direction for future research, emphasizing the importance of feature robustness over dataset-specific cues. The findings underscore the urgent need for unbiased datasets and standardized evaluation protocols, to ensure detection methods are truly effective in real-world scenarios. Ultimately, this work advances the understanding of vulnerabilities in current detection frameworks and points toward more reliable, bias-resistant solutions for AI-generated video identification.
Deep Analysis
Background
随着生成式视频模型的不断发展，AI视频的逼真度显著提升，带来了内容真实性的挑战。早期研究多集中在深度伪造（Deepfake）检测，依赖面部细节和微妙的不一致性。近年来，扩散模型等新技术推动了更通用的视频生成，检测方法也逐渐多样化，包括光流分析、频域特征、预训练特征和多模态融合。然而，这些方法在实际应用中面临泛化不足的问题，尤其是数据偏差和模型对偏差的依赖成为新的难题。偏差源于采样策略、预处理环节、数据集构建不平衡，导致模型学习到非本质特征，影响检测的真实性和可靠性。
Core Problem
现有运动基础检测器在特定数据集表现优异，但其性能高度依赖偏差，缺乏对真实场景的鲁棒性。偏差主要来自采样（如帧率不一致）、预处理（如分辨率、压缩）和数据集中的运动偏差。这些偏差被模型利用，导致在无偏或真实场景中性能大幅下降，限制了实际应用的可靠性。解决这一问题，要求深入理解偏差源，设计更稳健的检测策略，避免模型依赖非本质特征。
Innovation
本研究的创新点包括：
1）系统分析运动检测器中的偏差源，揭示采样、预处理和数据偏差对性能的影响；
2）提出去偏和数据增强策略，有效降低偏差对检测性能的干扰；
3）引入频率域特征检测器，展现其在不同数据集上的优越鲁棒性。创新在于将偏差分析与频率特征结合，为检测模型提供更稳健的特征基础，突破传统运动特征的局限。
Methodology
- �� 复现四个检测器（D3、ReStraV、Over-Coherence、NSG-VD）在原始数据集上的性能，分析偏差源；
- �� 调整采样策略（如帧率、时长），观察性能变化；
- �� 采用去偏技术（如平衡帧率、分辨率），验证偏差影响；
- �� 通过空间增强（裁剪、平移）测试模型对偏差的敏感性；
- �� 提取频率特征，训练频率检测器，评估其鲁棒性。
Experiments
使用GenVideo和VidProM等公开数据集，复现检测器，进行偏差消除和增强实验。指标包括AUC和AP，比较偏差前后性能变化。引入频率特征检测器作为对比，验证其在偏差环境中的表现。采用交叉验证和统计分析确保结论的稳健性。实验还分析不同偏差类型（采样、分辨率、压缩）对检测效果的影响。
Results
偏差消除后，运动检测器性能平均下降0.17左右，表现接近随机。频率检测器在所有偏差条件下保持高性能（AUC>0.96），验证其鲁棒性。采样偏差（如帧率差异）被模型利用，导致虚假高性能。空间增强显著影响检测效果，揭示模型对偏差的敏感性。结果表明，偏差是虚假性能的主要来源，频率特征提供了更稳健的检测途径。
Applications
该研究推动构建更可靠的AI视频检测系统，适用于内容审核、虚假信息识别、数字取证等场景。强调数据集无偏、模型稳健的重要性，为行业制定标准提供依据。未来结合多模态信息，将提升检测的全面性和实用性，适应复杂多变的实际环境。
Limitations & Outlook
分析主要基于公开数据集和特定检测器，实际场景中的偏差可能更复杂。频率检测在极端压缩或噪声环境下的鲁棒性尚未充分验证。模型对多模态融合和深层特征的依赖仍需探索。未来应考虑更复杂的偏差类型和多源信息，提升检测的适应性和实用性。
Plain Language Accessible to non-experts
想象你在一家工厂里，生产各种商品。检测员的任务是判断商品是否合格，但他们依赖一些表面特征，比如颜色或包装。实际上，有些不合格的商品会模仿这些表面特征，骗过检测员。这个检测系统就像那些运动检测器，它们依赖某些运动模式或细节，但这些模式可能被伪造或偏差所利用。研究发现，这些检测器其实是在“看假象”，而不是判断内容的本质。通过分析和改进检测方法，比如用“频率分析”检测视频中的深层细节，就能更准确地识别真假商品。这就像用更科学的方法，避免被表面特征欺骗，确保每个商品都是真的、合格的。
ELI14 Explained like you're 14
想象你在学校，有个老师用特殊的眼镜判断谁在作弊。开始时，老师只看动作，比如抖腿或偷看，但后来发现学生可以用假动作骗过老师。老师的眼镜其实依赖一些“偏差”，比如学生平时动作少，作弊时动作多。可是，这样的判断不可靠，因为学生可以用假动作骗过老师。科学家们也遇到类似问题，他们开发的检测器也会依赖一些特定的运动特征，但这些特征可能被伪造或偏差利用。研究发现，用“频率分析”检测视频中的细节，比只看运动更可靠。这就像老师用更聪明的眼镜，能看出学生的真正意图，而不是被假动作迷惑。未来，这些技术会让我们更容易识别虚假视频，保护我们免受虚假信息的影响。
Abstract
The visual quality of AI-generated videos has improved drastically in recent years, making it increasingly difficult for humans to distinguish between real and synthetic media. In this work, we evaluate the robustness and applicability of four state-of-the-art motion-based AI-generated video detectors. We identify significant preprocessing and sampling biases in these methods and demonstrate that they account for a substantial portion of their reported performance. Furthermore, we find that these detectors are highly sensitive to motion patterns specific to their evaluation datasets, where AI-generated videos generally exhibit less inter-frame movement than real videos. We show that for all detectors, performance collapses to near-random levels when evaluated on a dataset that does not contain this motion bias. Additionally, through dataset rebalancing and the application of simple spatial augmentations, we observe severe performance degradation across all evaluated models. In contrast, we find that an existing frequency-based detector maintains strong performance across all evaluated datasets, suggesting that frequency-based approaches may offer a more generalizable path forward for AI-generated video detection. We hope that our work raises awareness towards these vulnerabilities and encourages the development of more representative, unbiased datasets and more robust evaluation protocols.