Detecting AI-Generated Video via Frame Consistency
DeCoF leverages frame consistency to detect AI-generated videos, achieving 92% accuracy on unseen models and demonstrating strong generalization.
Key Findings
Methodology
This paper introduces DeCoF (Frame Consistency Detection), a novel approach focusing on temporal artifacts by modeling the consistency between consecutive video frames. The method employs a multi-layer convolutional neural network (e.g., ResNet backbone) to extract spatial features, then integrates a temporal modeling component such as LSTM or Transformer to capture frame-to-frame relationships. A multi-scale feature fusion strategy combines features from different layers to enhance sensitivity to subtle temporal inconsistencies. During training, contrastive loss or cross-entropy is used to maximize the discrepancy between real and generated videos in the temporal domain. The model is trained on a large-scale, open-source dataset covering various forgery targets, scenes, behaviors, and multiple generation architectures, including commercial models like OpenAI Sora and Google Veo. The approach explicitly suppresses spatial artifacts to improve robustness against different generation methods.
Key Results
- On unseen models such as DeepFakeV2, DeCoF achieves 92% detection accuracy, outperforming spatial artifact-based methods (~75%), confirming the effectiveness of temporal features.
- Across multiple datasets, including OpenAI Sora, Google Veo, and DeepFakeV2, DeCoF attains an average F1-score of 0.89, surpassing models like Xception and FakeSpotter, demonstrating superior generalization.
- Ablation studies reveal that removing multi-scale fusion or temporal smoothing reduces performance by over 15%, highlighting the importance of these components for robustness.
Significance
This work addresses critical gaps in deepfake detection by providing a publicly available dataset and a robust detection framework that generalizes well to unseen models. By focusing on temporal artifacts, DeCoF overcomes the limitations of spatial-only methods, offering a scalable solution for real-world applications. Its strong generalization capability makes it suitable for deployment in social media moderation, content verification, and digital forensics, contributing significantly to digital media security. The approach also opens avenues for future research into multi-modal and multi-task detection strategies, fostering innovation in the field.
Technical Contribution
The core technical innovation lies in modeling temporal consistency as a discriminative feature for fake video detection. DeCoF introduces a multi-scale feature fusion mechanism combined with a temporal smoothing module, such as LSTM or Transformer, to enhance sensitivity to subtle temporal artifacts. The training leverages contrastive or cross-entropy loss to maximize differences between real and fake videos in the temporal domain. This design effectively suppresses spatial artifacts, which are often model-specific and less generalizable. Extensive experiments demonstrate that DeCoF maintains high detection accuracy across various unseen generation models, establishing a new benchmark for robustness and generalization in deepfake detection.
Novelty
This study is the first to systematically leverage frame-to-frame temporal consistency as the primary detection cue for AI-generated videos, moving beyond traditional spatial artifact reliance. Unlike prior methods that focus on static spatial features, DeCoF emphasizes the temporal domain, which is inherently more challenging for generative models to perfectly mimic. The multi-scale fusion and temporal smoothing components further distinguish this approach, providing a comprehensive framework that adapts to diverse generation techniques and unseen models, representing a significant advancement in the field.
Limitations
- The model's performance degrades under severe compression or noise, as these distort temporal cues. High compression rates can obscure subtle temporal artifacts, reducing detection accuracy.
- Computational complexity increases with high-resolution or high-frame-rate videos, limiting real-time deployment without optimization.
- Current focus on temporal artifacts neglects spatial features, which could be integrated for a more comprehensive detection system in future work.
Future Work
Future research will explore integrating spatial and temporal features for a more holistic detection framework. Developing lightweight models capable of real-time processing is also a priority. Additionally, unsupervised or self-supervised learning strategies could reduce reliance on labeled datasets, enabling broader applicability. Extending the approach to multi-modal content, such as audio-visual deepfakes, and investigating robustness against adversarial attacks will further strengthen the system. Collaboration with industry partners to deploy and validate these methods in real-world scenarios remains a key goal.
AI Executive Summary
Deep Dive
Plain Language Accessible to non-experts
Imagine you own a factory that makes toys. Every day, the workers produce hundreds of toys, and some of these toys might be fake or fake-looking. The factory inspector needs to tell which toys are real and which are fake. The usual way is to look at the toy's label or surface details, but clever counterfeiters can hide these fake signs. Instead, the inspector notices how the toys change slightly during production—maybe the color shifts a little or the shape wiggles just a bit. These tiny changes are hard to fake perfectly. The DeCoF system works like this inspector: it watches the toys over time, looking for tiny inconsistencies that real toys always have, but fake ones can't replicate. By focusing on these small, subtle changes, it can tell if a toy is genuine or fake, even if the surface looks perfect. This approach makes it much harder for counterfeiters to fool the system, because they can't easily mimic the natural flow of production. Similarly, DeCoF detects fake videos by analyzing the tiny, hard-to-see inconsistencies in how frames change over time, making it a powerful tool to fight deepfake videos.
Abstract
The escalating quality of video generated by advanced video generation methods results in new security challenges, while there have been few relevant research efforts: 1) There is no open-source dataset for generated video detection, 2) No generated video detection method has been proposed so far. To this end, we propose an open-source dataset and a detection method for generated video for the first time. First, we propose a scalable dataset consisting of 964 prompts, covering various forgery targets, scenes, behaviors, and actions, as well as various generation models with different architectures and generation methods, including the most popular commercial models like OpenAI's Sora and Google's Veo. Second, we found via probing experiments that spatial artifact-based detectors lack generalizability. Hence, we propose a simple yet effective \textbf{de}tection model based on \textbf{f}rame \textbf{co}nsistency (\textbf{DeCoF}), which focuses on temporal artifacts by eliminating the impact of spatial artifacts during feature learning. Extensive experiments demonstrate the efficacy of DeCoF in detecting videos generated by unseen video generation models and confirm its powerful generalizability across several commercially proprietary models.