Unlocking the Capabilities of Large Vision-Language Models for Generalizable and Explainable Deepfake Detection
Proposes a knowledge-guided LVLM framework achieving 99.53% AUC for deepfake detection, enhancing generalization and explainability.
Key Findings
Methodology
This approach integrates a Knowledge-guided Forgery Detector (KFD), Forgery Prompt Learner (FPL), and a Large Language Model (LLM). KFD calculates correlations between image features and textual embeddings of pristine/deepfake descriptions, enabling classification and localization. FPL converts detection outputs into fine-grained prompts, combining visual and question prompts for the LLM to generate textual responses. Training involves simulated forgery data and multi-turn dialogues to improve robustness and interpretability.
Key Results
- On datasets like FF++, CDF2, DFD, and DFDC, the method surpasses SOTA, with an AUC of 99.53% on FF++, demonstrating excellent generalization. Cross-dataset tests confirm robustness across unseen forgery types.
- The model supports multi-turn dialogue, providing ongoing explanations and localization, which enhances user trust and interpretability.
- Incorporating external knowledge and detailed prompts significantly improves detection of subtle and unknown forgeries, especially in challenging scenarios.
Significance
This work advances deepfake detection by effectively combining external knowledge, multimodal features, and prompt tuning, addressing the key issues of generalization and explainability. It offers a scalable solution for content verification in social media, journalism, and security, facilitating real-world deployment. The multi-turn dialogue capability also opens new avenues for human-AI interaction in content moderation and forensic analysis.
Technical Contribution
The paper introduces a novel knowledge-guided correlation mechanism between image features and textual descriptions, enhancing robustness. It designs a fine-grained forgery prompt embedding system, enabling the LLM to reason about subtle artifacts. The integration of pre-trained multimodal encoders with prompt tuning and multi-task training sets a new standard for explainable, generalizable deepfake detection.
Novelty
This is the first work to embed external knowledge directly into a multimodal detection framework via correlation maps, coupled with a prompt learning strategy that captures subtle forgery cues. Unlike prior methods relying solely on feature extraction or supervised training, this approach leverages external semantic descriptions for improved generalization and interpretability.
Limitations
- The model's performance may decline on extremely compressed or low-quality fake videos due to feature degradation. Its reliance on simulated data might limit effectiveness on real-world unseen forgeries.
- High computational complexity and inference latency could hinder deployment in real-time scenarios. Further optimization is needed for practical use.
- The approach depends on accurate textual descriptions; inaccuracies or ambiguities in descriptions could affect detection accuracy.
Future Work
Future directions include developing unsupervised or semi-supervised training methods to adapt to evolving forgery techniques, expanding external knowledge bases for richer context, and optimizing model efficiency for real-time applications. Additionally, integrating more robust multimodal encoders and exploring zero-shot capabilities could further enhance adaptability.
AI Executive Summary
Deepfake technology has rapidly advanced, enabling highly realistic manipulations of visual content. Traditional detection methods, relying on handcrafted features or supervised learning, struggle to generalize across diverse forgery techniques and scenarios. As a result, there is an urgent need for more robust, explainable, and adaptable detection systems.
This paper introduces a novel framework that leverages large vision-language models (LVLMs) guided by external knowledge to address these challenges. The core components include a Knowledge-guided Forgery Detector (KFD), which aligns visual features with textual descriptions of pristine and fake images, producing consistency maps that highlight potential forgery regions. These maps are then processed by a Forgery Locator and Classifier to perform precise localization and binary classification.
To enhance interpretability and subtle artifact detection, the authors design a Forgery Prompt Learner (FPL) that converts detection outputs into fine-grained prompts. These prompts, combined with visual and question embeddings, are fed into a large language model (LLM) fine-tuned with prompt tuning and LoRA techniques, enabling the system to generate detailed textual explanations and localization cues.
Extensive experiments on datasets such as FF++, CDF2, DFD, and DFDC demonstrate the effectiveness of this approach. The method achieves an AUC of 99.53% on FF++, outperforming existing state-of-the-art models. Cross-dataset evaluations further confirm its strong generalization, especially in unseen forgery types. The multi-turn dialogue capability allows the system to provide continuous explanations, making it suitable for real-world forensic applications.
Overall, this work marks a significant step toward intelligent, explainable, and generalizable deepfake detection. By integrating external knowledge and multimodal reasoning, it addresses core limitations of prior methods, paving the way for more resilient content verification tools. Future efforts will focus on reducing computational costs, enhancing robustness against low-quality forgeries, and expanding zero-shot detection capabilities.
Deep Dive
Plain Language Accessible to non-experts
想象你在一个工厂里，工厂每天生产各种商品。有些商品可能被篡改，变成了假货。工厂的检测员（就像我们的模型）需要通过观察商品的细节，判断它们是否被篡改。传统的方法就像用放大镜，只看表面特征，但有些假货做得非常逼真，难以识别。现在，我们引入了一个聪明的助手（知识引导模型），它不仅用放大镜，还能参考工厂的知识库，比如“假货通常会有颜色不自然或者细节不一致”。这个助手还能和工厂的检测员多次交流，逐步确认商品的真伪。这样一来，不仅提高了检测的准确率，还能解释为什么判定为假货。这个系统就像一个智能的工厂检测线，能在各种复杂场景下识别假货，确保商品的真实性。
ELI14 Explained like you're 14
你知道在学校里，有时候有人会用照片或者视频做假，骗别人相信一些不是真的事情。比如有人用特殊软件把一个人的脸换成别人的，或者让视频看起来像是真的，但其实是假的。现在的技术变得非常厉害，假视频看起来和真的一样，让人很难分辨出来。科学家们想要开发一种聪明的检测工具，就像一个超级侦探，能找到这些假视频的线索。这个侦探不仅用眼睛观察，还能参考很多知识，比如“假视频通常会有颜色不自然或者脸部细节不一致”。它还能和你多次交流，告诉你哪里可能有问题。这样一来，我们就能更快、更准确地找到假视频，保护大家不被误导。这就像在学校里，有个特别聪明的老师，能帮你识别出那些伪装得很像真的作业，确保大家的学习环境真实可靠。
Abstract
Current Large Vision-Language Models (LVLMs) have demonstrated remarkable capabilities in understanding multimodal data, but their potential remains underexplored for deepfake detection due to the misalignment of their knowledge and forensics patterns. To this end, we present a novel framework that unlocks LVLMs' potential capabilities for deepfake detection. Our framework includes a Knowledge-guided Forgery Detector (KFD), a Forgery Prompt Learner (FPL), and a Large Language Model (LLM). The KFD is used to calculate correlations between image features and pristine/deepfake image description embeddings, enabling forgery classification and localization. The outputs of the KFD are subsequently processed by the Forgery Prompt Learner to construct fine-grained forgery prompt embeddings. These embeddings, along with visual and question prompt embeddings, are fed into the LLM to generate textual detection responses. Extensive experiments on multiple benchmarks, including FF++, CDF2, DFD, DFDCP, DFDC, and DF40, demonstrate that our scheme surpasses state-of-the-art methods in generalization performance, while also supporting multi-turn dialogue capabilities.