AntifakePrompt: Prompt-Tuned Vision-Language Models are Fake Image Detectors
AntifakePrompt leverages prompt tuning on InstructBLIP, boosting deepfake detection accuracy from 71.06% to 92.11% on unseen models.
Key Findings
Methodology
This work formulates deepfake detection as a visual question answering (VQA) task, utilizing pretrained InstructBLIP. By inserting a pseudo-word S* into a prompt, only the embedding of S* is tuned via language modeling loss, enabling the model to distinguish real from fake images across unseen generative models. The approach leverages the zero-shot capabilities of VLMs, combining prompt engineering with minimal parameter tuning. Experiments involve diverse datasets from 3 known and 20 unknown models, covering text-to-image, editing, and adversarial attacks, validating the method's robustness and generalization.
Key Results
- On a comprehensive test set including unseen models, detection accuracy improved from 71.06% to 92.11%, representing a significant enhancement in generalization performance.
- Across various tasks such as diffusion, inpainting, super-resolution, and face swapping, the method consistently outperformed baseline detectors, especially on models not seen during training.
- Training cost was minimized by tuning only the pseudo-word embedding (~4,864 parameters), with training completed within 10 hours on modest hardware, demonstrating high efficiency.
Significance
This research addresses the critical challenge of generalizing deepfake detection to emerging models. By exploiting the zero-shot and instruction-following abilities of VLMs, it offers a scalable, low-cost, and highly adaptable solution. This approach enhances digital content integrity, supports copyright enforcement, and provides a foundation for future multimodal security systems, marking a significant step forward in AI safety and content verification.
Technical Contribution
The core innovation lies in transforming deepfake detection into a VQA problem, employing prompt tuning to adapt pretrained VLMs with minimal parameters. This method departs from traditional feature-based or frequency domain techniques, offering a lightweight yet powerful alternative. The approach demonstrates how instruction tuning can significantly improve zero-shot generalization, opening new avenues for multimodal security applications.
Novelty
This is the first work to formalize deepfake detection as a VQA task with prompt tuning, leveraging the instruction-following capabilities of large vision-language models. Unlike prior methods relying on handcrafted features or domain-specific artifacts, this approach uses minimal parameter tuning to achieve broad generalization, representing a paradigm shift in deepfake detection.
Limitations
- The method depends on the quality and robustness of the pretrained VLM; biases or limitations in the base model can affect detection accuracy.
- While minimal tuning reduces costs, the approach still requires labeled datasets for pseudo-word optimization, which may limit scalability in some scenarios.
- Real-time deployment and high-resolution image processing pose computational challenges that need further optimization.
Future Work
Future directions include integrating multimodal cues such as audio and text, developing adaptive prompt tuning strategies for even better generalization, and extending the framework to video deepfake detection. Additionally, exploring unsupervised or semi-supervised tuning could further reduce labeling costs and improve robustness against adversarial manipulations.
AI Executive Summary
The rapid advancement of deep generative models like Stable Diffusion and DALL·E has revolutionized image synthesis, enabling the creation of highly realistic fake images. While these technologies have democratized content creation, they also pose significant threats related to misinformation, copyright infringement, and societal manipulation. Existing detection methods, primarily based on visual artifacts or frequency analysis, struggle to generalize to unseen models, limiting their effectiveness in real-world scenarios.
To address this challenge, this study introduces AntifakePrompt, a novel deepfake detection framework that exploits the zero-shot capabilities of pretrained vision-language models (VLMs), specifically InstructBLIP. By formulating the detection task as a visual question answering (VQA) problem, the approach asks the model whether an image is real or fake through a carefully designed prompt. The key innovation involves inserting a pseudo-word S* into the prompt and only tuning its embedding, drastically reducing training complexity while maintaining high adaptability.
Experimental results demonstrate that this method significantly outperforms traditional classifiers, achieving an average accuracy of 92.11% on unseen datasets, compared to 71.06% baseline. The approach is robust across diverse tasks, including diffusion, inpainting, super-resolution, and face swapping, confirming its broad applicability. The low training cost and minimal parameter tuning make it suitable for large-scale deployment, providing a scalable solution to emerging deepfake threats.
This work highlights the potential of leveraging instruction-following VLMs for security applications, paving the way for more resilient and efficient content verification systems. Future research will focus on multimodal integration and real-time detection, aiming to further enhance robustness and operational efficiency in dynamic environments. Overall, AntifakePrompt offers a promising direction for safeguarding digital authenticity amidst rapidly evolving generative AI technologies.
Deep Analysis
Background
近年来，深度生成模型如GAN、扩散模型（如Stable Diffusion）极大推动了图像合成技术的发展，从艺术创作到虚假内容制造。尽管这些模型带来便利，但也引发了深伪威胁，包括虚假新闻、版权侵权和社会操控。传统检测方法多依赖于特征工程或频域分析，效果在已知模型上表现优异，但面对新兴模型时泛化能力不足。近年来，预训练的视觉-语言模型（VLM）如BLIP、InstructBLIP展现出强大的跨模态理解能力，为深伪检测提供了新思路。
Core Problem
现有深伪检测技术在泛化能力方面存在瓶颈，难以应对不断涌现的高质量生成模型。传统方法多依赖于特定模型的特征或频域特征，容易被新型伪造技术绕过。如何利用预训练模型的零样本能力，设计低成本且具有强泛化能力的检测方案，成为亟待解决的问题。尤其是在多任务、多场景环境下，检测模型的鲁棒性和效率尤为关键。
Innovation
本研究的创新点包括：1) 将深伪检测转化为视觉问答（VQA）任务，充分利用VLM的跨模态理解能力；2) 引入prompt调优技术，通过调节伪词嵌入实现模型定制，显著降低训练成本；3) 只调优伪词参数，避免大规模参数调节，提高效率；4) 在多源、多任务数据上验证模型的泛化能力，展现出优异的性能和鲁棒性。这些创新突破了传统检测的局限，为深伪检测提供了新范式。
Methodology
- �� 设计问句“此图真实否？”作为检测指令。• 在问句中插入伪词S*，作为调优目标。• 只调优S*的词嵌入参数，冻结模型其他部分。• 使用多源真实和伪造图像数据进行训练，确保模型在未知模型上的泛化能力。• 采用语言模型的语言建模损失（如交叉熵）优化伪词嵌入。• 在不同生成任务（如文本转图、编辑、对抗）上进行验证，确保模型鲁棒性。• 通过调节伪词位置（前缀、后缀或替换）优化检测效果。• 结合多源数据集，评估模型在不同场景的表现。• 实验中只调优少量参数（约4,864个），显著降低训练成本。
Experiments
采用多源数据集，包括COCO、Flickr30k等真实图像，及多种生成模型（如SD2、SDXL、DALLE-2、DALLE-3、ControlNet等）生成的伪造图像，涵盖文本转图、编辑、对抗攻击等场景。测试集由未见模型组成，确保评估泛化能力。对比基线包括ResNet、FatFormer、DE-FAKE等，采用准确率、F1值等指标。调优参数包括伪词位置、调优范围（仅词嵌入）、训练轮次（最多10轮），验证不同配置的效果。实验还包括不同训练数据规模（15K-180K）对性能的影响分析。
Results
模型在未见模型上的平均检测准确率从71.06%提升至92.11%，提升幅度显著。不同生成任务中，检测性能均优于传统方法，尤其在高质量扩散模型上表现优异。调优成本低，参数调节仅涉及伪词嵌入，训练时间短（约10小时），资源消耗少。调优策略（位置、调优对象）对性能影响有限，但整体效果优越。模型在多场景、多模型环境中展现出强鲁棒性，验证了方法的实用性。
Applications
该方法适用于社交媒体、新闻验证、版权保护等场景，能够快速识别伪造内容，保护信息真实性。只需少量调优即可部署在不同平台，适合大规模应用。未来可结合实时检测系统，辅助内容审核和自动化监控，提升数字内容安全水平。
Limitations & Outlook
模型依赖预训练VLM的表现，若基础模型偏差或偏向特定伪造类型，检测效果可能受影响。调优过程仍需一定标注数据，且对极端复杂或高质量伪造样本的检测能力有待验证。在极端资源有限或实时检测场景中，模型部署可能面临计算瓶颈。未来需优化调优策略和模型结构，以适应更复杂的应用环境。
Plain Language Accessible to non-experts
想象你是一名工厂的质量检验员，工厂每天生产各种商品。有些商品可能被篡改或伪造，比如假冒的手机或衣服。传统的检验员会通过看外观、检测标签等方式判断，但这些方法在面对新型伪造品时常常失效。现在，科学家们开发了一种智能检测助手，就像一位聪明的侦探，它可以通过问一句话，比如“这个商品是真的还是假的？”来判断商品的真伪。这个助手会学习大量真实和伪造的商品图片，调整自己的“提问方式”，让自己变得更聪明。只需要让它调节几个关键词，它就能在面对新伪造品时，也能准确识别出来。这样一来，无论伪造技术如何变得更复杂，这个助手都能帮你识别出来，保护你的工厂和消费者的权益。
ELI14 Explained like you're 14
想象你在学校里有一个超级聪明的朋友，他能通过看一张图片就告诉你它是真的还是假的。以前，我们需要用很多复杂的方法来判断图片的真假，但这些方法只对以前见过的伪造图片有效。现在，这个朋友用了一种特别聪明的技巧，他会问自己一句话，比如“这张图片是真的吗？”然后，他会调整自己问话的方式，让自己变得更聪明。只要他学会了怎么问，他就可以在遇到新型的假图片时，也能准确判断出来。这个技巧就像是给他加了一点“魔法词”，让他变得更厉害。这样一来，无论未来出现什么新伪造图片，他都能帮你识别，保护大家不被假货骗。
Abstract
Deep generative models can create remarkably photorealistic fake images while raising concerns about misinformation and copyright infringement, known as deepfake threats. Deepfake detection technique is developed to distinguish between real and fake images, where the existing methods typically learn classifiers in the image domain or various feature domains. However, the generalizability of deepfake detection against emerging and more advanced generative models remains challenging. In this paper, being inspired by the zero-shot advantages of Vision-Language Models (VLMs), we propose a novel approach called AntifakePrompt, using VLMs (e.g., InstructBLIP) and prompt tuning techniques to improve the deepfake detection accuracy over unseen data. We formulate deepfake detection as a visual question answering problem, and tune soft prompts for InstructBLIP to answer the real/fake information of a query image. We conduct full-spectrum experiments on datasets from a diversity of 3 held-in and 20 held-out generative models, covering modern text-to-image generation, image editing and adversarial image attacks. These testing datasets provide useful benchmarks in the realm of deepfake detection for further research. Moreover, results demonstrate that (1) the deepfake detection accuracy can be significantly and consistently improved (from 71.06% to 92.11%, in average accuracy over unseen domains) using pretrained vision-language models with prompt tuning; (2) our superior performance is at less cost of training data and trainable parameters, resulting in an effective and efficient solution for deepfake detection. Code and models can be found at https://github.com/nctu-eva-lab/AntifakePrompt.