On the Holistic Approach for Detecting Human Image Forgery
HuForDet employs dual-branch architecture combining RGB/frequency face analysis and multimodal semantic consistency, achieving SOTA detection with 90.22% AUC.
Key Findings
Methodology
HuForDet features a dual-branch design: a face forgery detection branch integrates heterogeneous experts operating in RGB and frequency domains, including an adaptive LoG module to capture multi-scale artifacts; a full-body semantic analysis branch leverages a large multimodal language model (MLLM) combining visual encoding (e.g., CLIP) and language reasoning (e.g., Vicuna). The frequency experts use learnable multi-scale LoG approximations to detect fine-grained blending boundaries and coarse irregularities. The semantic branch generates high-level forgery representations and self-assessed confidence scores, which guide the fusion process. The fusion module dynamically weights contributions based on confidence, enhancing robustness against subtle artifacts.
Key Results
- On the newly curated HuFor dataset, HuForDet achieves 90.22% AUC, outperforming prior methods like NPR (87.75%) and M2F2-Det (86.73%). Its low false-positive detection rates (TPR95 of 70.87%, TPR99 of 33.45%) demonstrate strong robustness. In face manipulation subsets (FF++), detection reaches 87.80% AUC, surpassing many existing models. Ablation studies confirm the effectiveness of multi-scale LoG experts and multimodal fusion, with significant improvements over baseline architectures.
- The model's architecture allows it to detect both partial face swaps and full-body synthetic images, demonstrating excellent generalization. The frequency experts effectively identify artifacts at different scales, while the semantic analysis captures structural inconsistencies. The confidence-guided fusion reduces false alarms, especially in challenging scenarios.
- Experimental results show that combining frequency domain features with high-level semantic cues significantly boosts detection accuracy, setting new state-of-the-art benchmarks across multiple forgery types and datasets.
Significance
This work advances deepfake detection by integrating local frequency features with global semantic reasoning, addressing the limitations of prior single-modality approaches. The holistic framework enhances detection robustness across diverse forgery techniques, including subtle manipulations and full-body syntheses. Its high accuracy and generalization capacity make it suitable for real-world applications like media verification, legal evidence, and social media moderation. The innovative fusion of visual and language models opens new avenues for AI-based forensic tools, contributing to the integrity of digital content in an era of rapidly evolving generative AI.
Technical Contribution
The paper introduces a novel multi-scale, learnable LoG frequency expert module, enabling adaptive detection of artifacts at various spatial scales. It also develops a dual-branch architecture that fuses local frequency features with global semantic cues, guided by confidence scores derived from a large multimodal language model. This approach surpasses existing methods by effectively capturing subtle forgery traces and structural inconsistencies. The integration of visual encoding with language reasoning provides a new paradigm for forensic AI, offering both interpretability and robustness. The dataset HuFor, combining face and full-body forgeries, further supports comprehensive evaluation and training.
Novelty
This is the first work to combine multimodal large language models with frequency domain analysis for holistic human image forgery detection. Unlike prior methods focusing solely on local or global features, HuForDet unifies both through a confidence-guided fusion mechanism, enabling detection of a wider range of manipulations. Its adaptive multi-scale LoG modules and multi-branch design represent significant innovations, setting new standards in the field.
Limitations
- Despite high accuracy, the model struggles with extremely low-quality or heavily compressed images, where subtle artifacts are masked. Its computational complexity, especially in the frequency expert modules, limits real-time deployment. Additionally, the model's effectiveness against emerging forgery techniques that bypass current artifact patterns remains to be validated. Future work should focus on optimizing efficiency and enhancing adaptability to novel forgery methods.
Future Work
Future directions include developing more efficient frequency analysis modules for real-time applications, extending the framework to video forgery detection, and incorporating self-supervised learning to improve generalization. Further, integrating additional modalities like audio or temporal cues could strengthen detection robustness. Addressing adversarial attacks and continuously updating the dataset with new forgery types will be crucial for maintaining state-of-the-art performance.
AI Executive Summary
In recent years, deepfake technology has rapidly evolved, enabling the synthesis of highly realistic human images and videos. Traditional detection methods, often limited to facial regions or relying solely on either spatial or frequency features, struggle to keep pace with sophisticated forgery techniques. This gap necessitated a new, holistic approach capable of addressing the full spectrum of human image manipulations.
HuForDet emerges as a comprehensive solution, integrating a dual-branch architecture that combines detailed facial analysis with global semantic consistency checks. The face forgery detection branch employs heterogeneous experts operating in both RGB and frequency domains, including a novel adaptive LoG module that learns multi-scale features to detect subtle blending artifacts and structural irregularities. Simultaneously, the contextualized forgery detection branch leverages a large multimodal language model, which processes visual tokens and generates reasoning-based confidence scores, guiding the fusion of local and global features.
Experimental results on the newly curated HuFor dataset demonstrate the method's superiority, achieving 90.22% AUC—outperforming existing approaches by a significant margin. The model excels at detecting both partial face swaps and full-body synthetic images, with robust performance across diverse forgery types. Ablation studies confirm the importance of multi-scale frequency experts and confidence-guided fusion, highlighting the technical innovations that drive these improvements.
This work marks a significant step forward in digital media forensics, providing a versatile, interpretable, and highly accurate detection framework. Its ability to generalize across forgery types and scales makes it a promising tool for applications in media verification, legal evidence, and social media moderation. Nonetheless, challenges remain in optimizing computational efficiency and adapting to future forgery techniques, guiding future research directions toward more scalable and adaptive solutions.
Deep Analysis
Background
深度伪造技术的快速发展带来了前所未有的挑战。从早期的面部交换到现代的全身合成，生成模型如StyleGAN、Diffusion模型不断提升逼真度，推动虚假内容的泛化。代表性工作如FaceForensics++、Celeb-DF、DeepFake检测方法，主要集中于局部面部操控，依赖于特征提取和频域分析，难以应对全景式伪造。近年来，结合视觉与语言的多模态模型逐渐兴起，试图提升检测鲁棒性，但在细粒度识别和多尺度特征融合方面仍存不足。随着生成模型不断演变，检测技术亟需融合多模态、多尺度信息，提升对复杂伪造的识别能力。
Core Problem
现有检测方法多局限于面部区域或全身合成，缺乏统一的全景检测框架。面部操控检测依赖局部特征，易受压缩和噪声影响；全身合成检测则依赖语义一致性，难以捕获微妙的操控痕迹。两者结合不足导致检测鲁棒性不足，难以应对复杂、多样的伪造场景。尤其是在微妙伪造和高压缩环境下，现有模型表现不佳。如何融合频域多尺度特征与全局语义信息，提升检测准确率和泛化能力，成为核心问题。
Innovation
本研究的创新点包括：1）引入可学习的多尺度LoG频域专家，动态捕获不同尺度的伪造痕迹，优于固定尺度方法；2）设计双分支架构，面部分支融合RGB与频域专家，增强局部操控检测能力；3）利用多模态大模型结合视觉编码与语言推理，分析全身结构异常，输出置信度，提升鲁棒性；4）提出置信度引导融合机制，根据模型自我评估调节特征贡献。这些创新共同实现了面部操控与全身合成的同步检测，显著优于现有技术。
Methodology
- �� 输入：图像I，裁剪面部区域Iface；• 面部分支：采用四个异质专家（两个空间域、两个频域）分析Iface，利用可学习的多尺度LoG近似捕获不同尺度的伪造痕迹；• 频域专家：通过自适应LoG模块，学习不同尺度的高频与低频特征，增强细粒度与大尺度异常检测；• 频域专家输出经过门控网络加权融合，形成面部特征表示fface；• 全身分支：利用视觉编码（如CLIP）提取全局语义特征，结合大模型（如Vicuna）生成推理文本，输出语义特征fctx和自我置信度c；• 融合机制：根据置信度调节两个分支的特征贡献，最终输出伪造概率。
Experiments
在HuFor数据集上，采用AUC、准确率、TPR95、TPR99等指标评估性能。训练采用三阶段策略：第一阶段训练全身语义分析分支，第二阶段优化面部检测分支，第三阶段冻结两者，结合置信度调节融合。对比多种SOTA方法，进行消融实验验证频域专家、多尺度设计和多模态融合的有效性。模型在不同伪造类型（面部操控、全身合成）均表现优异，验证了其泛化能力和鲁棒性。
Results
模型在HuFor数据集上实现90.22%的AUC，优于NPR（87.75%）和M2F2-Det（86.73%），TPR95达70.87%，TPR99达33.45%。在面部操控子集，检测性能达87.80% AUC，优于多种深度学习模型。消融实验显示多尺度LoG和多模态融合机制显著提升检测效果，验证了频域专家和全身语义分析的协同作用。模型在不同伪造场景下表现稳定，说明其强泛化能力。
Applications
该方法适用于数字内容验证、媒体取证、社交平台内容审核等场景。只需输入图像，即可快速判断伪造概率，帮助识别虚假信息。未来可结合实时检测需求，优化模型推理速度，推广到视频监控、直播内容验证等领域，提升数字媒体的安全性。
Limitations & Outlook
模型对极端低质量或高压缩图像的检测能力有限，微妙伪造痕迹仍可能被遗漏。频域专家的多尺度设计增加计算复杂度，影响实时应用。对新型伪造技术的适应性有待验证，未来需增强模型的泛化能力。
Plain Language Accessible to non-experts
想象你在一家工厂，每天都在生产各种商品。有些商品被偷偷改动，比如标签换了、部件拼错了，但工厂的检测员只用肉眼看，容易漏掉细节。现在，工厂引入了一台新设备：它可以用不同的“眼睛”观察商品，不仅能看清表面，还能用特殊的“频率”检测隐藏的瑕疵。这台设备还能结合工厂的生产流程和工人讲的话，判断商品是否被篡改。这样一来，不管是外表被改还是内部结构出问题，都能被发现。这个系统就像HuForDet一样，结合多种“观察方式”，让伪造变得难以逃过检测。它不仅能找到明显的假货，还能识别那些微妙的伪造，确保每件商品都是真品。
ELI14 Explained like you're 14
想象你有个超级厉害的朋友，他能用不同的方法判断一张图片是真是假。有时候，他会仔细观察图片的细节，比如脸上的纹理或衣服的缝线，就像用放大镜一样；有时候，他会用特殊的“雷达”扫描图片，找隐藏的瑕疵；还会结合图片背后的故事，比如人物的姿势是否合理，背景是否符合逻辑。这个朋友还能告诉你，他有多大把握，说不定还会说：“我觉得这张图是真的，但也可能有点假。”他的方法就是结合了细节观察、频率分析和语义理解，像HuForDet一样，能全面判断图片的真假。这样一来，即使伪造技术再厉害，也难逃他的“火眼金睛”。
Abstract
The rapid advancement of AI-generated content (AIGC) has escalated the threat of deepfakes, from facial manipulations to the synthesis of entire photorealistic human bodies. However, existing detection methods remain fragmented, specializing either in facial-region forgeries or full-body synthetic images, and consequently fail to generalize across the full spectrum of human image manipulations. We introduce HuForDet, a holistic framework for human image forgery detection, which features a dual-branch architecture comprising: (1) a face forgery detection branch that employs heterogeneous experts operating in both RGB and frequency domains, including an adaptive Laplacian-of-Gaussian (LoG) module designed to capture artifacts ranging from fine-grained blending boundaries to coarse-scale texture irregularities; and (2) a contextualized forgery detection branch that leverages a Multi-Modal Large Language Model (MLLM) to analyze full-body semantic consistency, enhanced with a confidence estimation mechanism that dynamically weights its contribution during feature fusion. We curate a human image forgery (HuFor) dataset that unifies existing face forgery data with a new corpus of full-body synthetic humans. Extensive experiments show that our HuForDet achieves state-of-the-art forgery detection performance and superior robustness across diverse human image forgeries.