DinoLizer: Separating VAE and Diffusion Artifacts in Generative Inpainting Localization
DinoLizer leverages DINOv2 with LoRA fine-tuning to detect VAE and diffusion artifacts, achieving 20% higher IoU on forgery localization.
Key Findings
Methodology
This paper introduces a DINOv2-based localizer that employs LoRA to fine-tune the Query and Value projections of the Vision Transformer, adding a single linear layer for patch-level classification at 14×14 resolution. A sliding window approach aggregates predictions on larger images, enabling dense pixel-level localization. The model classifies three categories: real, VAE artifacts, and inpainted regions, improving discrimination over traditional binary classifiers. Extensive data augmentation enhances robustness against common post-processing operations.
Key Results
- On FOSSIL and SOTA datasets, DinoLizer achieves an average IoU increase of over 20%, reaching up to 45%, with F1 scores surpassing existing methods. It maintains stable performance under JPEG double compression, demonstrating strong anti-interference capabilities. Ablation studies show that LoRA fine-tuning and three-class training significantly boost performance while reducing trainable parameters to 2.4 million. The approach excels in detecting small and localized inpainted regions, outperforming baselines across various perturbations.
- Across different scales and noise levels, DinoLizer exhibits superior robustness, especially in JPEG compression and resizing scenarios. The sliding window dense prediction strategy effectively detects subtle forgery traces in large images, making it suitable for real-world forensic applications.
- Overall, the experimental results validate DinoLizer’s effectiveness and efficiency in localizing generated inpainting artifacts, with broad implications for digital content authentication and anti-deepfake efforts.
Significance
This work advances forgery detection by explicitly modeling VAE and diffusion artifacts as separate classes, significantly improving detection accuracy and robustness. Its lightweight architecture and sliding window inference make it practical for large-scale deployment. The method addresses a critical need for reliable, automated deepfake detection, contributing to the integrity of digital media and combating misinformation. It bridges the gap between high-fidelity generative models and forensic analysis, setting a new benchmark for local forgery localization.
Technical Contribution
The key innovation lies in integrating DINOv2 transformers with LoRA-based parameter-efficient fine-tuning, enabling high performance with minimal training overhead. Introducing a three-class classification scheme distinguishes generative traces from authentic regions, enhancing interpretability. The sliding window dense inference approach supports large image analysis without downsampling, maintaining spatial detail. These technical advances collectively push the state-of-the-art in forgery localization, offering a scalable and robust solution.
Novelty
This is the first work to explicitly differentiate VAE and diffusion artifacts as separate classes in a pixel-level forgery detection framework using DINOv2. The combination of LoRA fine-tuning, multi-class training, and sliding window dense inference is novel, enabling high accuracy on large images with limited parameters. Unlike prior methods focusing solely on global features or binary classification, this approach captures subtle local traces, setting a new standard for inpainting forgery detection.
Limitations
- While highly robust to JPEG compression and moderate noise, the model's performance slightly degrades under extreme noise or very high compression ratios. Its real-time inference capability on ultra-high-resolution images remains limited, requiring further optimization. The training depends on large annotated datasets, which may not cover all forgery types, potentially affecting generalization. Future work should address these limitations to enhance practical deployment.
Future Work
Future directions include integrating multi-modal cues such as text and video, improving real-time inference speed, and expanding training datasets to cover more forgery scenarios. Developing unsupervised or semi-supervised methods could reduce reliance on annotated data. Additionally, exploring explainability and robustness against adversarial attacks will be crucial for deploying these detectors in real-world forensic systems.
AI Executive Summary
Deep generative models have revolutionized image synthesis, enabling highly realistic inpainting and manipulation. However, this progress poses significant challenges for digital content authentication. Existing forgery detection methods often struggle with subtle, localized traces left by VAE and diffusion-based models, especially under compression or post-processing. To address this, we introduce DinoLizer, a novel detection framework built upon DINOv2 transformers, optimized with LoRA for parameter efficiency. The core innovation is a three-class classification scheme that explicitly models real, VAE, and inpainted regions, enabling finer discrimination of generative artifacts.
The methodology employs a dense sliding window inference, allowing pixel-level localization even in large images. This approach captures subtle local traces that global methods tend to miss. Extensive experiments on diverse datasets, including FOSSIL and multiple SOTA inpainting datasets, demonstrate that DinoLizer outperforms existing methods by at least 20% in IoU, maintaining robustness under JPEG double compression and various perturbations. The lightweight architecture, requiring only 2.4 million trainable parameters, ensures computational efficiency suitable for real-world forensic applications.
The significance of this work lies in its ability to reliably detect sophisticated generative manipulations, contributing to the integrity of digital media. Its robustness and efficiency open avenues for large-scale deployment in content verification, anti-deepfake systems, and digital forensics. Future work will focus on integrating multi-modal cues, optimizing inference speed, and expanding the detection capabilities to new forgery types, further strengthening the defenses against increasingly realistic synthetic content.
Deep Dive
Abstract
We introduce DinoLizer, a DINOv2-based localizer of manipulated areas in generative inpainting. The model is trained to focus on semantically altered regions by treating reconstructed areas outside the inpainted mask as a separate class, which yields significant improvements w.r.t. the conven- tional approach. We train the model with LORA on the Query and Value of the transformer blocks and simply add 1 linear layer on top of the backbone to predict manipulations on a 14 x 14 patch resolution. Because DINOv2 only accepts fixed- sized images, we use a sliding window approach to aggregate the predictions on larger images. Empirical results show that DinoLizer outperforms state-of-the-art methods on our proposed dataset and SOTA inpainting datasets. Furthermore, it is very robust to JPEG (double) compression. On average, DinoLizer achieves a 20% higher Intersection over Union score compared to the second best model. The code is publicly available here: https://github.com/anonyme610/dinolizer.