Backbone is All You Need: Assessing Vulnerabilities of Frozen Foundation Models in Synthetic Image Forensics

Recent advances in AI have led to highly realistic synthetic images, challenging digital media integrity. Early detection methods relied on CNNs to identify artifacts, but lacked generalization. The advent of large pre-trained models like CLIP, Swin, and DINOv2 improved accuracy but introduced new security concerns. Prior work focused on white-box attacks or transferability, leaving gray-box vulnerabilities underexplored. As these models are often deployed with frozen backbones, understanding their susceptibility to feature-space attacks is crucial. This paper builds on the premise that pre-trained ViT features exhibit shared structures exploitable by adversaries, prompting the development of targeted gray-box attacks.

Despite high detection accuracy, ViT-based fake image detectors are vulnerable when attackers possess only the backbone parameters. Existing defenses are insufficient against feature-space manipulations, which do not require access to classifier weights. The core challenge lies in crafting perturbations that manipulate high-dimensional feature representations to cause misclassification, without full model access. This vulnerability poses a serious threat in real-world scenarios where models are often deployed as frozen backbones, making it imperative to evaluate and mitigate such risks.

The paper introduces a novel attack framework that leverages feature alignment between CLIP text embeddings and ViT visual features, trained via a contrastive loss. During attack, PGD optimizes perturbations directly in this shared feature space, bypassing the classifier head. This approach is innovative because it exploits the shared structure of pre-trained models, enabling high success rates with minimal knowledge. Unlike prior methods limited to output gradients, SIAA operates solely on feature distributions, broadening the scope of gray-box vulnerabilities and offering a new perspective on model security.

• �� Training phase: Use real and fake images to train FP-Head, aligning CLIP text embeddings with ViT features via a contrastive loss that enforces tight clustering for matching labels.
• �� Attack phase: Add a small random perturbation to the input image, then extract features using the frozen ViT backbone and CLIP encoder.
• �� Feature mapping: Process features through linear layers to project them into a shared embedding space, approximating the classifier’s decision boundary.
• �� Optimization: Use PGD to iteratively maximize a distance-based loss between visual and textual features, updating the input image with constrained pixel perturbations.
• �� Evaluation: Test on multiple models, measure attack success rate (ASR), and analyze transferability across architectures.

Using the Synthbuster dataset, containing 9,000 synthetic images from models like DALL-E 2, Stable Diffusion, and 1,000 real images, the authors trained the feature alignment module with data augmentation. They tested against CLIP, Swin, and DINOv2 detectors, measuring ASR and AUC. Hyperparameters included 14 PGD iterations, 𝜖=8/255, and embedding dimensions of 768 and 1024. Ablation studies varied training data size, dataset mismatch, and classifier head depth, confirming robustness. Cross-model transferability was also assessed, revealing high attack efficacy especially between models with similar pre-training objectives.

SIAA achieved over 98% ASR on all detectors under matched training conditions, outperforming prior gray-box methods. Even with limited data or dataset mismatches, success rates remained above 96%. Cross-model tests showed 56% success between Swin and CLIP, but only 10-15% on DINOv2, indicating model-specific robustness. The attack's effectiveness persisted across various scenarios, demonstrating its practical threat potential. These results underscore the importance of considering feature-space vulnerabilities in model security.

The attack exposes critical vulnerabilities in current deepfake detectors relying on frozen ViT backbones, urging the development of defenses like adversarial training or feature regularization. It can be used to evaluate detector robustness, guide the design of more resilient models, and inform security policies for digital media. In industry, it highlights the need for multi-modal and adaptive defenses to prevent malicious exploitation, ensuring trustworthy AI-based content verification.

The method shows reduced effectiveness against models like DINOv2, which have more stable dense features. It depends on training a feature alignment model, which may not generalize to architectures with significant modifications. In highly mismatched training scenarios, success rates decrease slightly, indicating the need for more adaptive approaches. Future work should focus on improving transferability and robustness against diverse model architectures.