PIU: Proximity-guided Identity Unlearning in ID-Conditioned Diffusion Models

Deep generative models, especially diffusion architectures like Latent Diffusion Models (LDMs), have achieved remarkable success in high-quality image synthesis. Early methods relied on GANs, but diffusion models like Stable Diffusion have offered better stability and controllability. Identity-conditioned models such as Arc2Face enable identity-preserving face generation, useful for data augmentation and virtual avatars. However, these models also pose privacy risks, as they can memorize training identities, leading to potential misuse and regulatory concerns. Existing privacy techniques focus on data removal or concept erasure but lack targeted identity deletion methods suited for ID-conditioned diffusion models. As privacy laws tighten, developing precise, efficient, and reproducible identity unlearning methods becomes critical to ensure compliance and safeguard individual rights.

The core challenge is to effectively erase specific personal identities from ID-conditioned diffusion models without compromising the overall image quality or the integrity of non-target identities. Unlike discrete labels, identity features are continuous and highly interconnected, making naive removal prone to residual leakage or unintended attribute changes. Existing methods either require full retraining, which is computationally expensive, or lack geometric guidance, resulting in incomplete or unstable deletion. Achieving precise, scalable, and reversible identity removal remains an open problem, especially in models conditioned on dense feature embeddings rather than textual prompts.

This paper introduces a geometric, proximity-guided approach to identity unlearning. Key innovations include: 1) selecting an anchor identity based on cosine similarity thresholds within ArcFace’s hyperspherical space, balancing between too close and too distant identities; 2) formulating identity removal as a geometric replacement, guiding the target identity towards the anchor; 3) designing a loss function that enforces this mapping during diffusion process; 4) fine-tuning only critical cross-attention layers to minimize computational cost while maximizing effectiveness. These strategies collectively enable targeted, efficient, and reversible identity deletion, addressing limitations of prior methods that lacked geometric guidance or relied on full retraining.

• �� Extract identity embeddings using ArcFace for all training images, compute identity centroids. • Select an anchor identity for each target based on cosine similarity thresholds, ensuring intermediate distance. • During diffusion, define a loss that pushes the target identity’s condition vector towards the anchor’s, guiding the model to generate images aligned with the anchor. • Regularize non-target identities with a preservation loss, maintaining their recognition consistency. • Fine-tune only the most identity-sensitive cross-attention layers identified via layerwise separation analysis, reducing computational load. • Use synthetic convex combinations of identity embeddings to generalize the unlearning process. • Optimize the combined loss (target removal + preservation) iteratively, ensuring target identity suppression while maintaining overall image quality and identity fidelity.

Experiments utilized CelebA-HQ, with 27,996 images clustered into 9,683 identities. The model was trained with 400 steps, using AdamW optimizer, learning rate 10^-4. Evaluation metrics included identity similarity scores (ISM), retention rate (SRK), and image quality metrics (∆KD, eDIFFIQA). Baselines included WID, SISS, and UCE. Multiple target identities were tested, with targeted removal verified by significant reductions in similarity scores (from 0.78 to 0.31). The impact on non-target identities was minimal, with retention rates above 70%. Ablation studies confirmed the importance of anchor selection thresholds and layer-wise fine-tuning. Results demonstrated superior performance in both identity erasure and image quality preservation.

PIU achieved the lowest target identity similarity (0.31), outperforming WID (0.32) and other baselines. Retention of non-target identities remained high (~72%). Image quality metrics indicated minimal degradation, with ∆KD and eDIFFIQA scores better than competing methods. Fine-tuning only 4.29% of parameters proved sufficient for effective unlearning. The approach was robust across multiple identities and settings, confirming its scalability and stability. These results validate the geometric, anchor-guided strategy as a practical solution for privacy-preserving face generation.

This method can be directly applied in privacy-sensitive face synthesis, virtual avatar creation, and deepfake mitigation. It allows users or regulators to selectively delete personal identities from models without retraining, ensuring compliance with privacy laws. Long-term, it can facilitate safer AI content generation, personalized model management, and cross-modal privacy safeguards, transforming how personal data is handled in AI systems.

Dependence on ArcFace features may limit effectiveness if features are biased or insufficiently diverse. Fine-tuning is still computationally demanding, especially for large-scale models. The anchor selection process requires dataset-specific calibration, which may not generalize well. Future work should explore adaptive, multi-modal, and automated anchor strategies, as well as formal privacy guarantees to address residual risks.