ForgetMe: Evaluating Selective Forgetting in Generative Models
Proposes ForgetMe dataset and Entangled metric for selective forgetting in diffusion models, using prompt-layered editing and training-free local feature removal.
Key Findings
Methodology
This paper introduces an automatic dataset creation framework based on prompt-layered editing and training-free local feature removal. It constructs the ForgetMe dataset by employing SAM for segmentation, CLIP for similarity scoring, GPT for category verification, and LaMa plus Stable Diffusion for background inpainting. The Entangled metric combines similarity and consistency to evaluate unlearning effectiveness, supporting both paired and unpaired images in an unsupervised manner. Fine-tuning diffusion models with LoRA achieves targeted feature removal, validated across diverse datasets. The approach emphasizes scalability, efficiency, and minimal retraining, enabling precise, privacy-compliant unlearning in high-dimensional generative models.
Key Results
- On datasets including CUB-200-2011, Stanford-Dogs, ImageNet, and synthetic cats, the LoRA fine-tuned models achieved target region forgetting success rates exceeding 90%. The Entangled metric demonstrated superior performance with FID as low as 58.88 and CMMD at 0.80, outperforming state-of-the-art methods like CLIPAway. The framework effectively balances unlearning thoroughness with background consistency, validated through extensive quantitative metrics and ablation studies.
- Experimental results highlight the robustness and versatility of ForgetMe and Entangled as benchmarks for selective unlearning. The method maintains high image fidelity while removing sensitive features, supporting both supervised and unsupervised evaluation modes. Parameter analysis indicates that higher LoRA scaling factors (α≥0.9) significantly improve unlearning efficacy without compromising background integrity.
- Ablation studies confirm each component’s contribution, and cross-dataset evaluations demonstrate generalizability. The approach offers a scalable, adaptable solution for privacy-preserving generative AI, with potential applications in sensitive domains such as healthcare, security, and social media.
Significance
This work addresses the critical challenge of achieving precise, efficient, and verifiable selective unlearning in high-dimensional diffusion models. By providing a standardized dataset and a novel evaluation metric, it bridges the gap between theoretical privacy requirements and practical deployment. The proposed framework reduces computational costs compared to retraining, enhances model controllability, and supports unsupervised assessment, thus advancing the field of privacy-aware AI. Its broad applicability across multiple datasets and scenarios underscores its potential to influence industry standards and regulatory policies, fostering safer and more trustworthy AI systems.
Technical Contribution
The paper introduces a novel prompt-based layered editing framework for automatic dataset generation, combined with training-free local feature removal techniques. The Entangled metric innovatively quantifies unlearning effectiveness by integrating similarity and statistical consistency, supporting both paired and unpaired image evaluation without supervision. The integration of LoRA for targeted fine-tuning enables efficient, high-precision feature removal in large diffusion models. These contributions collectively push forward the state-of-the-art in privacy-preserving generative modeling, offering a practical, scalable, and rigorous solution.
Novelty
This research is the first to combine prompt-layered editing with training-free local feature removal for dataset generation tailored to unlearning tasks. The Entangled metric uniquely measures residual associations between foreground and background, supporting unsupervised, multi-modal evaluation. Unlike existing methods relying on retraining or pruning, this approach offers high efficiency, flexibility, and scalability, setting a new standard for privacy-preserving diffusion models.
Limitations
- 当前方法在处理透明对象或复杂背景时仍存在一定局限，背景重建的细节一致性有待提升，尤其在高分辨率场景中表现不足。
- 依赖预训练模型的性能，若模型本身存在偏差或不足，可能影响目标区域提取和背景重建效果。
- 在极端高维或极端复杂场景下，背景重建和遗忘效果仍需优化，未来需结合多模态信息增强鲁棒性。
Future Work
未来将探索多模态信息融合以提升背景重建质量，优化层级提示设计以增强目标提取的准确性，扩展到视频和3D场景中的选择性遗忘。同时，结合差分隐私和联邦学习技术，推动模型在实际应用中的隐私合规性，提升方法的实用性和普适性。
AI Executive Summary
The rapid advancement of diffusion models in image synthesis has brought about remarkable improvements in visual quality and diversity. However, as these models increasingly memorize sensitive information during training, privacy concerns emerge, necessitating effective unlearning techniques. Traditional approaches like fine-tuning or pruning are computationally expensive and often inadequate for high-dimensional generative models, lacking standardized evaluation metrics. This paper introduces ForgetMe, a comprehensive framework that combines a novel dataset and an innovative evaluation metric, Entangled, to address these challenges.
The core idea involves layered prompt-based editing, which isolates target features using SAM segmentation, verifies their identity with CLIP similarity scores and GPT validation, and then employs LaMa and Stable Diffusion for seamless background inpainting. The Entangled metric assesses the residual association between foreground and background regions, balancing similarity and statistical coherence to quantify unlearning effectiveness. Fine-tuning diffusion models with LoRA enables targeted feature removal, achieving high success rates (>90%) across multiple datasets.
Experimental results demonstrate that the proposed framework outperforms existing methods in both unlearning thoroughness and background consistency. Metrics such as FID (58.88) and CMMD (0.80) confirm the high quality of generated images post-unlearning. The approach’s scalability and unsupervised evaluation capability make it a promising tool for privacy-preserving AI deployment in sensitive applications like healthcare, security, and social media.
While the method shows strong performance, limitations include challenges in handling transparent objects and complex backgrounds, as well as reliance on pre-trained models. Future work aims to incorporate multi-modal data, improve background reconstruction, and extend to video and 3D scenarios, fostering safer, more controllable generative AI systems.
Deep Dive
Key Concepts
Prompt Layered Editing
一种通过提示逐层操作图像的技术，用于目标区域的提取和背景的重建，支持无训练的局部特征去除。
Entangled Metric
一种结合相似性和一致性指标的评估方法，用于量化目标区域遗忘的彻底性和背景保持的连贯性。
LoRA (Low-Rank Adaptation)
一种高效微调技术，通过低秩参数调整实现模型的目标区域遗忘，减少训练成本。
Background Reconstruction
利用LaMa和Stable Diffusion等模型对目标区域进行掩码修复，确保背景的自然连续性。
Unsupervised Evaluation
无需标注数据，通过指标自动衡量模型遗忘效果，提升评估的客观性和普适性。
Open Questions Unanswered questions from this research
- 1 如何进一步提升背景重建在极端复杂场景中的细节一致性仍是未解难题。
- 2 现有方法在处理透明或半透明对象时效果有限，未来需结合多模态信息增强鲁棒性。
- 3 如何在保证隐私保护的同时，兼顾生成效率和模型性能，是未来研究的重要方向。
Abstract
The widespread adoption of diffusion models in image generation has increased the demand for privacy-compliant unlearning. However, due to the high-dimensional nature and complex feature representations of diffusion models, achieving selective unlearning remains challenging, as existing methods struggle to remove sensitive information while preserving the consistency of non-sensitive regions. To address this, we propose an Automatic Dataset Creation Framework based on prompt-based layered editing and training-free local feature removal, constructing the ForgetMe dataset and introducing the Entangled evaluation metric. The Entangled metric quantifies unlearning effectiveness by assessing the similarity and consistency between the target and background regions and supports both paired (Entangled-D) and unpaired (Entangled-S) image data, enabling unsupervised evaluation. The ForgetMe dataset encompasses a diverse set of real and synthetic scenarios, including CUB-200-2011 (Birds), Stanford-Dogs, ImageNet, and a synthetic cat dataset. We apply LoRA fine-tuning on Stable Diffusion to achieve selective unlearning on this dataset and validate the effectiveness of both the ForgetMe dataset and the Entangled metric, establishing them as benchmarks for selective unlearning. Our work provides a scalable and adaptable solution for advancing privacy-preserving generative AI.