GptGet

Forgetting Only What Matters: Layer-Selective Unlearning toward Robust LLMs

TL;DR

FOM-UL achieves efficient forgetting by selectively updating Transformer layers, maintaining utility and privacy under 8/4-bit quantization.

cs.LG 🔴 Advanced 2026-09-10 77 views
Ravi Ranjan Olivera Kotevska Agoritsa Polyzou
AI Reader Arxiv Page Download PDF
LLMs machine unlearning privacy quantization robustness layer selection

Key Findings

Methodology

FOM-UL calculates a forget-to-retain significance score to identify Transformer layers with high influence on the forget set and low sensitivity to the retain set. It updates only these layers using a multi-objective optimization strategy combining forgetting, mismatch, and retain losses, iteratively expanding the layer set until forgetting criteria are met.

Key Results

  • On TOFU, KnowUnDo, and MUSE datasets, FOM-UL reduces residual memorization by 20%-30% compared to GA and NPO baselines, with retain-set utility close to the vanilla model.
  • Under 8-bit and 4-bit quantization, FOM-UL achieves 15%-20% lower residual memorization than global update methods, with minimal utility loss.
  • FOM-UL achieves the lowest adversarial leakage rate (11.6%) compared to SURE+NPO (16.5%) and LUNAR (19.8%) under adversarial prompts.

Significance

FOM-UL offers a practical, quantization-robust unlearning framework that addresses the trade-off between privacy protection and utility preservation. Its layer-selective strategy significantly reduces computational overhead and residual memorization, making it ideal for real-world LLM deployments.

Technical Contribution

FOM-UL introduces a novel layer-selection mechanism based on forget-to-retain significance scores, enabling precise and efficient unlearning. Its iterative layer expansion and multi-loss optimization strategy improve robustness against quantization-induced recovery and reduce computational costs.

Novelty

FOM-UL is the first framework to implement layer-selective unlearning in LLMs, distinguishing itself from prior global or fixed-layer update methods. Its dynamic layer selection ensures targeted forgetting with minimal collateral damage.

Limitations

  • Residual memorization increases under extreme quantization (e.g., 2-bit), highlighting limited robustness in such scenarios.
  • The method is sensitive to the choice of forget and retain sets, requiring careful tuning.
  • No formal guarantees of erasure are provided, limiting its applicability in high-security contexts.

Future Work

Future directions include improving robustness under extreme quantization, automating forget/retain set selection, and developing formal guarantees for stronger privacy assurances.

AI Executive Summary

Large Language Models (LLMs) excel at natural language tasks but pose risks of privacy leakage due to memorization of sensitive training data. Existing unlearning methods rely on global updates, which degrade utility and are vulnerable to quantization recovery.

FOM-UL (Forgetting Only What Matters via Unlearning Layers) introduces a novel layer-selective framework. By computing forget-to-retain significance scores, it identifies key Transformer layers for targeted updates. The method combines forgetting, mismatch, and retain losses, iteratively expanding the layer set to achieve efficient forgetting while preserving utility.

Experiments on TOFU, KnowUnDo, and MUSE datasets show that FOM-UL outperforms baselines like GA and NPO, reducing residual memorization by 20%-30%. It maintains strong performance under 8/4-bit quantization and achieves the lowest adversarial leakage rate. While limitations exist under extreme quantization, FOM-UL provides a scalable, robust solution for privacy-preserving LLM deployments, with potential for further refinement in future research.

Deep Analysis

Background

LLMs have revolutionized NLP but face challenges with privacy risks due to memorization of sensitive data. Existing unlearning methods, such as GA and NPO, rely on global updates, which are computationally expensive and degrade utility.

Core Problem

The core problem is achieving selective forgetting of specific training data without degrading model utility, especially under post-training quantization, which can restore forgotten knowledge.

Innovation

FOM-UL's key innovations include:

  • �� A forget-to-retain significance score for identifying critical Transformer layers.
  • �� A multi-objective optimization strategy combining forgetting, mismatch, and retain losses.
  • �� Iterative layer expansion to minimize updates while achieving effective forgetting.

Methodology

FOM-UL consists of the following steps:

  • �� Dataset selection: Define forget and retain sets.
  • �� Layer attribution: Compute forget-to-retain scores Sig(ℓ) for each layer.
  • �� Layer selection: Identify critical layers based on Sig(ℓ).
  • �� Targeted updates: Apply forgetting, mismatch, and retain losses to selected layers.
  • �� Iterative expansion: Expand layer selection until forgetting criteria are met.

Experiments

Experiments use TOFU, KnowUnDo, and MUSE datasets, comparing FOM-UL with GA, NPO, and other baselines. Metrics include residual memorization (M1, M2), privacy leakage (M3), and utility preservation (M4).

Results

FOM-UL achieves 20%-30% lower residual memorization than baselines on TOFU and MUSE datasets. It maintains utility close to the original model and shows strong robustness under 8/4-bit quantization.

Applications

FOM-UL is ideal for dynamic LLM deployments requiring frequent updates, such as personalized recommendation systems and privacy-sensitive applications.

Limitations & Outlook

FOM-UL struggles under extreme quantization (e.g., 2-bit) and requires careful tuning of forget/retain sets. It lacks formal erasure guarantees, limiting its use in high-security scenarios.

Plain Language Accessible to non-experts

Imagine a library where some books need to be removed because they contain outdated or sensitive information. Instead of reorganizing the entire library, FOM-UL identifies the specific shelves holding those books and removes only the necessary ones. This way, the library remains functional, and the unwanted books are gone without affecting the rest.

ELI14 Explained like you're 14

Think of your brain as a giant memory box. Some memories are awesome, like your favorite vacation, but others are embarrassing! FOM-UL is like a magic eraser that only erases the bad memories without touching the good ones. Plus, it makes sure no one can sneak in and find those erased memories later. Cool, right?

Glossary

Transformer Layer

A fundamental building block of neural networks that processes input data and generates output representations.

Used for selective updates to achieve forgetting.

Forgetting Loss

An optimization objective designed to maximize prediction errors on the forget set.

Used during unlearning to suppress specific knowledge.

Quantization

A technique that maps high-precision model parameters to low-bit representations to reduce storage and computation costs.

Evaluated for robustness under 8-bit and 4-bit quantization.

Forget-to-Retain Significance Score

A metric that measures the trade-off between forgetting and retaining knowledge for each Transformer layer.

Guides layer selection in FOM-UL.

Adversarial Prompt

A specially designed input to test a model's vulnerability to privacy leakage.

Used to evaluate FOM-UL's robustness against knowledge recovery.

Open Questions Unanswered questions from this research

  • 1 How can robustness be improved under extreme quantization (e.g., 2-bit)?
  • 2 What methods can automate the selection of forget and retain sets?
  • 3 How can formal guarantees of erasure be achieved for stronger privacy assurances?

Applications

Immediate Applications

Privacy Protection

Quickly remove sensitive user data upon request to comply with regulations like GDPR.

Personalized Recommendations

Dynamically update recommendation models by removing outdated or irrelevant user preferences.

Long-term Vision

Secure AI Deployment

Deploy privacy-preserving LLMs in sensitive domains like healthcare and finance.

Abstract

Large Language Models (LLMs) can memorize and reproduce sensitive, copyrighted, or otherwise undesirable training content, creating privacy, safety, and regulatory concerns. Machine unlearning offers a practical alternative to full retraining, but many existing methods apply broad or fixed parameter updates that can degrade utility and remain brittle under deployment changes such as post-training quantization, where forgotten knowledge may partially re-emerge. We propose Forgetting Only What Matters via Unlearning Layers (FOM-UL), a layer-level unlearning framework that selects transformer layers using a forget-to-retain significance score. This score identifies layers with high influence on the forget set and low sensitivity to the retain set, allowing FOM-UL to concentrate updates where they are most effective while leaving most of the model unchanged. This targeted update strategy improves the forgetting-utility trade-off and provides an empirical path toward quantization-resilient unlearning by reducing the chance that small, diffuse updates are erased by low-bit rounding. Across TOFU, KnowUnDo, and MUSE-style evaluations, FOM-UL reduces residual memorization compared with strong GA, NPO, KLD, SURE, ReLearn, and LUNAR-based baselines while preserving retain-set utility close to the vanilla model. Under 8-bit and 4-bit post-training quantization, FOM-UL maintains stronger memorization suppression and utility preservation than competing methods, and adversarial prompt evaluations show lower recovery of forgotten content. Overall, FOM-UL provides an efficient unlearning strategy that improves targeted forgetting, utility preservation, and deployment robustness without claiming formal guarantees of erasure.

cs.LG cs.AI