Ghosted Layers: Unconstrained Activation Alignment for Recovering Layer-Pruned LLMs
Proposes Ghosted Layers, a training-free method deriving a closed-form linear operator to recover pruned layer activations, improving accuracy and perplexity.
Key Findings
Methodology
This work formulates the layer pruning recovery as a boundary activation alignment problem. By analyzing the boundary activation error, it derives a closed-form optimal linear operator W* over the full operator space, capturing asymmetric components. The approach involves collecting boundary activations from a small calibration set, computing the minimum-norm least squares solution via SVD, and inserting this operator into the pruned model through hooks. This method surpasses existing constrained solutions like LinearPatch, which restricts W to symmetric matrices, by recovering the full space, including anti-symmetric parts, thus effectively reducing activation mismatch and performance degradation.
Key Results
- On LLaMA-3.1-8B with 7-layer pruning, Ghosted Layers improves accuracy by approximately 10% and perplexity by 20% over baseline, outperforming LinearPatch and ReplaceMe across multiple benchmarks. The method maintains efficiency and generalizes well across different pruning strategies and model architectures.
- In nine zero-shot QA benchmarks, the average accuracy after pruning is restored to within 1-2% of the full model, with improvements of about 12% over prior methods. Perplexity metrics on datasets like WikiText-2 and C4 show reductions of around 20%, indicating preserved generative capabilities.
- Ablation studies reveal that the optimal operator contains significant anti-symmetric components, which LinearPatch cannot approximate due to symmetry constraints, explaining its inferior performance. The per-channel activation error is uniformly reduced, demonstrating precise boundary correction.
Significance
This research addresses a fundamental challenge in model compression—restoring performance after layer pruning without retraining. By deriving a mathematically optimal linear correction that captures both symmetric and asymmetric components, it enables effective, efficient, and universal recovery. This advances the practical deployment of large language models, reducing hardware costs and energy consumption while maintaining high accuracy. The approach also provides theoretical insights into activation mismatch and opens avenues for further exploration of unconstrained linear corrections in neural network repair and compression.
Technical Contribution
The paper introduces a novel formulation of boundary activation alignment as an unconstrained least squares problem, deriving the closed-form solution W* that includes anti-symmetric parts. It proves that existing methods like LinearPatch are limited to symmetric W, which cannot reach the full optimal solution. The use of SVD for stable, efficient computation of the minimum-norm solution makes this approach broadly applicable. The method is plug-and-play, compatible with any pruning criterion and architecture, and significantly improves recovery performance while preserving inference efficiency, representing a substantial step forward in model compression and repair techniques.
Novelty
This work is the first to explicitly formulate boundary activation mismatch as an unconstrained linear alignment problem and derive its full-space closed-form solution. It reveals the critical role of anti-symmetric components in activation correction, which prior constrained methods overlook. The theoretical proof that the optimal operator W* contains non-symmetric parts distinguishes this approach from existing symmetric or block-structured methods, providing a new paradigm for model repair.
Limitations
- The method relies on a small calibration set; if this set poorly represents the deployment data distribution, the correction may be suboptimal.
- In scenarios with extreme pruning ratios or significant architecture modifications, linear correction alone may be insufficient, requiring nonlinear extensions.
- Current approach assumes linearity; future work could explore nonlinear activation corrections for further performance gains.
Future Work
Future directions include integrating nonlinear correction mechanisms, developing adaptive calibration strategies, and extending to multi-modal models. Exploring dynamic, task-specific boundary corrections and combining with fine-tuning could further enhance robustness and generalization. Additionally, investigating the theoretical limits of unconstrained linear repair and its application to other model compression techniques remains promising.
AI Executive Summary
Large-scale pre-trained language models (LLMs) have revolutionized natural language processing, yet their deployment remains costly due to immense parameter sizes. Layer pruning offers a practical solution by removing entire Transformer decoder blocks, significantly reducing model size and inference latency. However, this introduces a critical challenge: the activation mismatch at the pruning boundary, which degrades downstream performance. Existing training-free recovery methods like LinearPatch and ReplaceMe attempt to address this by inserting linear operators at the boundary, but are limited by structural constraints—such as symmetry or block-diagonal assumptions—that prevent them from capturing the full scope of activation discrepancies.
This paper introduces Ghosted Layers, a novel, training-free recovery framework that derives a closed-form, unconstrained linear operator to optimally align boundary activations. The core innovation lies in formulating the boundary mismatch as a least squares problem over the entire space of linear maps, including anti-symmetric components. By leveraging singular value decomposition (SVD), the method computes the minimal-norm solution efficiently, capturing both symmetric and non-symmetric parts of the optimal correction. This approach allows the insertion of a plug-and-play operator at each pruning boundary, significantly reducing activation error.
Extensive experiments across multiple models, including LLaMA-3-8B, LLaMA-3.1-8B, and DeepSeek-R1-Distill-LLaMA-8B, demonstrate that Ghosted Layers consistently outperforms prior methods in accuracy and perplexity metrics, especially at high pruning ratios. For instance, on LLaMA-3.1-8B with 7 layers pruned, accuracy improves by approximately 10%, and perplexity decreases by 20%, approaching the full model's performance. The analysis reveals that the optimal operator contains substantial anti-symmetric components, which previous symmetric methods cannot recover, explaining their inferior results.
This work offers a significant advancement in model compression, enabling efficient deployment of pruned models without retraining. Its theoretical foundation and practical effectiveness open new avenues for adaptive, scalable model repair, promising broader impacts in AI deployment and research. Future work will explore nonlinear extensions, dynamic calibration, and multi-modal applications to further enhance robustness and applicability.
Deep Analysis
Background
近年来，预训练大规模语言模型（如GPT、LLaMA）在自然语言理解和生成任务中取得巨大成功，但其庞大的参数规模带来部署成本高昂的问题。模型压缩技术不断发展，包括非结构化剪枝、结构化剪枝和层剪枝。层剪枝通过移除Transformer中的完整解码块，保持模型架构的连续性，便于硬件高效实现。已有方法如ShortGPT、SLEB和LLM-Streamline，主要关注层选择和剪枝策略，提升剪枝后性能，但在激活偏差修复方面仍存在瓶颈。
Core Problem
层剪枝引起的激活偏差在边界层表现为激活值不匹配，导致后续层输入分布偏离训练时的状态，造成模型性能显著下降。现有修复方法多受限于对称或块对角结构，难以捕获非对称成分，修复效果有限。如何在无需训练的情况下，精确修复剪枝引起的激活偏差，成为提升模型实用性的关键难题。
Innovation
核心创新包括：1）将激活偏差问题转化为边界激活对齐问题，导出全空间的闭式最优线性解；2）发现最优解包含大量反对称成分，突破对称限制；3）利用奇异值分解（SVD）实现稳定高效的最小范数解，兼容任意剪枝策略和模型架构。这一方法显著优于受限方案，恢复了模型的非对称表达能力，提升修复效果。
Methodology
- �� 采集边界激活：在原始模型上运行校准集，利用前向钩子捕获边界层激活Xpre和Xpost。• 计算激活偏差：∆ = Xpost - Xpre，作为偏差的实测值。• 求解最优线性操作：通过最小二乘问题，导出W* = X†pre ∆，利用奇异值分解（SVD）获得最小范数解。• 插入修复操作：在剪枝边界插入W*，用Xnew = Xpre · W*，实现激活偏差的精确修复。• 数值稳定性：采用正则化参数ϵ确保解的稳定性。• 理论分析：证明W*是全空间的最优解，且包含非对称成分，超越受限的对称方案。
Experiments
- �� 数据集：使用C4、WikiText-2、Penn Treebank作为校准集，采样长度2048，包含128个随机序列。• 模型：在LLaMA-3-8B、LLaMA-3.1-8B和DeepSeek-R1-Distill-LLaMA-8B上测试。• 剪枝策略：采用LLM-Streamline、ShortGPT和Shortened LLaMA，剪除7层或11层。• 评估指标：准确率、困惑度（PPL）和激活误差。• 比较方法：LinearPatch、ReplaceMe、Prune&Comp等。• 详细参数：采用正则化参数ϵ=10^-6，利用SVD或线性求解器实现。
Results
- �� Ghosted Layers在多模型、多剪枝比例下，平均准确率提升约10%，困惑度降低20%，显著优于LinearPatch和ReplaceMe。• 在高剪枝比例（11/32层）下，性能恢复更为明显，接近完整模型水平。• 激活误差分析显示，Ghosted Layers有效减少边界激活偏差，且恢复的激活误差在每个通道上均匀降低，优于对称方案。• 结构分析表明，最优线性操作包含大量反对称成分，LinearPatch无法捕获。
Applications
- �� 立即应用：在大规模预训练模型压缩部署中，利用Ghosted Layers实现模型剪枝后无需再训练，快速修复激活偏差，降低硬件成本，提升推理速度。• 长期愿景：结合动态剪枝和非线性修正，发展自适应模型修复机制，支持模型在不同场景下自动调整结构与性能，推动边缘设备智能化。
Limitations & Outlook
- �� 依赖校准数据的代表性，若偏差分布偏离实际场景，修复效果可能受影响。• 在极端剪枝或架构变化较大时，线性修复可能不足，需结合非线性方法。• 当前未考虑非线性激活偏差的潜在提升空间，未来可结合深度学习模型进行优化。
Plain Language Accessible to non-experts
想象你在厨房做饭，平时每个厨师（模型层）都按照固定的食谱（训练数据）操作。现在你把一些厨师（层）临时换掉，厨房的味道（模型表现）变差了。这就像把一部分调料（激活）弄错了，后续厨师接到的调料不对，导致菜肴变味。传统方法就像用一个固定的调料包（对称修复）试图补救，但效果有限。本文提出的方法像是用一个聪明的调味师（线性操作）根据少量试吃（校准数据）调出最合适的调料比例，快速修正味道偏差，不用重新学厨艺（训练），就能让菜肴恢复原有的美味。
ELI14 Explained like you're 14
你知道在学校里，每个学生（模型层）都按照老师的教导（训练）学习，做事都很有条理。但如果突然换了几个学生，大家的合作就会出问题，结果作业（模型输出）变得不对。这就像把一些调料（激活信息）搞错了，后续的厨师（后续层）就接收到错误的材料，做出来的菜（模型结果）也会变差。以前的方法就像用一个固定的调料包，试图让菜变好，但效果不太理想。现在这个新方法像是请一个聪明的调味师，根据少量的试吃（校准数据），快速调出最合适的调料比例，不用重新学厨艺，就能让菜恢复原来的味道。这就像魔法一样，能让厨房里的菜变得跟以前一样好吃！
Abstract
Layer pruning removes entire Transformer decoder blocks from large language models, but introduces a mismatch between the hidden state received by the next surviving layer and the distribution it was trained to process, leading to significant performance degradation. We propose Ghosted Layers, a training-free recovery module that addresses this issue by solving a boundary activation alignment problem. Our method derives a closed-form optimal linear operator from a small calibration set to reconstruct the activation discrepancy introduced by the pruned layers. We show that this solution corresponds to the unconstrained optimum of the alignment objective, whereas existing methods are restricted to constrained solutions over limited operator subspaces. Experiments across multiple LLM backbones and pruning strategies demonstrate that our method consistently improves accuracy and perplexity over prior training-free baselines, while preserving the efficiency gains of layer pruning. Official code repository: https://github.com/daniel-eai/ghosted_layers_official_repository/.