When Fewer Layers Break More Chains: Layer Pruning Harms Test-Time Scaling in LLMs
Layer pruning impacts LLM reasoning, especially long-chain reasoning.
Key Findings
Methodology
The study examines the impact of layer pruning on long-chain reasoning through test-time scaling. Experiments were conducted using Qwen3-8B and s1.1-7B models with pruning methods like ShortGPT, Reverse-order, and LaCo.
Key Results
- Result 1: On AIME24, pruning one layer caused Qwen3-8B's long-chain reasoning to collapse, with accuracy nearing zero.
- Result 2: Despite stable performance on knowledge-intensive tasks, long-chain reasoning significantly declined.
- Result 3: Supervised fine-tuning failed to recover lost reasoning capabilities.
Significance
The study reveals potential risks of layer pruning for reasoning-intensive LLMs, challenging the 'lossless pruning' narrative and emphasizing the need for cautious evaluation and deployment.
Technical Contribution
This research is the first to systematically reveal the fragility of test-time scaling under layer pruning, providing a new analytical framework and experimental validation, advancing the understanding of LLM reasoning capabilities.
Novelty
First to explore layer pruning's impact on long-chain reasoning through test-time scaling, revealing potential risks and challenges in this domain.
Limitations
- Limitation 1: Models perform poorly on long-chain reasoning tasks post-pruning, especially when multiple layers are removed.
- Limitation 2: Supervised fine-tuning failed to effectively restore reasoning capabilities.
Future Work
Future research could explore more effective pruning strategies and recovery methods to maintain LLM reasoning capabilities.
AI Executive Summary
Layer pruning is a common technique to improve the efficiency of large language models (LLMs), but its impact on long-chain reasoning remains underexplored. This study analyzes the effect of layer pruning on LLM reasoning capabilities through test-time scaling. Experiments show that even pruning one or two layers drastically reduces performance on long-chain reasoning tasks, while maintaining stability on knowledge-intensive tasks. Supervised fine-tuning failed to recover lost reasoning capabilities, indicating potential structural damage. The findings challenge the 'lossless pruning' narrative, calling for a reevaluation of pruning strategies and providing insights for developing methods that preserve reasoning capabilities.
Deep Analysis
Background
In recent years, large language models (LLMs) have made significant advances in natural language processing. However, these models often have a large number of redundant layers, leading to computational inefficiency. Layer pruning has become a common method to improve model efficiency, but its impact on model reasoning capabilities, especially long-chain reasoning, has not been fully explored.
Core Problem
While layer pruning improves model efficiency, it may weaken the model's long-chain reasoning capabilities. Long-chain reasoning is crucial for many real-world applications, making it important to study its performance post-pruning.
Innovation
This study is the first to analyze the impact of layer pruning on long-chain reasoning through test-time scaling, revealing potential reasoning capability declines and challenging the existing 'lossless pruning' narrative.
Methodology
- �� Experiments conducted using Qwen3-8B and s1.1-7B models
- �� Pruning methods include ShortGPT, Reverse-order, and LaCo
- �� Evaluated the impact of pruning on long-chain reasoning tasks
- �� Analyzed the effectiveness of supervised fine-tuning in recovering reasoning capabilities
Experiments
Experiments used datasets like MATH500, GPQA Diamond, and AIME24 to evaluate model reasoning capabilities at different pruning depths. Performance comparisons pre- and post-pruning were analyzed to assess the impact on long-chain reasoning.
Results
Experiments showed that pruning one layer caused Qwen3-8B's performance on long-chain reasoning tasks to collapse. Despite stable performance on knowledge-intensive tasks, long-chain reasoning significantly declined. Supervised fine-tuning failed to effectively restore reasoning capabilities.
Applications
The findings are significant for applications requiring long-chain reasoning, such as scientific reasoning and multi-step logical inference. Pruning strategies need reevaluation to ensure model performance in these tasks.
Limitations & Outlook
The study shows that layer pruning can significantly degrade long-chain reasoning capabilities, especially when multiple layers are removed. Supervised fine-tuning failed to effectively restore reasoning capabilities, indicating a need for new recovery methods.
Plain Language Accessible to non-experts
Imagine a factory where each worker on the production line has a specific task. Layer pruning is like reducing the number of workers to improve efficiency, but if key workers are removed, the production line may lose coordination, leading to reduced efficiency. Even if it performs well on some tasks, the production line may collapse on tasks requiring complex coordination.
ELI14 Explained like you're 14
Imagine you're playing a complex game that requires multiple steps to win. Layer pruning is like reducing the game's steps to speed things up, but if key steps are removed, you might not win the game. Even if you do well on simple levels, you might fail on levels requiring multi-step strategies.
Glossary
Layer Pruning
A method to reduce the number of model layers to improve computational efficiency.
Used to enhance the efficiency of large language models.
Long-chain Reasoning
A complex reasoning process involving multiple steps.
Evaluating the impact of layer pruning on model reasoning capabilities.
Test-time Scaling
Techniques that allocate more computational resources during inference to enhance reasoning capabilities.
Used to evaluate model reasoning capabilities.
Supervised Fine-tuning
Fine-tuning a model with supervised data to improve performance.
Attempted to recover reasoning capabilities lost after pruning.
Qwen3-8B
An 8-billion parameter instruction-tuned model supporting thinking and non-thinking modes.
Used to evaluate the impact of layer pruning on reasoning capabilities.
Open Questions Unanswered questions from this research
- 1 How to perform layer pruning without compromising long-chain reasoning capabilities?
- 2 Are there more effective fine-tuning methods to recover reasoning capabilities post-pruning?
Applications
Immediate Applications
Scientific Reasoning
Ensuring model reasoning capabilities are not compromised in applications requiring complex reasoning in scientific research.
Long-term Vision
Intelligent Assistants
Developing intelligent assistants capable of complex reasoning to enhance user experience and task completion rates.
Abstract
Layer pruning has emerged as a widely adopted technique for improving the efficiency of large language models (LLMs). Although existing methods demonstrate strong performance retention on general knowledge tasks, their effect on long-chain reasoning, a more brittle yet crucial capability, remains largely unexplored. In this work, we study the impact of layer pruning on long-chain reasoning through the lens of test-time scaling, a key mechanism in modern LLMs that enables strong reasoning capacity by allocating more computation at inference time. With extensive experiments, we demonstrate that pruning even one or two layers can severely impair test-time scaling, with performance collapsing drastically on long reasoning benchmarks even when performance on knowledge-intensive and shallow reasoning tasks remains stable. Furthermore, we find that standard supervised fine-tuning remedies fail to recover test-time scaling once it has deteriorated. Through in-depth analyses, we identify the mechanisms underlying this fragility of test-time scaling and highlight the fundamental risks of applying layer pruning to reasoning-intensive LLMs. These findings call for a rethinking of layer pruning strategies and provide insights for developing methods that preserve the robustness of reasoning. We open-source the codebase in \href{https://github.com/keyu-wang-2002/Layer-Pruning-Harms-Inference-Scaling}{https://github.com/keyu-wang-2002/Layer-Pruning-Harms-Inference-Scaling}.