Dominant-Layer ZO: A Single Layer Dominates Zeroth-Order Fine-Tuning of LLMs
Zeroth-order optimization reveals a single dominant decoding layer for efficient LLM fine-tuning, achieving up to 4.52× speedup.
Key Findings
Methodology
The study identifies a dominant decoding layer in zeroth-order optimization, leveraging activation outliers for pre-training layer selection without exhaustive fine-tuning.
Key Results
- On LLaMA2-7B, single-layer fine-tuning improves average performance by 0.57% and speeds up training by 4.52×.
- On Qwen3-8B, single-layer fine-tuning improves performance by 1.15%, with a 3.57% gain on CB tasks.
- Compared to Sparse-MeZO, dominant-layer fine-tuning achieves ~30× faster parameter updates.
Significance
This research significantly reduces LLM fine-tuning costs while maintaining or exceeding full-model performance, offering new insights into optimization signal distribution.
Technical Contribution
Introduces a dominant-layer selection method based on activation outliers and explains its effectiveness via residual propagation theory.
Novelty
First to reveal the dominant-layer phenomenon in zeroth-order optimization and propose activation-outlier-based layer selection for efficient fine-tuning.
Limitations
- The method is model-specific and may not generalize to all LLM architectures.
- Activation outlier detection could be sensitive to input data, requiring further validation.
Future Work
Future research can explore optimizing non-dominant layers and extending this method to diverse architectures and tasks.
AI Executive Summary
Zeroth-order optimization, which estimates gradients using only forward passes, has emerged as a memory-efficient approach for fine-tuning large language models (LLMs). However, the distribution of useful adaptation across layers remains unclear.
This study reveals that a single decoding layer dominates zeroth-order fine-tuning performance. By leveraging activation outliers, the dominant layer can be identified pre-training, eliminating the need for exhaustive layer-wise analysis. Experiments on LLaMA2-7B and Qwen3-8B show that single-layer fine-tuning matches or exceeds full-model performance while achieving up to 4.52× speedup.
The findings offer a new perspective on optimization signal propagation and pave the way for more efficient fine-tuning methods. Future work may focus on optimizing non-dominant layers and validating the approach across broader models and tasks.
Deep Analysis
Background
Zeroth-order optimization estimates gradients via random perturbations and loss evaluations, avoiding memory-intensive backpropagation. While prior work improves gradient estimation, layer-wise adaptation remains unexplored.
Core Problem
Current zeroth-order methods treat fine-tuning as a full-model process, leading to inefficiencies. Identifying key layers for adaptation could reduce computational costs and improve scalability.
Innovation
This study discovers a dominant layer phenomenon in zeroth-order fine-tuning and proposes a lightweight method to identify this layer using activation outliers, eliminating costly layer-wise analysis.
Methodology
- �� Analyze activation outliers to identify dominant layers.
- �� Propose a pre-training layer selection method based on activation statistics.
- �� Validate the approach on LLaMA2-7B and Qwen3-8B across nine tasks.
Experiments
Experiments use LLaMA2-7B and Qwen3-8B models across tasks like SST-2, COPA, and DROP. Comparisons include full-model MeZO, Sparse-MeZO, and ablation studies to validate dominant-layer selection.
Results
Single-layer fine-tuning achieves comparable or superior performance to full-model MeZO, with up to 4.52× speedup on LLaMA2-7B and 1.15% average improvement on Qwen3-8B.
Applications
Applicable to resource-constrained scenarios, such as edge-device LLM deployment and rapid task adaptation in dynamic environments.
Limitations & Outlook
The method relies on model-specific structures and may require further validation for generalization across diverse architectures.
Plain Language Accessible to non-experts
Imagine fine-tuning an orchestra where each instrument represents a layer in the model. Zeroth-order optimization is like adjusting the sound by listening, without direct control. This study finds that tuning just one instrument (the dominant layer) can make the entire orchestra sound perfect, saving time and effort.
ELI14 Explained like you're 14
Think of a team game where each player has different skills. Zeroth-order optimization is like deciding who to train based on their performance. The study shows that training just one key player (the dominant layer) makes the whole team better! Cool, right? And it saves tons of time too!
Glossary
Zeroth-Order Optimization
A method estimating gradients using only forward passes, avoiding backpropagation.
Used for memory-efficient fine-tuning of LLMs.
Activation Outlier
Activation dimensions with unusually high magnitudes compared to the average.
Used to identify dominant layers.
Dominant Layer
The single decoding layer contributing most to performance improvement in zeroth-order fine-tuning.
Identified via activation outliers.
Residual Propagation
Perturbations in early layers propagate through residual connections, amplifying their effects.
Explains dominant-layer phenomenon.
LLaMA2-7B
A large language model with 32 decoding layers.
Used to validate the proposed method.
Open Questions Unanswered questions from this research
- 1 How to optimize non-dominant layers effectively?
- 2 Is activation outlier detection sensitive to input data?
- 3 Can this method generalize to more architectures?
Applications
Immediate Applications
Edge-device fine-tuning
Quickly adapt LLMs for resource-constrained devices, enhancing task performance.
Task migration
Rapidly adapt models to new tasks, reducing fine-tuning time and costs.
Long-term Vision
Efficient model design
Develop architectures optimized for dominant-layer fine-tuning, reducing computational overhead.
Abstract
Zeroth-order (ZO) optimization enables memory-efficient fine-tuning of large language models (LLMs) using only forward passes, but it remains unclear how useful adaptation is distributed across layers. In this work, we reveal a surprising phenomenon: ZO fine-tuning is sharply dominated by a single decoding layer. Across multiple LLM families and downstream tasks, fine-tuning this dominant layer alone consistently matches or even exceeds full-model ZO fine-tuning. We further show that the dominant layer is task-agnostic but model-specific, and can be identified before training through a simple inference-only analysis of activation outliers. Specifically, the dominant layer consistently aligns with the first activation-outlier layer in the pre-trained model. To explain this phenomenon, we analyze how perturbation effects propagate under ZO optimization. We find that the dominant layer combines two key properties: high perturbation sensitivity and early placement in the residual stream, allowing perturbation-induced effects to propagate and accumulate through remaining subsequent decoding layers. As a result, this layer produces disproportionately strong and stable optimization signals under forward-only updates. Extensive experiments on LLaMA2-7B and Qwen3-8B across nine benchmarks show that dominant-layer ZO fine-tuning improves average performance over full-model MeZO and LoRA-based ZO fine-tuning while achieving up to 4.52$\times$ training speedup.