SubZero+: Efficient Zeroth-Order LLM Fine-Tuning via Large Learning Rates
SubZero+ enhances zeroth-order optimization stability with large learning rates, outperforming baselines on SuperGLUE.
Key Findings
Methodology
SubZero+ improves stability through multi-query gradient estimation within layer-specific low-rank subspaces, a subspace Adam optimizer for adaptive updates, and sign correction for QR-based subspace construction. This approach expands the stable learning rate range without significant memory overhead.
Key Results
- SubZero+ outperforms existing zeroth-order baselines across models from 1.3B to 32B on SuperGLUE, narrowing the gap to first-order methods.
- On the OPT-1.3B model, SubZero+ achieves 66.3% accuracy, surpassing SubZero's 65.5% and MeZO's 64.6%.
- On the Qwen2.5-32B model, SubZero+ achieves an average accuracy of 87.4%, outperforming SubZero by 2.3%.
Significance
SubZero+ addresses the instability caused by high-variance gradient estimators in zeroth-order optimization, significantly improving the stability and efficiency of fine-tuning large language models. It offers a memory-efficient fine-tuning solution without backpropagation, valuable for both academia and industry.
Technical Contribution
SubZero+ introduces multi-query gradient estimation and a subspace Adam optimizer, providing new theoretical guarantees and engineering possibilities. Its QR decomposition sign correction eliminates implementation-dependent orientation ambiguity, enhancing optimization stability.
Novelty
SubZero+ is the first to combine multi-query gradient estimation with subspace adaptive optimization in zeroth-order optimization, overcoming the multi-query paradox and providing a more stable optimization path.
Limitations
- SubZero+ remains sensitive to learning rates, though the range is expanded.
- Convergence instability may still occur in certain high-dimensional parameter spaces.
Future Work
Future research could explore further variance reduction methods and applications to larger-scale models.
AI Executive Summary
SubZero+ introduces multi-query gradient estimation and a subspace Adam optimizer in zeroth-order optimization, addressing the instability caused by high-variance gradient estimators in existing methods. The method outperforms existing zeroth-order baselines across models from 1.3B to 32B on the SuperGLUE benchmark, narrowing the gap to first-order methods.
The core technologies of SubZero+ include multi-query gradient estimation within layer-specific low-rank subspaces, adaptive updates using a subspace Adam optimizer, and sign correction via QR decomposition. These combined techniques enhance optimization stability and efficiency, particularly in fine-tuning large language models.
Experimental results show that SubZero+ achieves superior performance over existing methods across multiple tasks, notably achieving 66.3% accuracy on the OPT-1.3B model and 87.4% on the Qwen2.5-32B model. However, SubZero+ remains sensitive to learning rates, and future research could explore further variance reduction methods.
Deep Analysis
Background
Zeroth-order optimization is a backpropagation-free method suitable for fine-tuning large language models. Existing zeroth-order methods face high-variance gradient estimation issues, leading to unstable convergence. SubZero reduces variance by estimating gradients in layer-specific low-rank subspaces but remains sensitive to learning rates.
Core Problem
The core problem in zeroth-order optimization is the unstable convergence caused by high-variance gradient estimators, particularly in fine-tuning large language models. As parameter dimensions increase, the variance issue becomes more severe, limiting the application of zeroth-order optimization.
Innovation
SubZero+ addresses high variance in zeroth-order optimization through multi-query gradient estimation and a subspace Adam optimizer. Its QR decomposition sign correction eliminates implementation-dependent orientation ambiguity, enhancing optimization stability.
Methodology
- �� Perform multi-query gradient estimation within layer-specific low-rank subspaces to reduce variance.
- �� Use a subspace Adam optimizer for adaptive updates, improving training efficiency.
- �� Apply sign correction via QR decomposition to ensure Haar-distributed projection matrices.
Experiments
Experiments were conducted on the SuperGLUE benchmark, involving models from 1.3B to 32B. Using models like OPT, LLaMA, and Qwen, SubZero+ was compared against existing zeroth-order baselines, evaluating its performance under full-parameter tuning and LoRA schemes.
Results
SubZero+ outperforms existing zeroth-order baselines across multiple tasks, notably achieving 66.3% accuracy on the OPT-1.3B model and 87.4% on the Qwen2.5-32B model.
Applications
SubZero+ is applicable for fine-tuning large language models, especially in memory-constrained environments. It offers a memory-efficient fine-tuning solution without backpropagation.
Limitations & Outlook
SubZero+ remains sensitive to learning rates, though the range is expanded. Convergence instability may still occur in certain high-dimensional parameter spaces.
Plain Language Accessible to non-experts
Imagine you're in a kitchen, and SubZero+ is like a smart cooking assistant. It doesn't need you to guide it step-by-step but learns to help you better by observing your actions. It offers suggestions when needed, like adjusting the heat or adding spices, without you having to instruct it every step of the way. This assistant improves its skills over time by gaining experience across different dishes, helping you work more efficiently in the kitchen.
ELI14 Explained like you're 14
Hey there! Imagine you're playing a super complex game, and SubZero+ is like a smart game assistant. It helps you find the best way to win without pressing all the buttons. It watches your style and gives tips, like when to speed up or slow down. This way, you can win faster without starting from scratch every time. Isn't that cool?
Glossary
Zeroth-Order Optimization
An optimization method that doesn't require backpropagation, suitable for fine-tuning large language models.
Used to reduce memory overhead in fine-tuning large language models.
SubZero+
An improved zeroth-order optimization framework that enhances stability through multi-query and subspace optimization.
Outperforms existing zeroth-order baselines on the SuperGLUE benchmark.
QR Decomposition
A matrix decomposition method used to generate orthogonal matrices.
Ensures Haar-distributed projection matrices.
Haar Distribution
A probability distribution ensuring rotational invariance of matrices.
Eliminates implementation-dependent orientation ambiguity.
SuperGLUE
A benchmark for evaluating natural language understanding models.
Used to evaluate SubZero+'s performance.
Open Questions Unanswered questions from this research
- 1 How to further reduce gradient estimation variance in zeroth-order optimization?
- 2 How does SubZero+ perform on larger-scale models?
- 3 How to optimize SubZero+ in high-dimensional parameter spaces?
Applications
Immediate Applications
Fine-tuning Large Language Models
SubZero+ can efficiently fine-tune large language models in memory-constrained environments, suitable for scenarios requiring rapid adaptation to new tasks.
Long-term Vision
General AI
By improving stability in zeroth-order optimization, SubZero+ offers possibilities for more efficient general AI.
Abstract
Zeroth-order (ZO) optimization enables backpropagation-free fine-tuning of large language models, but existing ZO methods suffer from high-variance gradient estimators, making convergence unstable and highly sensitive to learning rates. We propose SubZero+, an improved SubZero framework that improves stability in three complementary ways: (i) multi-query gradient estimation within layer-specific low-rank subspaces to reduce variance without exhibiting the multi-query paradox; (ii) a subspace Adam optimizer that performs adaptive updates using in-subspace multi-query gradient statistics; and (iii) a sign correction for QR-based subspace construction to ensure Haar-distributed projection matrices, eliminating implementation-dependent orientation ambiguity. Experiments on models from 1.3B to 32B across SuperGLUE, under both full-parameter tuning and LoRA, show that SubZero+ consistently outperforms prior ZO baselines, enlarges the stable learning-rate range, and narrows the gap to first-order methods with minimal extra memory overhead.