Effective Learning Rate Governs Loss Dynamics in Language Model Pretraining
ELR (effective learning rate) governs loss dynamics; matching ELR schedules collapses loss trajectories across configurations.
Key Findings
Methodology
This study introduces ELR (η/∥W∥F) as the core variable controlling loss dynamics. Experiments across diverse models, datasets, and optimizers demonstrate that matching ELR schedules results in nearly identical loss trajectories, with collapse errors around a few ×10^-3. Ablation studies highlight the importance of normalization schemes (QK-Norm, RMSNorm gains) and the timescale of LR–norm variation. Controlled interventions show weight decay and Hyperball influence loss primarily through ELR schedules. The fitted functional scaling law (FSL), parameterized by ELR, successfully transfers across norm-control methods, explaining phenomena like delayed acceleration.
Key Results
- Across models like Llama-124M and Qwen3-MoE-586M, ELR-matched runs exhibit loss trajectory collapse with mean errors below 5×10^-3, significantly outperforming seed-to-seed variation (~10^-2).
- In practical norm control (weight decay, Hyperball), adjusting only the learning rate to match ELR reproduces target loss curves with errors around 10^-3, confirming ELR’s mediating role.
- The ELR-based FSL accurately predicts loss dynamics across different norm-control regimes, enabling transfer and explaining delayed acceleration effects.
Significance
This work redefines the understanding of training dynamics by positioning ELR as a unifying coordinate linking learning rate scheduling, norm regulation, and loss evolution. It simplifies the complex interplay of hyperparameters, offering a robust framework for optimizing large-scale language model training. The findings have broad implications for improving training stability, efficiency, and transferability across diverse architectures and datasets, thus advancing both theoretical insights and practical methodologies in deep learning.
Technical Contribution
The core contribution is establishing ELR as a high-precision predictor of loss behavior, independent of specific normalization schemes or model scale. The development of ELR-matched trajectory collapse, combined with the transferability of the ELR-parameterized FSL, provides a new paradigm for hyperparameter scheduling. The integration of normalization design and dynamic scale analysis enriches the theoretical foundation, enabling adaptive and transferable training strategies.
Novelty
This is the first systematic demonstration that ELR, rather than raw LR or norm alone, governs loss dynamics across diverse training configurations. It challenges the traditional scale invariance assumptions, showing that ELR remains predictive even in the absence of exact symmetry. The ability to transfer scaling laws via ELR parameterization marks a significant advancement in understanding training universality.
Limitations
- The experiments are predominantly on models up to 1B parameters; scalability to larger models remains to be validated.
- High-precision collapse relies on specific normalization schemes; applicability to models with different architectures or training regimes needs further exploration.
- The microscopic dynamics underlying ELR’s predictive power are not fully understood, warranting deeper theoretical investigation.
Future Work
Future research will focus on developing adaptive ELR scheduling algorithms, integrating dynamic normalization techniques, and extending validation to larger models and multi-task settings. The goal is to establish a comprehensive theoretical framework for ELR-driven training, enabling more efficient and robust large-scale model development.
AI Executive Summary
Recent advancements in large-scale language model pretraining have highlighted complex training dynamics that challenge traditional hyperparameter tuning. While learning rate and parameter norm have been studied independently, their interplay remains poorly understood. This paper uncovers a unifying principle: the effective learning rate (ELR), defined as the ratio of learning rate to parameter norm, governs the loss evolution throughout training. By systematically matching ELR schedules across diverse models, datasets, and optimizers, the authors demonstrate near-perfect collapse of loss trajectories, with mean errors below 5×10^-3. This phenomenon, termed ELR collapse, reveals that the combined effect of learning rate scheduling and norm control operates primarily through ELR, simplifying the understanding of training dynamics.
The study further explores practical norm control methods like weight decay and Hyperball, showing that adjusting only the learning rate to match ELR can replicate target loss behaviors. This confirms ELR’s mediating role in loss regulation. Additionally, by fitting a functional scaling law (FSL) parameterized by ELR, the authors enable the transfer of predictive models across different norm-control regimes, including out-of-distribution scenarios such as Hyperball. This transferability explains phenomena like delayed acceleration, where norm-controlled runs initially lag but eventually surpass uncontrolled baselines.
Overall, the findings position ELR as a fundamental coordinate linking hyperparameter scheduling, norm regulation, and loss dynamics. This insight offers a new lens for designing training strategies, improving stability, and enhancing transferability in large-scale language model pretraining. Despite some limitations related to normalization schemes and model scale, the work opens promising avenues for future research into adaptive ELR scheduling and theoretical understanding of training dynamics.
Deep Analysis
Background
The evolution of large-scale language models (LLMs) has driven extensive research into training dynamics, focusing on hyperparameters like learning rate, batch size, and normalization techniques. Early works such as Krogh and Hertz (1991) and Loshchilov and Hutter (2019) emphasized weight decay and adaptive optimizers. Recent studies introduced norm control mechanisms (e.g., Hyperball, spectral normalization) to stabilize training and improve transferability. However, the interplay between learning rate schedules and norm regulation remains underexplored. Existing literature often treats these factors independently, lacking a unified framework. This paper aims to bridge this gap by identifying ELR as a central variable that captures their combined effect, providing a more comprehensive understanding of loss dynamics in pretraining.
Core Problem
Despite advances, the fundamental question persists: do learning rate and norm control act independently, or are they mediated through a common variable? Traditional approaches adjust LR and norm separately, but this may overlook their intrinsic coupling. Understanding whether a unified variable like ELR can explain loss trajectories across diverse configurations is crucial for optimizing training efficiency and stability. The challenge lies in quantifying this relationship precisely, especially given the complex, non-linear dynamics of large models. Addressing this problem could lead to more robust scheduling strategies and theoretical insights into training behavior.
Innovation
The key innovations include: 1) defining ELR as the ratio η/∥W∥F, capturing the combined effect of LR and norm; 2) demonstrating that matching ELR schedules induces near-perfect collapse of loss trajectories across models, datasets, and optimizers; 3) establishing that practical norm control methods (weight decay, Hyperball) influence loss primarily through ELR; 4) developing an ELR-based functional scaling law (FSL) that transfers across norm-control regimes, explaining phenomena like delayed acceleration; 5) analyzing the impact of normalization schemes and the timescale of LR–norm variation on collapse precision.
Methodology
- �� Define ELR as ηeffk := ηk/∥Wk∥F, serving as the core scheduling variable.
- �� Design multiple LR schedules (linear, sinusoidal) and norm control schemes (weight decay, Hyperball).
- �� Conduct paired experiments matching ELR schedules while varying LR and norm trajectories.
- �� Measure loss residuals (rk) and collapse errors (Δcoll) across training steps.
- �� Incorporate normalization techniques (QK-Norm, RMSNorm) and ablate their effects on collapse precision.
- �� Validate ELR’s mediating role by adjusting LR alone to match ELR in practical norm control setups.
- �� Fit and evaluate ELR-parameterized FSL models on diverse configurations.
- �� Cross-validate transferability of scaling laws to unseen norm-control regimes, including out-of-distribution cases.
Experiments
- �� Use models like Llama-124M, Qwen3-MoE-586M trained on datasets such as FineWeb, C4, OpenWebText.
- �� Implement optimizers including AdamW, Muon, Signum, with varied LR schedules.
- �� Match ELR schedules across configurations with different LR and norm trajectories.
- �� Quantify loss trajectory collapse via mean absolute residuals (Δcoll), typically below 5×10^-3.
- �� Conduct ablations on normalization schemes and LR–norm variation timescales.
- �� Test the transferability of fitted FSL models across configurations, including out-of-distribution scenarios like Hyperball.
- �� Analyze phenomena such as delayed acceleration through ELR schedule manipulation.
Results
- �� Consistent across models and optimizers, ELR-matched runs show loss collapse with errors around a few ×10^-3, outperforming seed variability (~10^-2).
- �� Practical norm control experiments confirm that adjusting LR to match ELR reproduces target loss curves with errors <10^-3.
- �� The ELR-based FSL accurately predicts loss dynamics across diverse regimes, enabling transfer and explaining delayed acceleration phenomena.
Applications
- �� The findings inform the design of adaptive learning rate schedules and norm regulation techniques, improving training stability and efficiency.
- �� In industry, ELR-based scheduling can reduce hyperparameter tuning costs, facilitate model transfer, and accelerate large-scale training in NLP, vision, and multimodal tasks.
Limitations & Outlook
- �� Validation is limited to models up to 1B parameters; scalability to larger models needs further testing.
- �� Dependence on normalization schemes suggests potential limitations in architectures with different normalization or no normalization.
- �� The microscopic mechanisms behind ELR’s predictive power remain to be fully understood, requiring future theoretical work.
Plain Language Accessible to non-experts
想象你在经营一家工厂，生产各种商品。工厂的效率取决于两个因素：工人的工作速度（学习率）和工厂的规模（参数范数）。如果你调节工人的速度和工厂的规模，但保持一个叫做ELR（有效工作速率）的比例不变，那么生产效率就不会变化。也就是说，只要这个比例保持不变，无论你怎么调节，工厂的产出都一样。这就像你在厨房做饭，火力和食材的比例一样，只要比例不变，菜就会熟得一样快。这说明，训练模型时，调节学习率和参数范数实际上是通过ELR这个“效率比”在控制，只要保持它不变，模型的学习效果就会稳定。
ELI14 Explained like you're 14
你在玩一款游戏，你的角色升级速度取决于两个东西：用的技能强度（学习率）和装备的等级（参数范数）。如果你调节技能和装备，但让技能和装备的比例（ELR）保持不变，你的升级速度其实没变。这就像你在厨房做饭，火大和食材多，只要比例一样，菜就会熟得一样快。这告诉我们，训练模型其实也是在调节一个叫ELR的“效率比”，只要它保持不变，训练效果就很稳定。
Abstract
We uncover ELR collapse in language model pretraining: learning rate (LR) and parameter norm govern loss dynamics primarily through their ratio, the effective learning rate (ELR). When ELR is matched across runs, their loss trajectories collapse throughout training despite substantially different LRs and parameter norms. Across optimizers, architectures, datasets, and model scales, mean collapse errors are typically a few x 10^-3, below the seed-to-seed variation measured in a representative configuration. Systematic ablations identify normalization design and the timescale of LR-norm variation as key determinants of collapse precision. Controlled interventions further show that weight decay and Hyperball shape loss dynamics primarily through the ELR schedules they induce. Replacing LR with ELR enables a fitted functional scaling law (FSL) to transfer across norm-control methods. The resulting ELR-based FSL also explains delayed acceleration, a recurring effect of norm control. Together, these results establish ELR as a common coordinate linking LR scheduling, norm control, and loss dynamics.