Special Properties of Gradient Descent with Large Learning Rates

Training neural networks with GD or SGD is standard, yet the role of learning rate is still not fully explained. Prior work studied noise-induced smoothing, implicit bias, and preferences for flatter minima; other studies examined saddle-point avoidance and the “edge of stability.” Jastrzębski et al. highlighted the learning-rate/batch-size ratio, while work on gradient flow suggested continuous and discrete trajectories might be close. This paper argues that those views miss an essential effect: large step size can alter the actual path through a nonconvex landscape, not just the amount of stochasticity.

The central question is whether large learning rates help only because they amplify SGD noise, or whether the step size itself changes optimization dynamics. This matters because if noise were the whole story, then increasing randomness at small γ should be enough. The paper asks a sharper question: can discrete GD with large γ escape a local minimum and reach a different, better minimum in settings where small γ fails?

First, the paper constructs a function class Cl where large-step and small-step GD provably converge to different minima. This is a trajectory-level separation, not just a rate result. Second, it introduces a region-skipping analysis for arbitrary bad sets X, tying visitation probability to γ and the size of X. Third, it brings theory and practice together by showing analogous behavior in neural-network training and by explicitly separating learning-rate effects from stochastic-noise effects via repeated-batch experiments.

• �� Problem setup: minimize f?=minx∈Rdf(x) using full-batch GD and, separately, SGD with additive noise ξt.
• �� Smoothness model: assume L-smoothness to control gradient variation globally or outside exceptional regions.
• �� Local geometry: impose μ†-OPSC near a local minimum x† and μ?-OPSC near the target minimum x?, so the local basin is sharp while the rest of the landscape still guides descent.
• �� Lemma 1: if γ is large enough, specifically 2μ†<γ≤μ?L2 in the paper’s notation, then once GD enters the local region M it escapes rather than settling there.
• �� Theorem 1: combine Lemma 1 with global descent properties on the remainder of the landscape to prove that large-step GD converges to x?, while small-step GD can converge to x†.
• �� Lemma 2: for any bad region X, bound the probability that discrete GD ever visits X within T steps, with dependence on d, T, γ, and Lebesgue measure L(X).
• �� Theorem 2: specialize the bound to regions that are far enough from the global minimum and derive a monotone dependence on step size.
• �� SGD experiment: use repeated-batch training to hold noise approximately fixed while changing γ, isolating the trajectory effect of learning rate.

The paper combines theoretical toy landscapes and a neural-network experiment. In theory, Figures 1, 3, 4, and 6 visualize how different initializations and step sizes lead to convergence to different minima or to escape from a sharp basin. In practice, the authors compare standard SGD at γ=0.01 versus γ=0.001, and also run a repeated-batch variant until epoch 200 followed by 10 extra epochs to preserve stochasticity. The excerpt does not specify the dataset name, but it reports train and test accuracy curves and emphasizes the qualitative gap between step sizes.

The strongest theoretical result is a clean separation: large-step GD nearly surely exits the local basin and converges to the global minimum, whereas small-step GD can remain trapped in the local minimum. The second key result is geometric: the probability of hitting a bad region decreases with the region’s Lebesgue measure and increases with step size, capturing a discrete skipping effect that continuous gradient flow lacks.

The most immediate use is learning-rate scheduling in deep learning: start with a larger γ to promote basin escape, then reduce γ for fine convergence. The analysis also suggests a principled benchmark for optimizer design—new methods should be tested on whether they reproduce discrete-step escape behavior, not only on whether they add noise or preserve smooth convergence.

The results are proved under structured assumptions that may not literally hold in modern networks, so the paper is best read as a mechanism study. The empirical part is directionally convincing but not a full benchmark suite in the provided text, and the exact dataset/architecture details are not fully exposed. Finally, the skip-region bound is probabilistic and does not cover every pathological landscape scenario.