Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNs
Training only BatchNorm γ/β yields 82% on CIFAR-10 and 32% top-5 on ImageNet in deep ResNets.
Key Findings
Methodology
The authors freeze all convolutional, shortcut, and classifier weights at random initialization and train only BatchNorm’s affine parameters γ and β. BatchNorm is used in the pre-activation order: normalize by mini-batch mean/variance, then apply per-channel scale-and-shift. This isolates the expressive power of affine feature transforms while preserving the full reservoir of random CNN features.
Key Results
- On CIFAR-10, a fully trained ResNet-110 reaches 93.3% test accuracy, while training only 8.3K BatchNorm parameters (0.48% of the model) still achieves 69.5%. With more random features, performance rises further: ResNet-866 reaches 82% and WRN-14-32 reaches 73%.
- On ImageNet, a ResNet-200 trained only through BatchNorm achieves 32% top-5 accuracy. If the output layer is trained jointly with BatchNorm, performance increases to 57% top-5 and 32% top-1. By contrast, training an equal number of randomly selected parameters yields only about 56% on CIFAR-10 and under 4% top-5 on ImageNet.
- The learned γ values become sparse: in BatchNorm-only ResNet-110, about 27% of γ values and in ResNet-101 about 33% satisfy |γ|<0.01, effectively disabling roughly one quarter to one third of channels. Under full training, the corresponding fractions are only around 4% and 5%.
Significance
This paper elevates BatchNorm’s γ/β from a supporting detail to a primary object of study. It shows that even when all learned features are removed, a CNN can still achieve substantial classification accuracy by merely rescaling and shifting random features. The result changes how we think about representation learning: feature quality matters, but so does the expressive power of per-feature affine control.
Technical Contribution
The main technical contribution is a clean, controlled protocol: freeze the entire CNN backbone at random initialization, train only BatchNorm affine parameters, and compare against strong baselines such as full training, training the output layer, or training the same number of random weights elsewhere. The authors also perform mechanistic analysis of γ distributions and threshold-clamping experiments, demonstrating that performance comes in part from learned channel suppression rather than from parameter count alone.
Novelty
The novelty is not merely that random networks can learn, but that BatchNorm parameters have measurably greater expressive power than an equivalent number of arbitrarily chosen weights. Unlike classic random-feature models with a single trainable linear readout, here affine parameters are distributed throughout the depth of the network and appear before nonlinearities, yielding a layered random-feature reweighting scheme.
Limitations
- Performance remains well below full end-to-end training: for example, ResNet-110 drops from 93.3% to 69.5% on CIFAR-10, so random features plus affine control cannot replace learned feature extraction. The method is therefore an adapter, not a full alternative to supervised representation learning.
- The approach depends on sufficiently deep or wide architectures that supply many random features. The paper also observes linear gains with BatchNorm parameter count and hints at saturation, implying diminishing returns and no guarantee of scalability.
- On ImageNet, freezing the output layer is especially restrictive because 1000-class discrimination requires a strong classifier head. BatchNorm-only therefore captures much of the representational power, but not all of the class-separating burden.
Future Work
Natural follow-ups include identifying which random initializations are most amenable to affine reweighting, combining BatchNorm sparsity with structured pruning, and testing whether the same phenomenon holds for detection, segmentation, multi-task learning, or self-supervised adaptation. A broader goal is to build a theory of layered random-feature networks with trainable affine gates.
AI Executive Summary
This paper spotlights a deceptively small part of modern CNNs: the BatchNorm scale and shift parameters γ and β. Instead of asking how well a network can learn when everything is trainable, the authors ask a sharper question: what if all convolutional filters are frozen at random initialization, and only BatchNorm is allowed to learn? The answer is surprising—random features plus affine control can still classify images far better than chance.
The study evaluates ResNet and Wide ResNet families on CIFAR-10 and ImageNet. A fully trained ResNet-110 reaches 93.3% on CIFAR-10, but training only 8.3K BatchNorm parameters, just 0.48% of the model, still yields 69.5%. Scaling the same idea upward produces even stronger results: ResNet-866 reaches 82% on CIFAR-10, WRN-14-32 reaches 73%, and ResNet-200 reaches 32% top-5 on ImageNet. Crucially, training the same number of randomly chosen parameters elsewhere in the network performs much worse, which shows that γ and β are not interchangeable with arbitrary weights.
Mechanistically, BatchNorm acts like a network of tiny valves. The learned γ values often collapse toward zero, effectively turning off a large fraction of random channels. In BatchNorm-only ResNet-110, about 27% of γ values are below 0.01 in magnitude; in ResNet-101 on ImageNet, the figure is about 33%. That means the model is not merely using random features—it is learning which random features to ignore, which to amplify, and how to shift them into useful ranges. Depth and width help because they provide more random features to sift through, and the authors find near-linear gains as BatchNorm parameter count increases.
The broader message is that affine transformations are far more expressive than they may appear. Even without learned convolutional kernels, a deep network retains meaningful capacity if it can rescale and shift random features at many layers. For researchers, this reframes BatchNorm as a major source of representational power; for practitioners, it suggests a practical route to lightweight adaptation by freezing large backbones and training only a tiny set of affine parameters.
Deep Analysis
Background
BatchNorm is ubiquitous in CNNs because it stabilizes training and often improves accuracy. In its standard form, it first normalizes activations using the mini-batch mean and variance, then applies learned γ and β. Prior explanations emphasized internal covariate shift, smoother optimization, reduced exploding activations, or better neuron utilization. However, much of that literature treats γ and β as secondary to the normalized features themselves. This paper builds on work on BatchNorm, parameter-efficient adaptation, and random-feature models, but isolates the affine parameters as the central object of study.
Core Problem
The core question is whether BatchNorm’s affine parameters have independent expressive power when the learned features are removed. If every convolutional kernel, shortcut, and classifier weight is frozen at random initialization, can the network still learn useful functions? The problem is scientifically important because γ and β are usually trained only jointly with the rest of the model, making their role hard to separate from the features they act on. It is also technically challenging because only 0.27%–0.64% of parameters remain trainable.
Innovation
First, the authors propose an unusually clean intervention: freeze all non-BatchNorm weights and train only γ and β. This turns a CNN into a hierarchy of random features with learnable affine gates. Second, they scale the number of available random features by varying depth and width, showing that representational capacity improves predictably with more channels and layers. Third, they analyze mechanism rather than only accuracy: γ distributions, near-zero spikes, and clamping tests reveal that channel suppression is a major part of the explanation. Fourth, they compare against equal-sized random-parameter baselines, establishing that γ/β are special because they control entire channels.
Methodology
- �� Architecture choice: use He et al.’s CIFAR-10 ResNets, ImageNet ResNets, and Wide ResNets; BatchNorm is placed before the activation, following He et al. (2016).
- �� Parameter freezing: all convolutional weights, shortcuts, and output layers are fixed at random initializations; only BatchNorm γ and β are optimized.
- �� Initialization: β is set to 0; γ is sampled uniformly from [0,1]. The paper averages 5 runs on CIFAR-10 and 3 runs on ImageNet.
- �� Baselines: compare against full training, BatchNorm-only, training two random parameters per channel, BatchNorm + output layer, and output-only training.
- �� Scaling studies: increase depth and width to expand the pool of random features while monitoring how accuracy changes with BatchNorm parameter count.
- �� Mechanistic probes: plot γ histograms, measure the fraction with |γ| below thresholds such as 0.01, 0.05, and 0.1, then clamp small γ values to zero and observe the accuracy drop.
- �� Metrics: use CIFAR-10 test accuracy and ImageNet top-1/top-5 accuracy.
Experiments
The experiments span CIFAR-10 and ImageNet, covering ResNet-14 through ResNet-866, WRN-14-2 through WRN-14-32, and ImageNet ResNet-18/34/50/101/200. The primary comparison is among four training regimes: full training, BatchNorm-only, training an equal number of random parameters per channel, and BatchNorm plus the classifier head. The paper also studies depth versus width scaling and the effect of zeroing small γ values.
Results
The headline result is that BatchNorm-only learning is surprisingly strong. On CIFAR-10, ResNet-110 reaches 69.5% with just 8.3K trainable parameters, while ResNet-866 reaches 82%. On ImageNet, ResNet-200 reaches 32% top-5. These numbers are far above chance and far above equal-size random-parameter baselines, which top out around 56% on CIFAR-10 and 4% top-5 on ImageNet. Training the output layer alongside BatchNorm further boosts ImageNet to 57% top-5 and 32% top-1. The γ analysis shows substantial sparsification: 27%/33% of γ values are near zero in BatchNorm-only ResNet-110/ResNet-101.
Applications
The most immediate uses are in parameter-efficient adaptation settings: multi-task learning, style transfer, personalized models, federated learning, and edge deployment. In these cases, a large pretrained or frozen backbone can remain fixed while only affine parameters are updated, greatly reducing communication and compute. The method is especially attractive when storage, privacy, or training time are constrained.
Limitations & Outlook
The method is not a substitute for full feature learning. It works best when the architecture is deep or wide enough to offer a large random basis, and it still trails full training by a wide margin. It is also less effective when the classifier head is frozen on large-label problems like ImageNet. A key open challenge is to determine when affine control alone is sufficient and how to extend the idea beyond classification.
Plain Language Accessible to non-experts
Think of a big factory that makes toy cars. Usually, you would both build better machines and train workers to use them. This paper does something unusual: it locks all the machines in place and only lets workers turn knobs that make each machine work a little louder, softer, left, or right. You might expect the factory to stop being useful. Surprisingly, it still makes pretty good toy cars.
Why? Because the factory starts with many machines already doing different random things. Some are useful, some are noisy, some are repetitive. The workers do not rebuild the machines; they simply decide which ones to turn up, which ones to turn down, and which ones to nearly switch off. If you have enough machines, this simple control can still produce good products. That is why bigger factories in the paper do better: they have more random machines to choose from.
The clever part is that the workers learn to shut down a lot of the unhelpful machines. So the factory is not using everything equally; it is filtering. This shows that sometimes smart control matters almost as much as smart machinery. In deep learning terms, the little control knobs inside BatchNorm can carry a surprising amount of the load, even when the main parts of the network are frozen.
ELI14 Explained like you're 14
Imagine a game where you build a huge robot, but then your teacher says, “Nope, you can’t change the robot’s body anymore. You can only use tiny sliders on each part to make it stronger, weaker, or a bit shifted.” Sounds unfair, right? But that’s exactly what this paper tests. And guess what? The robot still works way better than random guessing!
On CIFAR-10, a big ResNet that usually gets 93.3% accuracy can still reach 69.5% if you only train those tiny sliders called γ and β. That’s crazy because only 0.48% of the model is actually learning. And when the model gets bigger—more layers or more channels—it gets even better, like having more LEGO pieces to rearrange.
The coolest trick is that lots of sliders move close to zero. That means the model is basically saying, “This part? Not useful. Turn it off.” Kind of like deleting useless apps on your phone so it runs faster. The model isn’t making random chaos work by magic; it’s learning what to keep and what to ignore.
So what’s the big lesson? A neural network does not always need to rewrite everything from scratch. Sometimes, if you already have a bunch of random stuff, the smartest move is just to arrange it well. It’s like using a messy pile of clothes to make a cool outfit—you do not need new clothes, just better styling!
Glossary
BatchNorm
A layer that standardizes activations using batch statistics and then restores flexibility with learnable scale and shift. In plain language, it keeps signals well-behaved while still allowing each channel to be adjusted. This paper studies the adjustment part in isolation.
The central object whose γ and β are trained while all other weights are frozen.
γ (gamma)
The per-channel multiplicative factor in BatchNorm. Technically, it rescales normalized activations; in this paper it often shrinks toward zero, effectively suppressing entire channels.
Used to quantify channel importance and sparsity under BatchNorm-only training.
β (beta)
The per-channel additive offset in BatchNorm. It shifts normalized activations so the network can reposition features even when the main weights are fixed.
Trained together with γ as the only learnable parameters in the main experiments.
Random features
Features produced by a fixed randomly initialized network rather than learned filters. The model then uses a small trainable layer or affine parameters to exploit those fixed features.
The paper reframes frozen ResNets as deep random-feature systems with trainable affine gates.
Top-5 accuracy
The fraction of test examples whose true label appears among the five highest-scoring predictions. It is standard for large-label problems like ImageNet because it is less strict than top-1 accuracy.
A key metric for the ImageNet experiments, especially when only BatchNorm is trained.
Open Questions Unanswered questions from this research
- 1 What random initializations or architectural motifs make random features especially amenable to γ/β reweighting remains unclear. The paper shows the effect, but not a full theory of which bases are best.
- 2 It is still unknown how far BatchNorm-only adaptation transfers to structured tasks such as detection, segmentation, or self-supervised representation learning, where the output space and loss geometry differ substantially.
Applications
Immediate Applications
Fast adaptation with frozen backbones
Teams can freeze a large CNN backbone and train only BatchNorm affine parameters for a new task, reducing training cost and memory while preserving much of the base model’s capability.
Low-bandwidth personalization
In federated or on-device settings, users may update only γ/β instead of full weights. This minimizes communication and can still give meaningful task adaptation with very few trainable parameters.
Long-term Vision
Random-feature CNNs as a design paradigm
The paper points toward a broader architecture family where expressive power comes from random features plus shallow affine control. Such models could offer a useful balance of efficiency, modularity, and interpretability.
Abstract
A wide variety of deep learning techniques from style transfer to multitask learning rely on training affine transformations of features. Most prominent among these is the popular feature normalization technique BatchNorm, which normalizes activations and then subsequently applies a learned affine transform. In this paper, we aim to understand the role and expressive power of affine parameters used to transform features in this way. To isolate the contribution of these parameters from that of the learned features they transform, we investigate the performance achieved when training only these parameters in BatchNorm and freezing all weights at their random initializations. Doing so leads to surprisingly high performance considering the significant limitations that this style of training imposes. For example, sufficiently deep ResNets reach 82% (CIFAR-10) and 32% (ImageNet, top-5) accuracy in this configuration, far higher than when training an equivalent number of randomly chosen parameters elsewhere in the network. BatchNorm achieves this performance in part by naturally learning to disable around a third of the random features. Not only do these results highlight the expressive power of affine parameters in deep learning, but - in a broader sense - they characterize the expressive power of neural networks constructed simply by shifting and rescaling random features.