Sharpness-diversity tradeoff: improving flat ensembles with SharpBalance
SharpBalance preserves ensemble diversity while reducing sharpness, reaching 76.12% on CIFAR-10 versus 75.45% for SAM.
Key Findings
Methodology
The paper studies SAM-trained deep ensembles through sharpness, predictive variance, DER, KL diversity, and ensemble improvement rate (EIR). SharpBalance assigns member i a sharpness-aware set DiSAM formed by the union of the top-k% per-sample-sharp examples identified by all other members. It applies SAM on DiSAM and standard training on DiNormal, reducing local curvature without forcing all members toward the same solution.
Key Results
- Across CIFAR-10, CIFAR-100, and TinyImageNet, increasing SAM radius ρ generally lowers individual sharpness but also lowers variance, DER, and KL diversity; the resulting EIR declines. The pattern is consistent across all three diversity measures.
- Theorem 1 gives D(θSAMk)=ϕ(2k,0)σ² and sharpness bounds for full-data SAM; Theorem 2 proves a more favorable trade-off for subset-trained members. On CIFAR-10 with three-member ResNet18, reported accuracy is 74.19% for Deep Ensemble, 75.45% for SAM, and 76.12% for SharpBalance.
- The effect persists on CIFAR-C, CIFAR-100-C, TinyImageNet-C, altered widths, pruning sparsity, and WRN-40-2. Narrower and sparser models show steeper trade-off curves, making diversity preservation especially important under efficiency constraints.
Significance
The work shows that “flatter” does not automatically mean “better ensemble.” SAM can improve individual generalization while making members functionally similar and reducing the ensemble’s complementary-error benefit. This reframes robustness as a joint optimization of local stability and population diversity. The result matters for reliable vision systems, corrupted-data robustness, and compact ensembles, where every member must be both accurate and meaningfully different.
Technical Contribution
The theoretical framework assumes Gaussian random design, quadratic optimization, and random initialization, then uses Wishart moments and Narayana numbers to characterize output variance and sharpness bounds under SAM. Theorem 2 extends the analysis to members trained on different data submatrices, directly motivating SharpBalance. Engineering-wise, the method uses ||∇θℓj||² as a Fisher-trace proxy, top-k selection, and set union; it changes training allocation rather than network architecture or ensemble inference.
Novelty
The main novelty is not another generic flatness regularizer. It identifies and formalizes a sharpness–diversity trade-off induced by SAM, then actively restores diversity by assigning different members different high-sharpness samples. Unlike standard Deep Ensembles and full-data SAM, SharpBalance targets individual flatness and inter-member complementarity simultaneously.
Limitations
- The theory relies on Gaussian matrices, quadratic objectives, simplified teacher models, and random initialization. These assumptions do not directly cover deep nonlinear networks trained with cross-entropy, class imbalance, or realistic long-tailed distributions.
- Experiments focus mainly on three-member ResNet18 ensembles and image classification. The paper reports strong trends but does not fully map sensitivity to top-k, ensemble size, compute budget, or very large architectures.
Future Work
Future work should make top-k selection, ensemble size, and distribution shift jointly adaptive, and extend the method to detection, segmentation, language models, calibration, and active learning. Theory for non-Gaussian, non-convex networks is needed, as is a direct comparison between parameter-space sharpness and function-space diversity. Practical studies should quantify extra gradient cost, memory use, and benefits under severe label or covariate shift.
AI Executive Summary
Deep ensembles improve reliability by combining several neural networks, but their benefit depends on the members making complementary errors. Sharpness-aware Minimization (SAM) addresses a different goal: it finds solutions whose loss changes less under parameter perturbations, often improving an individual model. This paper shows that the two goals can conflict. As SAM reduces sharpness, members may converge to increasingly similar predictors, reducing diversity and eventually weakening the ensemble gain.
The authors propose SharpBalance. For each member, they estimate per-example sharpness with the squared gradient norm, ||∇θℓj||², a trace-of-Fisher proxy. The top-k% sharp examples selected by the other members are united into DiSAM. The member applies the SAM objective on DiSAM and ordinary empirical-risk minimization on the remaining DiNormal. Theorem 1 analyzes full-data SAM; Theorem 2 extends the analysis to different data submatrices and shows that subset training can retain more diversity at comparable sharpness.
Experiments cover CIFAR-10, CIFAR-100, TinyImageNet, their corrupted variants, width and pruning changes, and WRN-40-2. Three-member ResNet18 studies show consistent trends under predictive variance, DER, and KL: lower sharpness accompanies lower diversity and lower EIR. Reported CIFAR-10 accuracy rises from 74.19% for Deep Ensemble to 75.45% for SAM and 76.12% for SharpBalance. The authors also report increasing advantages as corruption severity grows. The broader message is methodological: ensemble design should treat individual flatness and collective diversity as coupled objectives. SharpBalance offers a simple data-allocation mechanism, although its theory and computational cost remain important open questions.
Deep Analysis
Background
Deep Ensembles (Lakshminarayanan et al., 2017; Fort et al., 2019) exploit prediction diversity for accuracy and uncertainty. SAM (Foret et al., 2021) improves individual generalization by minimizing a neighborhood maximum loss. Yet work by Dinh et al. and Yang et al. cautions that parameter flatness alone is incomplete. This paper studies how sharpness and diversity jointly affect ID and OOD ensemble generalization.
Core Problem
Reducing SAM sharpness can make independently trained members produce more similar predictions, reducing ensemble improvement. The challenge is structural: sharpness is local loss geometry, whereas diversity is a distribution of member outputs, and both depend on data size, width, sparsity, optimizer radius, and initialization. A method must improve stability without collapsing complementary errors.
Innovation
- ��Discovers and empirically validates a sharpness–diversity trade-off.
- ��Theorem 1 derives SAM diversity and sharpness bounds.
- ��Theorem 2 proves a better trade-off for subset-trained members.
- ��SharpBalance uses other members’ high per-example-sharpness samples to construct DiSAM, applying SAM there and normal training elsewhere.
- ��Evaluation spans ID, corrupted OOD data, architectures, widths, and sparsity.
Methodology
- ��Define LD(θ)=1/n∑ℓ(fθ(xi),yi), with ensemble logits obtained by averaging member logits.
- ��Measure adaptive sharpness as max||Tθ^-1ε||≤ρ0[LD(θ+ε)−LD(θ)], making the metric scale-aware.
- ��Measure diversity through D=ED[Var(fθ(D))], DER, and KL; measure ensemble benefit with EIR.
- ��Train full-data SAM using max||ε||≤ρLD(θ+ε)+λ||θ||².
- ��Estimate each example’s sharpness by ||∇θℓj||², approximating the Fisher trace.
- ��For member i, union other members’ top-k% examples into DiSAM; optimize SAM on DiSAM and the ordinary loss on DiNormal.
- ��Analyze the process with Gaussian random matrices, Wishart moments, and Narayana-number expressions in Theorems 1–2.
Experiments
Experiments use CIFAR-10, CIFAR-100, TinyImageNet, and CIFAR-10-C, CIFAR-100-C, TinyImageNet-C. The main setup trains three-member ResNet18 ensembles; supplementary tests vary width, pruning sparsity, and use WRN-40-2. Sharpness is estimated from 100 training batches of size 5. DER is the primary OOD diversity metric, with variance and KL as checks. Baselines are SGD Deep Ensemble and full-data SAM Ensemble; varying SAM radius ρ traces the trade-off.
Results
Three-member ResNet18 experiments show that changing ρ creates a consistent curve: lower sharpness accompanies lower predictive variance, DER, and KL, and lower EIR. Reported CIFAR-10 accuracy is 74.19% for Deep Ensemble, 75.45% for SAM, and 76.12% for SharpBalance. Theory and simulation broadly agree. On corrupted CIFAR-10, CIFAR-100, and TinyImageNet, the authors report that SharpBalance’s advantage grows with corruption severity; narrower and sparser models display steeper trade-offs.
Applications
SharpBalance is relevant to autonomous-driving perception, industrial inspection, medical imaging, remote sensing, and other systems that combine several classifiers under distribution shift. It requires parallel member training and per-example gradient estimates, but no architectural change or special inference procedure. It is particularly attractive for compact or sparse ensembles, where the paper finds the sharpness–diversity conflict to be stronger.
Limitations & Outlook
The theoretical assumptions—Gaussian design, quadratic optimization, simplified teacher models, and random initialization—do not fully describe deep nonlinear networks or cross-entropy training. Per-example gradient scoring and member synchronization add computation and implementation complexity. Although the paper covers three datasets, corrupted OOD data, and WRN-40-2, it does not provide a complete study of ensemble size, top-k sensitivity, class imbalance, severe covariate shift, or large language models. These factors may change the optimal allocation.
Plain Language Accessible to non-experts
Imagine a restaurant with several chefs preparing dishes for one shared banquet. SAM is like a strict quality inspector: each chef repeatedly checks the steps most likely to cause a mistake, so one chef’s dish becomes more stable. But if every chef checks exactly the same steps and follows the same corrections, the dishes start tasting alike. When diners combine them, the banquet loses the advantage of variety.
SharpBalance gives each chef a different inspection assignment. It first finds the ingredients or steps that other chefs handle most precariously, then asks the current chef to focus on that union while practicing everything else normally. Each dish becomes reliable, but the chefs keep distinct strengths. The researchers measure stability by how much performance changes after a small disturbance, and variety by how different the predictions are.
On CIFAR-10, the reported accuracy rises from 74.19% for an ordinary ensemble to 75.45% with SAM and 76.12% with SharpBalance. The lesson is simple: a strong team needs dependable members, but not identical ones.
ELI14 Explained like you're 14
Picture three friends teaming up in a game where you must label pictures as cats, cars, or planes. Combining answers is powerful because one friend may notice a clue the others miss. But what if all three practice exactly the same questions and copy the same strategy? They may all become good, yet they will also make the same mistakes. The team loses its special advantage!
SAM is like a training mode that makes each player less likely to panic when the game changes slightly. Great for one player—but if everyone trains in the same way, they can become too similar. SharpBalance changes the practice schedule: for each player, look at the questions where the other players struggle most, then make that player focus on those questions while practicing the rest normally.
The paper tested this idea on image datasets. On CIFAR-10, an ordinary three-player team scored 74.19%, SAM scored 75.45%, and SharpBalance scored 76.12%. It also worked better on damaged or noisy images.
So what is the big idea? The best team is not made of clones. You want everyone to be dependable, but you also want different viewpoints—like a goalkeeper, a strategist, and a fast attacker. That mix helps the team survive tricky situations!
Glossary
Sharpness
In plain language, it is how fragile a model is near its current solution. Technically, the paper measures the maximum loss increase under a perturbation radius, using an adaptive parameter normalization.
It compares SAM and SharpBalance solutions.
Diversity
It is the amount by which ensemble members disagree in their predictions. The paper uses output variance, DER, and KL divergence.
It explains the extra benefit obtained by combining models.
SAM
Sharpness-Aware Minimization searches for parameters with low worst-case loss in a radius-ρ neighborhood. It is the full-data baseline and the local optimizer inside SharpBalance.
It supplies individual flatness improvements.
DER
Disagreement Error Ratio compares prediction disagreement with average prediction error. Higher disagreement can indicate more complementary members.
It is the main OOD diversity metric in the experiments.
EIR
Ensemble Improvement Rate is the relative reduction from average member error to ensemble error. It measures the practical value of combining models.
It shows how diversity loss can reduce ensemble gains.
SharpBalance
A training strategy that applies SAM to member-specific data subsets and ordinary training to the remaining data. The subsets are selected from other members’ high per-example-sharpness samples.
It is the paper’s proposed method.
Open Questions Unanswered questions from this research
- 1 The theory is built on Gaussian designs and quadratic objectives. It remains unclear whether the same formulas or trade-off shape hold for realistic deep networks, cross-entropy, long-tailed labels, and nonstationary data.
- 2 The joint effects of top-k, ensemble size, training budget, and distribution shift are not fully mapped. Adaptive policies and larger-scale benchmarks are needed.
Applications
Immediate Applications
Robust image-classification ensembles
Vision teams can train ResNet or WRN members with SharpBalance on industrial, medical, or remote-sensing images. The system needs per-example gradients and parallel members. Expected outcomes are preserved individual accuracy and stronger performance on corrupted or OOD inputs.
Sparse edge-device ensembles
After pruning models for edge deployment, teams can rank examples by per-example sharpness and apply SharpBalance during retraining. Because the paper finds stronger trade-offs in narrow and sparse models, this setting may obtain especially meaningful relative gains.
Long-term Vision
Reliable model populations
The principle could support autonomous driving, clinical decision support, and scientific imaging systems whose models must be stable yet independently informative. Obstacles include training cost, calibration, safety validation, and theory under severe real-world shift.
Abstract
Recent studies on deep ensembles have identified the sharpness of the local minima of individual learners and the diversity of the ensemble members as key factors in improving test-time performance. Building on this, our study investigates the interplay between sharpness and diversity within deep ensembles, illustrating their crucial role in robust generalization to both in-distribution (ID) and out-of-distribution (OOD) data. We discover a trade-off between sharpness and diversity: minimizing the sharpness in the loss landscape tends to diminish the diversity of individual members within the ensemble, adversely affecting the ensemble's improvement. The trade-off is justified through our theoretical analysis and verified empirically through extensive experiments. To address the issue of reduced diversity, we introduce SharpBalance, a novel training approach that balances sharpness and diversity within ensembles. Theoretically, we show that our training strategy achieves a better sharpness-diversity trade-off. Empirically, we conducted comprehensive evaluations in various data sets (CIFAR-10, CIFAR-100, TinyImageNet) and showed that SharpBalance not only effectively improves the sharpness-diversity trade-off, but also significantly improves ensemble performance in ID and OOD scenarios.