Surrogate assisted diversity estimation in neural ensemble search

Neural network ensemble techniques have long been recognized for their ability to improve prediction accuracy and robustness. Early methods like Deep Ensembles train multiple models independently, but lack explicit control over diversity, often leading to correlated models. Neural Architecture Search (NAS) automates model design, yet its high computational cost hampers large-scale deployment. Recent research emphasizes the importance of diversity in ensembles, but existing approaches rely on heuristic or implicit measures, limiting their effectiveness. This context motivates the development of methods that explicitly optimize for both accuracy and diversity, leveraging surrogate models to reduce search costs and improve ensemble quality. The integration of graph neural networks (GNNs) into NAS further enables structural encoding of architectures, opening new avenues for multi-objective optimization.

The core challenge in neural ensemble construction is balancing the trade-off between individual model strength and collective diversity. Existing NAS methods focus predominantly on accuracy, neglecting the diversity crucial for ensemble robustness. Explicitly modeling diversity is difficult due to the high cost of training numerous architectures and the lack of effective metrics. Without a systematic approach, ensembles tend to consist of highly similar models, diminishing their benefits. Moreover, the exponential growth of the search space makes joint optimization computationally infeasible with traditional methods. Addressing these issues requires a scalable framework that can evaluate and optimize both objectives efficiently, guiding the search toward architectures that are not only accurate but also diverse enough to improve ensemble performance.

The key innovation is the design of a dual surrogate framework that explicitly models both accuracy and diversity. Using graph attention networks (GATs), architectures are encoded as DAGs, and two separate surrogate models predict their performance and diversity potential. The diversity surrogate employs triplet loss to embed architectures into a latent space where distances reflect prediction dissimilarity, enabling explicit diversity control. This approach allows for efficient, training-free evaluation of architectures in the latent space, guiding the selection process through a greedy algorithm that maximizes diversity while maintaining high accuracy. This integrated multi-objective strategy significantly reduces search complexity and improves ensemble robustness, marking a departure from traditional single-objective NAS methods.

• �� Architecture Representation: Convert neural network architectures into DAGs, with nodes as operations and edges as data flow. • Surrogate Models: Train two GAT-based models—one for accuracy prediction, another for diversity embedding—using a dataset of ~3000 sampled architectures evaluated on validation sets. • Diversity Learning: Use triplet loss to embed architectures into a latent space where Euclidean distances encode prediction dissimilarity, promoting explicit diversity modeling. • Search Guidance: During search, filter architectures with high predicted accuracy, then select models that maximize pairwise distances in the latent space to ensure diversity. • Ensemble Construction: Use a greedy algorithm to iteratively pick architectures that balance high accuracy and maximal diversity, forming a robust ensemble. • Evaluation: Compare ensemble performance on datasets like CIFAR-100, measuring accuracy, negative log-likelihood, and diversity metrics.

The experimental setup involves training surrogate models on ~3000 architectures sampled from a reduced NAS search space across FashionMNIST, CIFAR-10, and CIFAR-100. Baselines include Deep Ensembles and random search. Metrics such as Top-1 accuracy, NLL, Brier score, and diversity indices are used for evaluation. Ablation studies analyze the impact of each surrogate component and dataset size on performance. Results show that the proposed method achieves higher accuracy and diversity, especially on CIFAR-100, with accuracy improvements of about 0.2-2% over baselines. The surrogate models demonstrate high correlation with true performance metrics after training on ~3000 architectures, validating their predictive utility.

On CIFAR-10, the ensemble achieves 97.8% accuracy, outperforming Deep Ensembles by 0.2%. On CIFAR-100, accuracy reaches 85.17%, surpassing baseline methods. The surrogate models trained on ~3000 architectures exhibit strong correlation with actual performance (correlation coefficient >0.65). Ablation results confirm that joint optimization of accuracy and diversity yields better ensemble results than single-objective approaches. The method significantly reduces search costs compared to exhaustive NAS, demonstrating scalability and effectiveness in complex datasets.

This framework can be directly applied to automated model design in industries requiring high robustness, such as autonomous vehicles, medical diagnostics, and financial modeling. It enables rapid construction of diverse, high-performing ensembles with reduced computational costs. Long-term, integrating transfer learning and weight sharing could facilitate real-time architecture adaptation and deployment in resource-constrained environments, transforming AI system development and deployment workflows.

Despite efficiency gains, the initial dataset construction still demands evaluating thousands of architectures, which is computationally intensive. The surrogate models' accuracy may decline on architectures outside the training distribution, affecting search quality. The multi-objective optimization process may converge to local optima, necessitating further global search strategies. Future work should focus on improving surrogate generalization, reducing sampling costs, and extending the framework to include additional objectives like latency and energy consumption.