Efficient Neural Architecture Search via Parameter Sharing

The field of neural architecture search (NAS) emerged as a promising approach to automate the design of neural networks, reducing reliance on manual trial-and-error. Early methods like Zoph and Le (2017) utilized reinforcement learning controllers to generate architectures, achieving state-of-the-art results in image classification and language modeling. However, these approaches required training each candidate architecture from scratch, often consuming hundreds to thousands of GPU hours, which limited their practical deployment. Recent efforts incorporated transfer learning, proxy tasks, and weight-sharing techniques to mitigate costs, but many still faced scalability issues. The core challenge remained: how to efficiently explore vast architecture spaces without prohibitive computational expense. This paper advances the field by integrating a parameter sharing scheme within a reinforcement learning framework, enabling rapid and scalable architecture search.

Despite the success of NAS in automating model design, its high computational cost remains a critical barrier. Traditional NAS methods involve training each candidate architecture independently, which is extremely resource-intensive. For example, NASNet required 450 GPUs over 3-4 days. This limits accessibility, especially for researchers with limited resources, and hampers rapid iteration. Additionally, the vast search space—potentially billions of models—makes exhaustive search infeasible. The key problem is how to drastically reduce the search time while maintaining or improving model performance. The challenge is to develop a method that can evaluate many architectures efficiently, ideally within hours, without sacrificing the quality of the discovered models.

The primary innovation of this work is the introduction of a weight-sharing mechanism into NAS, allowing all candidate architectures to share a common set of parameters ω. This eliminates the need to train each model from scratch. The search space is represented as a large DAG, with a controller RNN sampling subgraphs that define specific architectures. The controller is trained via policy gradients (REINFORCE) to maximize validation rewards, guiding the search toward high-performing models. The framework supports both recurrent and convolutional architectures, with the ability to explore billions of configurations. This approach significantly reduces computational costs, making NAS accessible and scalable, while achieving competitive or superior performance compared to prior methods.

• �� Construct a large DAG representing the entire architecture search space, with nodes as computation units and edges as information flow.
• �� Use an RNN controller (LSTM) to sample subgraphs by sequentially choosing edges and operations (e.g., convolutions, pooling).
• �� Implement a weight-sharing scheme where all candidate architectures share a common set of parameters ω, trained to minimize cross-entropy loss on training data.
• �� Alternately optimize ω (via SGD) with the controller fixed, and optimize the controller θ (via REINFORCE) with ω fixed, based on validation rewards.
• �� During training, the controller samples architectures, which are evaluated using shared weights, and the best-performing architecture is selected for final training.
• �� For recurrent cells, the search space includes sampling previous nodes and activation functions; for convolutional cells, decisions include skip connections and operation types.
• �� The process enables exploration of billions of models efficiently, with the entire search completed in hours rather than days.

The experimental setup involves two main tasks: language modeling on Penn Treebank and image classification on CIFAR-10. For Penn Treebank, the goal was to discover recurrent cells with less than 24 million parameters, achieving perplexity 55.8 in under 16 hours GPU time. The architecture was evaluated on validation perplexity, with the best model retrained from scratch. For CIFAR-10, the search space included convolutional cells with over 10^11 configurations; the discovered models achieved test error 2.89%, comparable to NASNet’s 2.65%, with only 7 hours of GPU search. Ablation studies confirmed the importance of parameter sharing and reinforcement learning in accelerating search while maintaining performance. Hyperparameters such as learning rates, regularization, and reward functions were carefully tuned to ensure stability and robustness.

ENAS achieved a perplexity of 55.8 on Penn Treebank, outperforming NAS and other baselines, with GPU time less than 16 hours. On CIFAR-10, the discovered convolutional architecture reached 2.89% test error, close to NASNet’s 2.65%, but with a fraction of the search time. The parameter sharing scheme enabled exploration of over 10^15 recurrent models and 10^11 convolutional models, demonstrating that high-performance architectures can be found efficiently. Ablation experiments showed that removing parameter sharing led to a 50x increase in search time and performance degradation, confirming its critical role. The results validate the scalability and effectiveness of ENAS across tasks.

ENAS can be applied to automate architecture design in natural language processing, computer vision, and speech recognition, significantly reducing manual effort and expertise required. Its efficiency makes it suitable for resource-constrained environments, enabling rapid prototyping and deployment of models. Additionally, the framework can be extended to multi-objective optimization, such as balancing accuracy and latency, facilitating hardware-aware model design. In industry, ENAS could accelerate the development cycle, reduce costs, and democratize AI research by lowering the barrier to high-performance model discovery.

Despite its efficiency, ENAS’s reliance on weight sharing may introduce bias toward certain architectures, potentially limiting diversity and optimality. The reinforcement learning component can suffer from high variance in gradient estimates, affecting stability and convergence. The current focus on cell-level search may not fully capture the complexity of entire architectures, especially for large-scale models. Future work should address these issues by integrating other optimization techniques, expanding search spaces, and improving stability and robustness.