Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster

The evolution of large language models (LLMs) has been driven by the transformer architecture, with models like GPT-2, GPT-3, and open-source variants such as GPT-NeoX and OPT achieving remarkable performance. These models demonstrated that increasing parameters and data improves capabilities, but at a steep computational cost. Recent research emphasizes the importance of scaling laws, revealing predictable performance gains with model and dataset size. The Chinchilla paper highlighted the efficiency gains from optimal data usage, challenging the traditional focus on larger models alone. Despite progress, training large models remains resource-intensive, prompting efforts to optimize compute utilization, data efficiency, and stability. Open-source models have advanced transparency but often lack compute-efficient scaling strategies, limiting practical deployment. This background underscores the need for systematic, reproducible scaling laws and training techniques that balance performance with resource constraints.

The core challenge lies in training large language models efficiently, balancing compute costs, data utilization, and stability. Traditional methods often lead to unstable training at scale, requiring extensive hyperparameter tuning and hardware resources. Moreover, the lack of open, reproducible scaling laws hampers community efforts to predict model performance and optimize training strategies. Addressing these issues is crucial for democratizing access to powerful models, reducing environmental impact, and enabling deployment in real-world applications where resources are limited. The problem is compounded by the difficulty of transferring hyperparameters across scales and ensuring training stability without sacrificing accuracy.

This work introduces several innovations: 1) Applying Chinchilla's power-law scaling to validate compute-efficient training on the Pile dataset, providing a theoretical foundation for model scaling. 2) Integrating μP to stabilize training dynamics, enabling hyperparameter transfer across scales and reducing tuning complexity. 3) Training a series of open-source models up to 13B parameters on Cerebras hardware, demonstrating practical scalability. 4) Developing a cost-aware framework that balances pretraining FLOPs and inference costs, guiding resource allocation. These innovations collectively address stability, efficiency, and reproducibility challenges in large-scale language model training.

• �� Design Transformer decoder architecture with dense attention, adjusting dimensions per model size. • Use Eleuther Pile dataset, tokenized with BPE and GPT-2 vocabulary, ensuring diversity. • Initialize weights with truncated normal, applying μP to control layer-wise parameters for stability. • Optimize with AdamW, employing linear or cosine learning rate decay, with warm-up over 375M tokens. • Monitor FLOPs and loss to validate power-law scaling, adjusting batch sizes for stability. • Train on Cerebras CS-2 clusters, leveraging wafer-scale hardware for massive parallelism. • Use mixed precision (bfloat16) to improve numerical stability and training efficiency. • Transfer hyperparameters via μP, reducing tuning efforts across scales, and validate on downstream tasks.

Models trained on Eleuther Pile across scales, measuring test loss and efficiency via FLOPs. Comparisons with GPT-J, GPT-NeoX, and Pythia highlight superior compute efficiency. Hyperparameter transfer experiments demonstrate μP's stability benefits, with loss reductions of 0.4%. Downstream evaluations on reasoning tasks confirm the models' robustness, with average accuracy improvements of 1.7%. Ablation studies examine the impact of data tokens per parameter and hyperparameter settings, validating the theoretical scaling laws. Hardware deployment on Cerebras CS-2 confirms practical scalability and efficiency gains.

Cerebras-GPT models consistently outperform comparable open models in test loss at equivalent FLOPs, with the 13B model achieving 0.5% lower loss than GPT-NeoX 20B. The power-law scaling law accurately predicts performance at larger scales, with loss decreasing predictably as model size increases. μP enhances training stability, enabling hyperparameter transfer and reducing variance in results. Downstream tasks show the 13B model surpasses others in average accuracy, confirming the effectiveness of compute-optimal scaling. These results demonstrate the feasibility of training large, efficient models with open resources, setting new benchmarks for open science.

The models are suitable for applications requiring high-quality natural language understanding, such as chatbots, content creation, and automated reasoning. They can be deployed in environments with limited resources by leveraging model compression and sparsity techniques. The open models facilitate research in AI safety, interpretability, and domain adaptation, enabling broader community participation. In industry, they support scalable AI solutions for customer service, knowledge extraction, and decision support, especially where compute resources are constrained.

Despite efficiency gains, training large models remains costly and hardware-dependent, limiting accessibility. Transferability to specialized domains needs further validation, and the models' environmental footprint warrants concern. Hyperparameter tuning, although simplified by μP, still requires expertise. Future work must focus on reducing costs, improving domain adaptation, and integrating model compression to broaden applicability.