Green AI

The evolution of AI since the advent of deep learning has been marked by rapid growth in model complexity and dataset sizes, exemplified by models like AlexNet, BERT, and GPT. These models have achieved remarkable breakthroughs in tasks such as image classification, language translation, and game playing. However, this progress has come with escalating computational costs, exemplified by the exponential increase in training resources, which has led to significant energy consumption and carbon emissions. Studies such as [40] highlight the environmental impact of training large models, equating their emissions to hundreds of cars' annual output. Despite awareness of these issues, the AI community predominantly emphasizes accuracy metrics, neglecting efficiency and environmental sustainability, resulting in a phenomenon termed 'Red AI.' This imbalance underscores the urgency for a paradigm shift towards greener practices.

The core challenge lies in the diminishing returns of scaling models in terms of accuracy relative to the exponential increase in computational resources. As models grow larger, their training costs and environmental footprint escalate sharply, yet the performance gains plateau or marginally improve. This creates barriers for researchers with limited resources and hampers sustainable development. The lack of standardized efficiency metrics hampers the ability to compare models fairly and to prioritize resource-conscious innovations. Addressing these issues requires redefining evaluation criteria to include resource consumption and environmental impact, fostering a more inclusive and sustainable AI ecosystem.

The paper introduces FPO (floating point operations) as a universal, hardware-agnostic measure of computational effort, enabling precise quantification of training and inference costs. It establishes a formal relationship between FPO, model complexity, and performance, revealing the law of diminishing returns in model scaling. The authors propose a linear cost estimation model based on per-example processing, dataset size, and hyperparameter tuning efforts, providing a practical framework for resource-aware model development. They advocate for integrating efficiency metrics into academic peer review and publication standards, promoting transparency and incentivizing resource-efficient innovations. This approach fundamentally shifts the evaluation paradigm from solely accuracy-centric to a balanced focus on resource utilization and environmental sustainability.

• �� Conduct comprehensive literature review and empirical analysis of models like ResNet, BERT, and GPT, focusing on parameters, FPO, and accuracy. • Collect data on training costs, dataset sizes, and hyperparameter tuning efforts from public sources and experiments. • Define FPO as a recursive function based on core operations (add, multiply), enabling hardware-independent cost estimation. • Analyze the relationship between FPO, model size, and accuracy to identify diminishing returns. • Develop a linear cost model: Cost(R) ∝ E · D · H, where E is per-example processing cost, D is dataset size, H is hyperparameter trials. • Propose reporting FPO and environmental metrics in research publications to promote efficiency transparency.

• �� Gather FPO, parameter count, and accuracy data for models like ResNet, ResNext, BERT-large, and GPT-3 across datasets such as ImageNet and COCO. • Analyze the correlation between FPO and accuracy, observing diminishing returns as resource investment increases. • Conduct ablation studies on model size and hyperparameter tuning to quantify efficiency gains. • Compare models of different scales, controlling for architecture, to evaluate FPO versus performance improvements. • Survey 60 recent top conference papers to assess the focus on accuracy versus efficiency, confirming prevalent overemphasis on the former. • Simulate hardware variations to validate FPO’s hardware-agnostic property, ensuring fair comparison.

• �� The exponential growth in training costs has not proportionally improved model accuracy, exemplified by a 35% FPO increase yielding only 0.5% accuracy gain in ResNet variants. • Analysis of 60 papers shows over 90% prioritize accuracy, with efficiency improvements under 20%. • Empirical data confirms the law of diminishing returns: larger models and datasets lead to smaller accuracy gains per resource invested. • FPO provides a reliable, hardware-independent measure of computational effort, facilitating fair comparison across models and platforms. • The cost estimation formula enables researchers to predict resource requirements and optimize model design for sustainability.

• �� Industry adoption of efficiency metrics like FPO can optimize training pipelines, reduce energy costs, and lower carbon emissions, especially for large-scale models. • Academic institutions can incorporate efficiency standards into peer review, promoting resource-conscious research and democratizing AI development. • Combining FPO with hardware energy monitoring enables real-time optimization of model training and deployment. • Techniques such as model pruning, quantization, and distillation can be guided by FPO metrics to produce lighter, more sustainable models. • Broader deployment in edge devices and mobile platforms benefits from resource-efficient models, expanding AI accessibility.

• �� FPO measures computational effort but does not directly quantify actual energy consumption or carbon emissions, which depend on hardware and energy sources. • The approach is primarily validated on deep learning architectures, requiring adaptation for other AI paradigms. • Cost estimates rely on assumptions about hardware efficiency and implementation quality, which may vary significantly. • Further integration with real energy and carbon footprint data is necessary for comprehensive environmental assessment. • The focus on resource efficiency might overlook other important factors like model robustness and fairness.