Position: LLM Inference Should Be Evaluated as Energy-to-Token Production
Proposes energy-to-token model using Leontief function; emphasizes energy consumption as key constraint; recommends reporting Joules/token.
Key Findings
Methodology
This paper introduces a Token Production Function combining effective compute (Keff) and delivered power (PIT) to analyze LLM inference under physical constraints. Using a Leontief model, the minimum of compute throughput and energy supply bounds the token output at fixed quality (q*) and service (s*). System optimizations like KV-cache compression, sparse attention, and quantization are framed as energy levers that reduce FLOPs/token and Joules/token, thus expanding the energy boundary. The framework advocates reporting Joules/token, active binding constraints, and PUE-adjusted delivered power to better reflect real deployment limits.
Key Results
- Empirical data from 2024-2026 shows global data center electricity rising from 415 TWh to 945 TWh, with energy per token (etok) decreasing by factors of 3-10 through optimizations like INT4 quantization and KV compression. GPU measurements indicate etok can be reduced to 10-30% of initial values while maintaining accuracy.
- Advanced techniques such as FlashAttention, MLA, NSA achieve over 10× improvements in energy efficiency, validating the energy-to-token frontier shift. Long-context generation now faces power constraints, with optimized architectures supporting million-token inputs at lower energy costs.
- The paper emphasizes reporting Joules/token and utilization-adjusted token output alongside traditional metrics, fostering a more comprehensive understanding of deployment efficiency and guiding infrastructure investments towards sustainable AI.
Significance
This work shifts the evaluation paradigm from purely model-centric metrics to physical resource constraints, highlighting that infrastructure efficiency and energy consumption are now central to deploying large models. It provides a theoretical framework and practical metrics for assessing AI systems' true cost and sustainability, aligning AI development with environmental goals. By quantifying energy boundaries, the study enables more informed decisions on hardware design, system optimization, and policy, fostering greener AI practices.
Technical Contribution
The paper formalizes a physical Token Production Function grounded in Leontief’s model, integrating compute throughput and delivered power as joint constraints. It maps micro-level optimizations to macro-level energy boundaries, introduces standardized reporting metrics, and demonstrates how system-level improvements (compression, sparse attention) act as energy multipliers. This approach bridges hardware, system engineering, and energy policy, offering a unified framework for energy-aware AI evaluation.
Novelty
First to formalize inference as an energy-to-token production process with a Leontief-based model, emphasizing physical constraints over traditional FLOPs-centric metrics. It innovatively links system optimizations to macro energy boundaries, providing a new perspective on sustainable AI deployment and evaluation. This approach addresses the gap between hardware capabilities and real-world energy limitations, offering a comprehensive assessment framework.
Limitations
- Energy consumption estimates depend on hardware and system parameters, which vary across platforms and operational conditions, introducing uncertainties.
- The framework primarily relies on GPU data (H100), requiring validation on other hardware like TPUs or FPGA for broader applicability.
- Optimizations may impact latency or model accuracy; balancing energy efficiency with performance remains a challenge, especially in latency-sensitive applications.
Future Work
Future research should develop real-time energy monitoring tools, standardize reporting metrics, and extend the model to diverse hardware architectures. Exploring dynamic scheduling and adaptive algorithms could further optimize energy use in varying deployment scenarios, advancing sustainable AI infrastructure.
AI Executive Summary
As large language models (LLMs) become integral to AI applications, their deployment faces a critical bottleneck: energy consumption. Traditional evaluation metrics such as accuracy, latency, and throughput, while necessary, do not fully capture the physical constraints of real-world infrastructure. This paper advocates a paradigm shift, framing LLM inference as an energy-to-token production process constrained by physical limits of compute and delivered power.
The authors introduce a formal Token Production Function based on Leontief’s model, which captures the joint constraints of effective compute throughput (Keff) and delivered power (PIT). Under fixed quality (q*) and service (s*) targets, the model identifies whether the bottleneck is computational capacity or energy supply. System optimizations—like KV-cache compression, sparse attention, and quantization—are reinterpreted as energy levers that effectively raise the energy boundary, enabling more tokens to be produced per unit energy.
Empirical data from 2024 to 2026 shows a significant increase in global data center electricity consumption, from 415 TWh to 945 TWh, while energy per token (etok) decreases by factors of 3-10 due to these optimizations. Techniques like FlashAttention, MLA, and NSA achieve over tenfold improvements in energy efficiency, demonstrating that system-level innovations are crucial for sustainable AI. The authors emphasize the importance of reporting Joules/token, active binding constraints, and PUE-adjusted delivered power alongside traditional metrics, to better reflect deployment realities.
This framework provides a comprehensive basis for evaluating and designing energy-efficient AI systems, aligning technological progress with environmental sustainability. It encourages industry adoption of standardized energy metrics, promotes hardware-software co-optimization, and guides infrastructure investments toward greener, more efficient AI deployment. Ultimately, this work aims to transform AI evaluation from a purely algorithmic focus to a physically grounded, sustainability-oriented paradigm.
Deep Dive
Plain Language Accessible to non-experts
想象你在经营一家工厂，生产玩具。以前，我们只关心每天能生产多少玩具，花多少钱，但没有考虑工厂的电费和冷却系统。其实，工厂的产量不仅取决于机器的速度，还受电力和冷却的限制。如果电力不足或冷却不够，工厂就不能多生产玩具。这个研究告诉我们，AI模型就像这个工厂，它们需要电力和冷却才能工作。通过优化硬件和算法，我们可以让模型用更少的电生产更多的Token，就像让工厂变得更节能。这样，AI不仅能变得更快，还能更环保，减少对能源的依赖。
ELI14 Explained like you're 14
你知道吗？就像一个工厂每天生产玩具一样，AI模型也需要电和冷却才能运行。以前，我们只关心模型能生产多少Token（就像玩具的数量）和速度，但其实，电力和冷却的效率也很重要。如果电不够，模型就不能多工作。现在，科学家们发现，除了让模型变得更快外，还可以通过让它们用更少的电来生产相同的Token。比如用新技术让硬件更省电，或者优化算法减少能耗。这样，未来的AI既能快又能省电，就像一个环保又高效的工厂！
Abstract
LLM inference is still evaluated mainly as a model or software problem: accuracy, latency, throughput, and hardware utilization. This is incomplete. At deployment scale, the relevant output is a quality-conditioned token produced under joint constraints from effective compute, delivered data-center power, cooling capacity, PUE, and utilization. We argue that the ML community should treat inference as \emph{energy-to-token production}. We formalize this view with a dimensionally consistent Token Production Function in which token rate is bounded by both compute-per-token and energy-per-token ceilings. Listed API prices vary by over an order of magnitude across providers, but we use price dispersion only as directional motivation, not as causal evidence of marginal cost. The core physical question is instead: under fixed quality and service targets, when does the binding constraint move from theoretical peak compute toward delivered power, cooling, and operational efficiency? Under this framing, system optimizations -- latent KV-cache compression, sparse or heavily compressed attention, quantization, routing, and difficulty-adaptive reasoning -- are not merely local engineering tricks. They are energy-to-token levers because they reduce FLOPs/token, joules/token, memory traffic, or utilization losses under fixed $(q^{*},s^{*})$. We therefore call for inference papers and benchmarks to report Joules/token, active binding constraint, PUE-adjusted delivered power, and utilization-adjusted token output alongside accuracy and latency.