The 1/W Law: An Analytical Study of Context-Length Routing Topology and GPU Generation Gains for LLM Inference Energy Efficiency

A team of researchers from Carnegie Mellon University and McGill University has published an analytical study revealing a fundamental trade-off in large language model (LLM) inference: energy efficiency plummets as context length increases. They've formalized this as the '1/W Law,' which states that the number of tokens a GPU cluster can process per watt of energy halves every time the context window doubles. This occurs because a larger context window consumes more GPU memory for the KV-cache, drastically limiting how many user queries (sequences) can be processed concurrently. For example, an NVIDIA H100 GPU can handle 256 sequences at a 4K context window for high efficiency (17.6 tokens/watt), but only 16 sequences at a 64K context, crashing efficiency to 1.5 tokens/watt.

The study's most striking finding is that intelligent system design can outweigh raw hardware gains. Using their inference-fleet-sim framework, the researchers show that implementing a 'two-pool context-length routing' topology—dubbed FleetOpt—can improve tokens-per-watt by roughly 2.5x over a standard homogeneous server fleet. In comparison, upgrading from an NVIDIA H100 to a next-generation B200 GPU yields only about a 1.7x gain. These gains are independent and multiplicative; combining FleetOpt routing with B200 hardware projects a 4.25x total improvement. The analysis also highlights the inherent energy advantage of Mixture-of-Experts (MoE) models like Qwen2.5-32B, which can achieve 5.1x better efficiency than dense models like Llama 3.1 70B, as their decode time scales with activated parameters, not total model size.