New arXiv study cuts LLM inference emissions 50.9% with carbon-aware routing

Aleks Bernhard and Arif Baran Yardimci's new arXiv paper (2608.06188) tackles one of AI's fastest-growing costs: the carbon footprint of LLM inference. Rather than retraining models or swapping hardware, they propose carbon-aware request routing—sending each inference to the grid region with the lowest marginal operating emissions rate (MOER) at that moment. The team validated this live on multi-region GPU testbeds, using NVIDIA DCGM telemetry to attribute per-request energy via measured concurrency curves instead of nameplate TDP. Their blind baseline was an actual production pressure-based router, not uniform placement, and every request was settled against historical MOER—not just the forecast that drove routing decisions. The live experiment showed feasibility: the MOER signal steered traffic across regions as a strict, reversible overlay with zero dispatch failures.

To quantify the impact, they replayed a year of hourly MOER data across a grid-diverse CONUS fleet. In the primary modeled configuration, carbon-aware placement reduced GPU-attributable operational emissions by 50.9% versus round-robin (95% block-bootstrap CI 48.5–53.3%). Because the replay dispatched against historical MOER rather than forecasts, this is an upper bound under that configuration; forecast error would reduce realized savings. Hourly lowest-MOER routing contributed about 22.4 percentage points—roughly 40% of the 54.0% placement reduction—beyond a static annual-mean-MOER policy. The authors also share a practical tip: when comparing regions, rank by absolute MOER, not percentile signal indices, which answer temporal rather than spatial questions. For data-center operators, this suggests significant decarbonization is possible with purely scheduling changes.