New study: idle AI models waste 26-66W via 'parking tax'

The AI inference industry has long kept models loaded in GPU memory around the clock to avoid cold-start latency, treating idle power as a fixed cost. A new arXiv paper by Sai Sathvik Vadari, 'The Model Parking Tax: Quantifying the Hidden Energy Cost of Always-On GPU Model Deployment,' provides the first empirical decomposition of this cost across multiple GPU architectures. The study combines 18 days of production telemetry (335,267 samples from 14 NVIDIA H100 GPUs) with controlled dose-response experiments on three architectures spanning three memory technologies: H100 (HBM3, 80 GB), A100 (HBM2e, 80 GB), and L40S (GDDR6, 48 GB). The key finding: idle power is piecewise constant on all architectures. The CUDA context forces a discrete DVFS transition consuming +26-66 W over bare idle (26-50 W on HBM architectures, 66 W on GDDR6), while the marginal VRAM effect is bounded below measurement relevance (|β| < 0.02 W/GB) on every device tested. The CUDA context accounts for >98% of the parking tax regardless of memory technology. Validation with a real HuggingFace model (Qwen2.5-7B) confirms less than 0.5 W difference from empty tensors on every device, and cold-start power profiles during model loading are captured.

Beyond measurement, the paper derives a cold-start breakeven model showing that energy-optimal behavior depends on request arrival rate and loading latency—not model size. Breakeven intervals range from 1 to 5 minutes. This means that for deployments with sporadic inference traffic, it is often more energy-efficient to unload models between requests rather than keep them parked. The practical implication is significant: AI inference providers can reduce energy costs by dynamically managing model residency based on traffic patterns, rather than assuming always-on is the only option. The study's findings are consistent across all tested GPU architectures, suggesting a universal hardware constraint: idle-with-context power is determined by DVFS state, not memory occupancy. This challenges common assumptions and provides a clear target for optimization in the rapidly growing AI inference infrastructure.