New study: LLM hosting costs vary 36x based on concurrency, not token count

A new research paper by Chitral Patil exposes a fundamental flaw in every public LLM cost calculator surveyed: they treat GPU utilization as a fixed input, never measured against actual load. On identical H100 hardware, effective cost per million output tokens ranged from $0.21 to $15.25—a 36.3x penalty near idle and 2.5-24x at low-to-moderate enterprise loads (1-10 rps). The root cause is concurrency, governed by Little's Law via the operator's offered request rate lambda, a variable no open-source calculator exposes. Because calculators assume 100% utilization, any utilization-naive estimate understates true cost by exactly 1/U, systematically mispricing self-hosting—especially for low-traffic workloads.

Patil proposes a measurement methodology parameterizing effective cost as C_eff = f(H, M, Q, lambda, L), validated with 42 benchmarks across dense, ultra-sparse MoE, and sparse MoE models. He releases vllm-cost-meter, an open-source cost meter that attaches to a live vLLM server and reports real $/M-tokens against the operator's own traffic. Additional findings: FP8 quantization benefits MoE architectures roughly 2.2-2.4x more than dense models (+69 to +74% vs. +31% peak throughput), and active parameter count, not total model size, appears to be the primary predictor of saturation economics. The core sweep repeated on A100 80GB PCIe reproduced the load-driven spread at 7.0-11.4x, confirming the concurrency effect is hardware-independent.