TokenPowerSandbox predicts LLM energy use with 6.23% error

Researchers face a costly dilemma when optimizing LLM serving for energy efficiency: exhaustive GPU profiling of every configuration is expensive, but cheap predictors can be dangerously overconfident outside their training scope. Chenxu Niu's TokenPowerSandbox, submitted to arXiv, tackles this with an evidence-gated pipeline that combines an interpretable CPU-resident projector, short target-GPU probes, full-workload verification, and tamper-evident freeze-before-measurement provenance. After calibrating on three anchor repeats and six development workloads, the frozen model was tested on a blind holdout and a separate predeclared no-refit confirmation totaling 51 post-freeze runs. The results on an NVIDIA H100 80GB serving Qwen2.5-7B-Instruct with vLLM: energy mean absolute percentage error (MAPE) of 6.23% and 7.35%, and Spearman rank correlations of 0.976 and 0.933, showing strong predictive fidelity.

Notably, the system also integrates a predeclared time-to-first-token (TTFT) gate that passes at concurrency four (9.27% MAPE) but triggers abstention below four (64.80% MAPE). This deliberate abstention is a key design choice: it acknowledges that an energy-focused model cannot automatically certify latency metrics. By forcing verification at low concurrency, TokenPowerSandbox trades a small amount of automation for reliability. The workflow is designed to be auditable and provenance-backed, making it suitable for production environments where model freeze and measurement integrity matter. For teams running LLM inference workloads, this offers a path to energy-aware configuration tuning with dramatically lower GPU profiling overhead, while keeping errors in the low single digits and providing honest uncertainty signals when data is insufficient.