NVIDIA GPU Frequency Scaling Under ML Workloads Challenges Latency Predictions

A new arXiv preprint by Truong-Thanh Le and colleagues investigates how NVIDIA GPUs dynamically scale their operating frequency when executing machine learning workloads. The preliminary findings show that on lower-performance GPUs, the frequency is strongly affected by the recent workload history, typically within an 80ms time window. This behavior contradicts a fundamental assumption behind many state-of-the-art ML latency-prediction techniques — that individual GPU kernel latencies are independent and can be summed to estimate total execution time. Instead, the GPU's dynamic frequency scaling creates inter-kernel dependencies, meaning the execution time of one kernel can influence the performance of subsequent kernels.

The implications are significant for anyone optimizing ML inference on GPUs. Current prediction models that ignore these dependencies may produce inaccurate latency estimates, leading to suboptimal resource allocation or scheduling. The researchers outline several promising directions to leverage this observation: building improved latency-prediction models that account for frequency coupling, developing GPU kernel-reordering strategies to mitigate frequency variation, and creating NAS-driven guidelines for frequency-, latency-, and energy-aware model design. For professionals deploying models on cost-sensitive or energy-constrained hardware (e.g., edge devices with lower-tier GPUs), understanding this frequency behavior could unlock more efficient inference pipelines.