Researchers Build Transferable Latency Model to Speed LLM Screening on Edge Devices

Accurate latency prediction is critical for deploying large language models (LLMs) on edge devices where hardware varies widely. Researchers from Georgia Southern University and Intel Labs present a transferable latency prediction framework that accounts for model architecture, prompt behavior, runtime backend, DVFS, and thermal variation. The system separates each inference into prefill and decode phases, then uses a gated model to fuse static descriptors (model/prompt features) with dynamic hardware telemetry (CPU/GPU frequency, temperature). This approach enables runtime-aware LLM screening without exhaustive on-device profiling.

Evaluated across five platforms — from Pixel smartphones to Jetson Nano, Orange Pi 5 Pro, and an RTX 3090-class GPU — the framework shows significant gains. On Pixel 8, total-latency R-squared improves from 0.953 to 0.960, while decode-latency jumps from 0.957 to 0.973. Cross-device transfer with calibration boosts Pixel 8 Pro to Pixel 8 total-latency R-squared from -0.974 to 0.940. The paper also highlights extreme device variance: the same SmolLM2 model achieves 8.42 tokens/s on Orange Pi 5 Pro vs. 64.38 tokens/s on an RTX 3090. This work enables cost-effective, latency-aware LLM deployment across fragmented edge ecosystems.