Multi-SPIN boosts edge AI goodput by 88% with cooperative token generation

Multi-SPIN extends speculative inference (SPIN) to a multi-user edge system, where resource-constrained devices run small language models (SLMs) to produce draft tokens, while a powerful edge server runs a large language model (LLM) to verify those drafts in parallel. This cooperative approach reduces latency and balances computational load. The key challenge is handling heterogeneity in device compute and communication capabilities. The authors formulate a joint optimization of draft length and bandwidth allocation under frequency-division multiple access (FDMA), maximizing sum token goodput (throughput of correctly generated tokens).

Two cases are analyzed: homogeneous draft lengths (simpler batching but less flexibility) and heterogeneous draft lengths (more efficient but requires careful scheduling). The optimal bandwidth allocation compensates weaker devices in the homogeneous case due to synchronization, whereas in the heterogeneous case, devices with higher acceptance rates are rewarded. Using Llama-2 and Qwen3.5 model pairs across diverse tasks, Multi-SPIN achieves up to 88% goodput improvement over heterogeneity-agnostic baselines. This work, from researchers at the University of Hong Kong and Singapore University of Technology and Design, provides a practical framework for deploying efficient LLM inference at the edge, enabling faster and more reliable AI applications on smartphones, IoT devices, and other resource-limited platforms.