Edge-cloud speculative decoding only pays off for multi-tenant servers

Speculative decoding (SD) accelerates LLM inference by 1.5–3x when draft and target models are co-located. A distributed variant (DSD) places the draft model on an edge device while the target remains in the cloud. In a new paper, Lyu, Irukulapati, and Champati derive closed-form inequalities showing DSD's per-request latency benefit is severely limited under WAN edge-cloud communication. If the server can host both models, co-located SD has lower latency and communication than synchronous DSD, with the same per-output FLOPs and model-weight memory. Pipelining can make DSD competitive only in low-RTT regimes where round-trip time is shorter than the edge drafting time window; at WAN RTTs, the cloud round trip remains too large for pipelined DSD to beat co-located SD. Against cloud autoregressive decoding, DSD can reduce latency only inside a bounded window dependent on target-model speed, acceptance rate, and RTT. DSD is also infeasible against closed-source APIs lacking a verifier-only interface.

The main case for DSD emerges in multi-tenant capacity. Under cross-client overlap, offloading draft compute lets a saturated cloud server sustain (1+γ t_d/t_v) times more concurrent clients at the same per-client rate, where γ is the speculation length and t_d, t_v are the per-step draft and verification times. The authors argue DSD should be evaluated primarily by multi-tenant capacity and server throughput, not single-request latency. This reshapes the design trade-off for edge-cloud LLM inference: the real value lies in server consolidation rather than per-query speedups.