AsymSpec speeds cloud-edge AI decoding 28x on asymmetric networks

Speculative decoding is a powerful technique to accelerate large language model inference by using a small draft model to generate candidate tokens that a larger target model verifies in batches. In cloud-edge setups, the draft model runs on an edge gateway while the target model lives in the cloud. But this introduces a critical bottleneck: every speculative block requires communication over asymmetric networks, where uplink bandwidth is often far more limited than downlink. When candidates queue at the edge, verification stalls, and optimistic runahead strategies risk wasted computation if a rejection invalidates dependent drafts.

AsymSpec solves this with two key mechanisms. First, its asymmetric verification protocol keeps the common-case acceptance upload compact, sending only a small top-K distribution when a total-variation (TV) certificate confirms the target's residual distribution is simple enough. If not, it progressively escalates to proposal-based exact recovery before falling back to the full distribution—all while moving richer, rejection-only correction information to the more abundant downlink. Second, the confirmed-prefix pipeline exposes only independent, valid requests to the edge scheduler and lets the cloud re-batch arrived blocks, hiding verification waits without risky same-request runahead. In end-to-end tests across three draft-target pairs, two workloads, and three asymmetric network profiles, AsymSpec achieved 2.82-28.03x the output-token throughput of the strongest baseline, making it a promising approach for real-world edge inference.