Cascade's latency-budget scheduler boosts LLM inference goodput 2.4x

Cascade is a new LLM inference serving system from Muhammad Adnan (Microsoft) and collaborators at UC Berkeley and Microsoft that tackles a core inefficiency in modern serving platforms: requests sharing the same response-latency SLO can differ by orders of magnitude in input length, generation length, and execution cost. The paper defines a per-request "latency budget" as the difference between the SLO deadline and the predicted remaining service time. This budget captures how much headroom a request has before it would violate its SLO. Cascade continuously estimates this budget from request characteristics, KV-cache state, and system load, updating it in real time.

Unlike prior SLO-aware schedulers that use deadlines only for request ordering, Cascade uses this single budget to coordinate both scheduling and KV-cache management across the memory hierarchy. The scheduler prioritizes requests with little remaining budget, while the memory manager uses the same budget to decide whether non-resident KV state should be restored, prefetched from deeper tiers, retained in HBM, or recomputed. This ensures queueing and data-movement overheads are directed toward requests that can absorb them, preserving fairness across heterogeneous request classes. Evaluated on production traces across three LLMs, Cascade achieves up to 2.4x higher SLO-satisfied goodput and 40% fewer SLO violations compared to vLLM's default first-come, first-served scheduler, showcasing the potential for latency-budget-aware inference serving to handle mixed interactive and long-running agentic workloads.