Talaria cuts LLM serving latency by 5.3x for agent sessions

Talaria is a new system from researchers (Meng et al.) that rethinks how serverless LLM serving handles multi-step agent sessions. Traditional systems treat each request independently, but tool-using agents make several LLM calls in rapid succession with shared context. This breaks the stateless abstraction: each continuation needs the same model weights and often a long reusable KV prefix. Naive round-robin scheduling can separate continuations from their model state, causing huge delays.

Talaria solves this with session-aware routing: it ranks placements by model residency, KV locality, and instance pressure, using soft reservations to account for likely returns. A session-prefill mechanism admits budget-eligible continuations before the active model slot closes. Combined with stable HBM addresses and host-restorable KV, it avoids expensive weight reloads and KV reconstruction. On a single 8-GPU server running three 100B+ parameter models over 30 SWE-Bench sessions (960 calls), Talaria achieved 5.3x faster median session completion (189s vs 1000s) and 2.6x at the 95th percentile (867s vs 2296s).