Leyline's KV cache directives cut agentic LLM latency by 241ms

Modern LLM serving systems assume a chatbot workload: prompts arrive once, the cache grows append-only, and prefix caching works perfectly. But agentic LLMs break this assumption. Their conversations evolve through policy-driven edits—failed tool calls are retried, stale outputs dropped, and trajectories pivoted. Existing KV cache management can't handle this: identical content moves to new positions (breaking exact-prefix caches) and policies need to actively remove or replace cached spans. Production systems fall back to full re-prefill on every edit, wasting compute. Now, researchers introduce Leyline, a serving-side primitive that closes this gap. It accepts a declarative 4-tuple directive that separates what to edit from how to preserve position correctness. Two modes are supported: in-place splice (reuses existing KV with a RoPE-rotation correction) and prefix-trimmed re-prefill (for semantic forgetting). The interface is architecture-agnostic, routing to per-architecture kernels.

Results are striking. The splice kernel lifts replay cache-hit by +11.2 percentage points and cuts latency by up to 241 milliseconds. A ten-line truncation rule, routed through the same interface, lifts agentic solve rate by +14.3 percentage points on the debug-gym benchmark. Leyline's key insight is that cache eviction should accept policy directives from outside the kernel, rather than making autonomous decisions. The paper (arXiv:2606.01065) frames this as an open mechanism—the policy space it enables is the real agenda. For production agentic systems, this means dramatically lower inference costs and faster iteration on complex tasks like multi-step reasoning, tool use, and code debugging.