HYPIC speeds hybrid-attention LLM serving 2.45x with position-independent caching

Retrieval-augmented generation (RAG) and agentic LLM workloads often assemble prompts from independent segments, making the prefill stage the dominant cost. Two promising techniques have emerged: position-independent caching (PIC) reuses KV caches across non-contiguous segments, and hybrid-attention models reduce computation by replacing most full-attention layers with linear attention. However, they have been incompatible since PIC's per-token caching primitives don't work with the recurrent state of linear attention layers.

HYPIC, presented in a new arXiv paper, solves this with two key innovations. For linear attention layers, it identifies the segment-cumulative transition operator as the missing algebraic primitive and caches it alongside each segment's zero-start end-state, enabling near-exact, constant-time composition of cached segments. For the remaining full-attention layers, HYPIC finds that attention deviation concentrates at segment boundaries, so it recomputes only a small seam window at each boundary to restore cross-segment lookback. Additionally, HYPIC exploits segment-level self-containment to parallelize cache-miss prefill across GPU instances, turning long cold requests—a major tail-latency culprit—into an accelerable workload.

Tested across four hybrid-attention models and five workloads, HYPIC reduces time-to-first-token by 2.45x on average and improves peak throughput up to 2.0x over existing systems, while maintaining accuracy within 3.3 points of full recompute. This breakthrough makes RAG and agentic LLMs significantly faster and more cost-effective.