AWS SageMaker HyperPod separates LLM prefill and decode for faster streaming

Amazon has introduced Disaggregated Prefill and Decode (DPD) for LLM inference on SageMaker HyperPod, addressing a key bottleneck in production deployments. When prefill and decode share a GPU, a single long prompt (4K+ tokens) can stall concurrent token generation for all other requests, causing unpredictable latency spikes. DPD solves this by running the compute-bound prefill phase and the memory-bound decode phase on separate GPU pools, connected through Elastic Fabric Adapter (EFA) with RDMA.

The architecture uses an intelligent router that tokenizes each prompt and applies a configurable threshold. Long prompts go through a prefiller that computes KV cache and pushes it to a decoder via LMCache, while short prompts skip the prefiller to avoid unnecessary cross-GPU transfer. Prefiller pods use LMCache to cache recurring prefixes (system prompts, multi-turn history) in CPU memory, yielding significant TTFT gains. Decoder pods reserve GPU memory for incoming KV transfers with configurable PD buffer size.

This approach is ideal for high-concurrency streaming workloads such as chat assistants, document analysis, and RAG with large contexts. It allows independent tuning of TTFT and ITL, better tail latency control than chunked prefill, and automatic handling of mixed prompt lengths. The implementation builds on vLLM and the vLLM Production Stack router, with LMCache handling the KV cache transfer layer over NIXL and EFA.