AWS HyperPod's Curvine cache delivers 100% hit rate, 2.7x faster TTFT
New tiered KV cache reuses prompts across replicas at near-local-disk speeds—slashing GPU costs.
Running large LLM inference at scale forces a painful trade-off: over-provision GPU instances or recompute identical prompts on every request. AWS's new tiered KV cache on SageMaker HyperPod solves this by building a three-level hierarchy—L0 (GPU HBM), L1 (CPU memory), and L2 (shared NVMe via Curvine)—plus cache-aware request routing. Curvine pools node-local NVMe drives into a single namespace, mounted as a ReadWriteMany PVC across vLLM replicas, making it the key enabler for cross-Pod cache reuse. This means a request routed to any replica can benefit from cached attention keys and values built on another node, instead of suffering a cold start.
In testing, AWS achieved up to 100% cross-Pod cache hit rates, a 2.7x improvement in time-to-first-token (TTFT), and roughly 56ms L2 read latency for a ~1,900-token prompt—all on ml.g6e.4xlarge instances (48 GB per GPU). The architecture is especially impactful for teams serving model catalogs like Qwen, Llama, and DeepSeek across RAG pipelines or multi-turn dialogue apps. By extending cache capacity beyond GPU and local CPU, applications that previously required expensive P5 instances can now run on lower-cost G6e instances. The post walks through enabling HyperPod Tiered Storage, deploying Curvine workers, and patching the Inference Operator for filesystem-backed L2.
- Three-tier KV cache (GPU HBM → CPU → shared NVMe) with Curvine's distributed filesystem for cross-replica reuse
- Up to 100% cross-Pod cache hit rate and 2.7x TTFT improvement in AWS benchmarks on G6e instances
- Lower-cost G6e instances can replace P5 for some workloads; L2 read latency ~56ms for a 1,900-token prompt
Why It Matters
For teams running multiple LLM endpoints, this slashes infrastructure costs and latency by reusing KV caches across replicas at near-local speed.