Amazon's Cloud Trick Makes AI Chatbots Reply 77% Faster

Amazon SageMaker Inference has a new routing strategy called prefix-aware routing. When you build an application on top of a large language model, the prompt you send typically has two parts: a fixed part that sets up context (instructions, reference documents, conversation history) and a variable part with the actual user input. In a customer service bot example from the article, the instructions at the top might be 3,000 tokens while the customer's question at the bottom might be 50 tokens — so across hundreds or thousands of requests, the model processes that same beginning over and over.

LLM serving frameworks like vLLM and TensorRT-LLM cache the computed key-value (KV) pairs for prompt prefixes that have been seen before, reusing that computation and only processing the new tokens at the end. That's prefix caching, and it can significantly reduce time-to-first-token. The problem comes when you scale beyond a single instance: requests get spread across the fleet, and each instance computes the prefix from scratch because none of them see it often enough to build a reliable cache.

Prefix-aware routing looks at the beginning of each request and consistently sends requests with the same beginning to the same instance, so that instance's KV cache builds up and gets reused. You don't need to tag requests or manage affinity yourself. Two built-in safeguards: overload protection, which routes to a less busy instance if the target one is at capacity, and stable behavior during scaling, so caches don't get invalidated every time you add or remove instances.

In benchmarks on Llama 3.1 70B with prefix caching enabled, across 16 test configurations, this reduced P50 TTFT by up to 77 percent and increased throughput by up to 16 percent, pushing KV cache hit rates from roughly 25 percent to over 80 percent. The routing logic adds 1.3–1.9 milliseconds per request, while model TTFT in these tests ranged 63–280 milliseconds — a negligible cost.