KernelFlume: New architecture cuts LLM serving costs by 61% with elastic attention scaling

A new research paper from Guangyu Xiang, Xueze Kang, and colleagues introduces KernelFlume, a decode-centric architecture designed to handle the growing demands of agentic, long-context LLM workloads. Traditional elastic scaling adds entire model replicas, wasting memory and incurring startup latency. KernelFlume instead disaggregates the stable projection/FFN path from core-attention computation: weight nodes handle dense kernels while stateless attention nodes store token-range KV partitions, scaling only the memory needed for request-state demand. It uses a dynamic routing table to map token ranges to attention nodes, updating routes at token boundaries with host-visible graph signals for low-overhead UCX communication.

On real GPU testbeds running Llama-3.1-8B under dynamic long-context agentic traffic, KernelFlume achieves flat p99 time-per-output-token (TPOT) of ~34ms on H100 and ~74ms on A6000, while lowering cost per million output tokens by 32% (A6000) and 61% (H100) compared to ServerlessLLM, a state-of-the-art instance-startup method. Simulation at larger model scales projects 56-66% cost reductions over ServerlessLLM, widening to 80-85% when using cheaper heterogeneous attention-node hardware, and persisting even into the million-token context range. The architecture combines query-first core-attention dispatch with inter-layer kernel pipelining to keep per-token latency low despite disaggregation.