New Research Makes AI Chatbots 17% Faster Without Buying New Chips
Faster AI replies and lower bills — using the hardware we already own.
A new paper on arXiv proposes a way to make LLM serving more efficient. Prefill-decode disaggregation splits serving into two phases and improves efficiency through specialization and isolation — but those benefits rest on a static partitioning, and phase demand is not static. The authors observed that in a large LLM fleet, the ratio of uncached input to output tokens has peak-to-mean ratios up to 4.7x at minute timescales, and that in a public agentic trace the hourly ratio spans a median 24.5x within a single day — while reassigning a replica takes tens of minutes. Sizing each pool at its ninety-fifth percentile leaves up to 17% of cluster capacity unused; sizing below it converts the same imbalance into queueing and unrealized throughput.
The paper, by Yi Xu and 9 other authors, presents Crossflow, which makes this boundary elastic without changing node roles. Each decode node publishes a short-lived, revocable lease that bounds local-prefill compute, KV capacity, transfer work, and projected output. Across public and internal traces, Crossflow improves token throughput by 16.2–17.4% on geometric mean over static P/D, and by up to 43.4% at high load, while reducing mean TTFT at every evaluated point.
- AI servers split work into reading your question and writing the answer — and today that split is fixed, so half the machines often sit idle
- Crossflow lets the two sides share power on demand, boosting output 16-17% normally and up to 43% when traffic spikes
- Real traffic swings by up to 25 times in a single day, which is exactly why fixed planning wastes so much expensive hardware
Why It Matters
Cheaper, faster AI means snappier chatbots, lower subscription prices, and less energy burned per answer.