Dual-Flow Transformers cut inference costs by separating prefill and decode
New architecture adds decode-only compute while sharing the same KV cache and weights
Large language model inference is dominated by two phases: prompt prefill, which is parallel and compute-bound, and autoregressive decode, which is sequential and memory-bandwidth-bound. Conventional architecture scaling—whether by width or depth—increases both costs together because every added layer runs in both phases. Researchers at Georgia Tech (Liming Liu, Mingze Wang, and Tuo Zhao) propose a novel solution: the Dual-Flow Transformer. This design separates the primary prefill path from auxiliary decode computation. The primary flow is a complete causal language model that processes the prompt and writes a single persistent key-value (KV) cache. The auxiliary flow is skipped during prompt processing and activates only from the final prompt position onward, adding continuation-prediction computation without writing persistent state or interfering with the primary flow. The two flows share major attention, MLP, and output matrices while using separate token embeddings and lightweight coupling, creating opportunities to reuse loaded weights and cached keys/values during grouped execution.
In matched-token comparisons, Dual-Flow achieves lower validation loss across different architectures and data configurations. The benefits are especially pronounced in mixture-of-experts (MoE) models, where the separation makes primary and auxiliary expert fan-outs independent controls over prompt cost, continuation cost, and predictive quality. The researchers studied two regimes: increasing decode computation while keeping prefill expert compute fixed, and reallocating a fixed decode expert budget between the two flows. These experiments reveal a fundamental prefill-decode-quality trade-off and demonstrate the potential of phase-specific expert allocation. By letting developers scale decode compute independently from prefill, Dual-Flow could dramatically reduce cumulative inference costs for high-throughput LLM serving, making advanced reasoning at scale more economical.
- Dual-Flow architecture decouples prefill and decode, enabling decode-only compute without expanding the persistent KV cache
- Achieves lower validation loss across architectures and data configurations in matched-token comparisons
- For MoE models, separates expert fan-out controls for prefill vs. decode, revealing a prefill-decode-quality trade-off
Why It Matters
This could slash cumulative inference costs for LLMs by scaling decode compute independently from prefill.