New paper maps design space for heterogeneous LLM inference and serving

A comprehensive new arXiv paper by Zhixin Wang and 17 co-authors provides the first systematic analysis of heterogeneous prefill-decode (PD) inference architectures now entering production. In these setups, the prefill phase runs on cost-efficient or supply-available accelerators while decode runs on bandwidth-strong ones, with KV state traversing mixed interconnects and numerical formats. The paper organizes the design space along four axes — accelerator type, precision, interconnect, and KV residency — and examines how different workload regimes (stage pressure) constrain each axis.

The authors identify three recurring boundary decisions that become binding once PD inference becomes heterogeneous: compute placement (which phase runs on which accelerator), KV representation (precision and format compatibility between producer and consumer), and KV ownership (lifecycle management of KV caches across prefill and decode). Key findings include that precision policy should be assigned per runtime role rather than system-wide, as low-bit formats relieve different bottlenecks on each side. KV transfer engines move raw bytes, not tensor semantics, making representation compatibility an explicit concern. The handoff also requires explicit ownership for reservation, release, and failure recovery. The paper provides actionable guidance grounded in industrial deployment observations and source-code inspection.