StateFlow boosts MoE inference on 6G edge-cloud by 2x concurrency

A team of researchers led by Bo Liu has unveiled StateFlow, a novel distributed inference policy designed for Mixture-of-Experts (MoE) architectures operating across 6G edge-cloud networks. The core challenge StateFlow tackles is the inherent tension between KV state locality and elastic expert dispatch. In multi-turn inference, each turn extends the KV state that must persist across the dialogue, creating a conflict: KV state relocation incurs substantial transfer overhead, while expert computation benefits from spreading across the network to exploit available capacity. StateFlow resolves this by decoupling persistent KV state from transient sparse computation — it pins the KV state at a sticky serving site for cross-turn reuse and jointly optimizes expert dispatch and aggregation placement.

The researchers implemented a real-world testbed with kernel-level network emulation to validate StateFlow. Experimental results show the policy sustains more than 2x higher stable dialogue concurrency compared to distributed baseline solutions, and reduces turn-level p95 latency under multi-turn inference by 53.0%. This breakthrough is particularly relevant as MoE models — known for sparse activation that reduces computational footprint — become increasingly deployed across edge-cloud networks in the 6G era. The work directly addresses the scalability bottlenecks that have limited practical deployment of large language models in low-latency, resource-constrained edge environments.