Route-block membership drives exact arithmetic in MoE inference
A single-fixture study reveals how route-block membership selects packed-AWQ arithmetic trajectories.
Mixture-of-Experts (MoE) inference relies on aligning routed tokens into padded expert blocks before executing packed quantized matrix multiplication. Until now, this preprocessing has been treated as mere bookkeeping. Lukas Stepanek's study on a Qwen3-Coder AWQ layer-6 fixture (vLLM/Marlin on RTX 3090) proves that route-block membership directly selects exact packed arithmetic trajectories. Two fixed preconstruction histories produced distinct native alignments and exact trajectories. Injecting the opposite alignment transferred W13, activation, routed-W2, and final outputs, demonstrating causal dependence.
Further experiments showed that swapping two routes within one block preserved each native trajectory, but exchanging two routes across the boundary between expert-106 blocks 40 and 41 transferred the complete opposite trajectory. Source- and binary-derived schedule geometry maps those blocks to direct/full-K and split/global-reduction classes. Forcing a single-slice 200-block grid made W13 bitwise equal. Stable canonical construction made both histories converge to a third exact trajectory. The confirmatory cohort contains 70 valid cold processes and 7 required perturbation rejections. The paper is a causal mechanism result for one fixture, not a prevalence claim.
- Route-block membership determines exact packed quantized multiplication trajectories in MoE inference.
- Exchanging routes across specific block boundaries (expert-106 blocks 40 and 41) transferred the complete opposite arithmetic trajectory.
- Forcing a single-slice 200-block grid made W13 bitwise equal; stable canonical construction produced convergence to a third trajectory.
Why It Matters
Proves deterministic arithmetic based on route-block membership in MoE, enabling reproducibility and potential optimization.