MoE quantization study: route flips detectable but harm not discernible

Mixture-of-Experts (MoE) models route each token to a subset of expert networks via a learned gate. That routing decision is discontinuous: tiny numerical disturbances can flip which experts fire, changing the model's output unexpectedly. Parvel Gu's new arXiv paper, "Detecting a Route Flip Is Easier Than Knowing Whether to Fix It," quantifies this effect under a common deployment condition: 4-bit KV-cache quantization, where the key-value cache is compressed to reduce memory, while the gate itself stays in BF16. On OLMoE-1B-7B, a 7-billion-parameter MoE with 1B active parameters, the author finds that roughly a third (RMF ≈ 0.31) of quantization-induced damage is routing-mediated—not intrinsic to weight precision, but caused by tokens being pushed across decision boundaries and activating different experts.

The paper doesn't propose a fix; it establishes a causal apparatus and a sobering detection-limit result. A token-level attribution decomposes damage by mechanism, and cross-architecture tests (three MoE models) confirm that the "signed-flip tax"—where flipping the set of active experts carries a measurable effect—persists across designs. A deployable router margin detects that a flip occurred with AUC 0.772, but crucially, no locally observable router statistic can tell a harmful flip from a helpful one; performance on predicting the flip's loss sign is at chance. That's an empirical benefit-detection barrier: selective repair strategies that rely on router diagnostics alone are fundamentally limited. Gu also tests a clean-reference remedy and finds its payout varies by architecture, suggesting gate normalization convention scales damage magnitude but doesn't enable route recovery. The study includes real int4 KV kernel results that are consistent with simulated quantization, though underpowered, and all hypotheses were pre-registered with misses reported.