MODE Quantization Limits MoE Multimodal LLM Accuracy Drop to Just 2.9%

Mixture-of-Experts Multimodal Large Language Models (MoE-MLLMs) deliver impressive performance but come with prohibitive GPU memory costs, making compression essential. Existing expert-level mixed-precision quantization methods, effective for standard MoE-LLMs, suffer notable degradation on MoE-MLLMs due to two overlooked biases in expert importance estimation. First, at the cross-modal level, the numerical dominance of vision tokens causes expert selection frequency to be skewed by visual inputs, masking experts critical for text. Second, within the vision modality, a large proportion of redundant tokens further distorts frequency statistics, obscuring experts relevant to informative visual content. These biases lead to suboptimal bit-width allocation and accuracy loss.

To address this, the authors introduce MODE (Modality-Decomposed Expert-Level Mixed-Precision Quantization), a framework that decomposes expert selection frequency by modality, filters redundant vision tokens to obtain denoised visual frequency, and evaluates quantization sensitivity per modality as a complementary signal. These signals are integrated into an Integer Linear Programming formulation to assign per-expert bit-widths under a given memory budget. Extensive experiments demonstrate MODE's effectiveness: at W3A16 (3-bit weights, 16-bit activations), the average performance loss is limited to just 2.9%, with even larger gains at the extreme 2-bit setting. This work opens the door to deploying resource-intensive MoE multimodal models on edge devices or reducing cloud inference costs while maintaining near-lossless accuracy.