TEXAS: New MoE fine-tuning method wins 17 of 18 benchmarks

MoE (Mixture-of-Experts) LLMs route each token through a small subset of experts, making routing patterns a promising signal for downstream adaptation. But existing methods have two key limitations: they identify task experts from aggregate routing statistics that only reflect usage, not actual success, and they fail to use expert activation signals to guide supervision allocation. TEXAS, proposed by Guanzhi Deng and eight colleagues, tackles both issues with a two-part approach. First, it discovers task experts by comparing expert activations on instances the base model solves successfully versus those it fails to solve, keeping experts that fire more strongly on successful examples. Second, during fine-tuning, it upweights answer tokens in failed instances that activate these discovered experts.

TEXAS leverages existing routing behavior without restricting adaptation to a fixed expert subset or imposing an explicit target routing distribution. Across three MoE models and six benchmarks, TEXAS achieves the best or tied-best performance in 17 of 18 settings, and beats the strongest baseline by 1.3–1.5 points on average. Ablations confirm that both the discovered experts and the resulting supervision strategy independently contribute to the gains. The method is architecture-agnostic and adds no inference overhead, making it a practical drop-in improvement for fine-tuning large-scale MoE models.