MoEKD: Mixture-of-Experts Knowledge Distillation for Robust and High-Performing Compressed Code Models

A research team from the University of Saskatchewan has introduced MoEKD (Mixture-of-Experts Knowledge Distillation), a novel framework designed to create smaller, more efficient, and significantly more robust AI models for code analysis. The core innovation addresses a key weakness in standard knowledge distillation (KD), where compressing a large model into a smaller one often results in a dramatic loss of adversarial robustness. MoEKD overcomes this by leveraging a Mixture-of-Experts architecture, training multiple specialized 'expert' models and using a learned router to intelligently aggregate their knowledge before distilling it into a single, compact student model.

The framework was rigorously evaluated on the critical task of software vulnerability detection using established models like CodeBERT and GraphCodeBERT. The results are striking: models distilled with MoEKD showed an improvement in adversarial robustness of up to 35.8% while simultaneously boosting standard predictive performance by up to 13% compared to leading KD baselines. An ablation study further demonstrated its power for extreme compression, showing that ultra-compact models could maintain competitive performance even when their size was reduced by approximately half. This breakthrough suggests that multi-source knowledge aggregation is fundamentally more effective than distilling from a single teacher model.

Accepted to the EASE 2026 conference, this research provides a practical path forward for deploying powerful code intelligence in resource-constrained environments. By solving the robustness-performance trade-off in model compression, MoEKD enables the creation of efficient models that are not only accurate but also reliable and secure enough for real-world software engineering applications, from integrated development environments (IDEs) to CI/CD pipelines.