Mixture of Experts with Soft Nearest Neighbor Loss: Resolving Expert Collapse via Representation Disentanglement

A team of researchers has proposed a novel solution to a persistent problem in large-scale AI models: expert collapse in Mixture-of-Experts (MoE) architectures. In standard MoE models, a gating network routes inputs to specialized 'expert' neural networks. However, overlapping features in the raw input data often cause multiple experts to learn redundant representations, a failure mode known as expert collapse. This forces the gating network into rigid, inefficient routing patterns, wasting computational resources and limiting model performance.

The researchers' key innovation is pre-conditioning the model's latent space using a Soft Nearest Neighbor Loss (SNNL) function before the data reaches the gating and expert networks. SNNL works by minimizing distances between data points of the same class in the latent representation, effectively clustering similar items and disentangling the feature space. This structural disentanglement prevents experts from converging on the same solutions, forcing them to develop highly orthogonal and specialized weights. The team quantified this improvement using new metrics: Expert Specialization Entropy and Pairwise Embedding Similarity.

In experiments across four major image classification benchmarks—MNIST, FashionMNIST, CIFAR10, and CIFAR100—the SNNL-augmented MoE models demonstrated structurally diverse experts. This diversity empowered the gating network to adopt a more flexible and effective routing strategy. The result was a significant boost in classification accuracy, particularly on the more complex FashionMNIST, CIFAR10, and CIFAR100 datasets, showcasing the method's potential to improve efficiency and performance in large, sparse AI models.