Researchers Automate 4-Expert MoE Search, Expose Alphabetical Bias
Over 1,000 models tested in 28 days; alphabetical ordering skewed results.
This paper presents a systematic, automated approach to designing heterogeneous 4-expert Mixture-of-Experts (MoE) architectures. The researchers replaced manual design with a deterministic code-assembly generator that combines base architecture families from the LEMUR neural network dataset into MoE4 ensembles. Each ensemble uses a convolutional gating network with temperature scaling, mixup augmentation, and cosine-annealed learning rate scheduling. Over a 28-day campaign on a single NVIDIA RTX 4090, the pipeline produced 4,463 candidate models across 197 batches, with 1,021 successfully evaluated.
A critical finding emerged: due to alphabetical enumeration via `os.listdir()`, the entire search space (4.8% of the theoretical 23,751 possible 4-family combinations) was anchored to a single family—AirNet. This coverage bias is precisely characterized, and a stratified random sampling fix is proposed. Within the AirNet-anchored scope, ShuffleNet + MobileNetV3 consistently produced the highest accuracy (mean up to 0.632), while FractalNet and MNASNet were low-yield. The pipeline, analysis artifacts, and corrected generator are open-sourced as part of the NNGPT project.
- Automated pipeline generated 4,463 candidate MoE models, evaluating 1,021 in 28 days on an RTX 4090.
- Alphabetical ordering in file enumeration caused search space to anchor exclusively to the AirNet family, covering only 4.8% of possible combinations.
- ShuffleNet + MobileNetV3 ensemble achieved top mean accuracy of 0.632; FractalNet and MNASNet identified as low-yield families.
Why It Matters
Highlights subtle biases in automated ML pipelines and provides a scalable method for MoE architecture exploration.