LLMs Optimize Neural Network Widths, Cutting Parameters 14x in New Study
New research shows LLMs designing networks 14x smaller while boosting accuracy.
A new arXiv paper from Uzun et al. extends prior work on using LLMs for closed-loop feature channel configuration—essentially, letting a language model propose and test neural network width designs via executable code generation and accuracy feedback. The researchers scaled the search to 250 candidate networks per fine-tuning cycle, running 8 full cycles for a total of 2000 generated candidates. After filtering, 462 verified evaluations on CIFAR-100 were analyzed. The mean accuracy per cycle showed a statistically significant positive trend (slope 9.87e-4, p=0.043), while the best observed accuracy improved by over 17% from 0.3144 to 0.3676.
The scaled run also revealed dramatic parameter efficiency: the best model achieved 0.3676 accuracy with only 11.8M parameters, compared to an earlier high-performing model at 0.3144 with 166.5M parameters—a 14x reduction. Beyond raw numbers, the larger sample exposed architectural regularities not visible in sparse evaluations. Non-power-of-two channel widths occurred in 41.8% of verified candidates, and top models shared structured allocation patterns featuring moderate early widths and expanded middle or later blocks. These findings suggest that LLM-guided neural architecture search can uncover efficient, unconventional designs that outperform hand-tuned heuristics, with significant implications for model compression and automated ML.
- Scaled from sparse evaluations to 250 candidates per fine-tuning cycle, with 462 verified CIFAR-100 results.
- Best accuracy rose 17% (0.3144→0.3676) while parameters dropped 14x (166.5M→11.8M).
- 41.8% of candidates used non-power-of-two channel widths; top models showed structured allocation patterns.
Why It Matters
LLMs can automate neural architecture search, producing far smaller models with higher accuracy—key for edge deployment.