COLE uses code embeddings to cut NAS search cost by 34%

A new study introduces Code-Oriented LM Embeddings (COLE), a low-cost approach to constructing performance predictors (surrogates) for Neural Architecture Search (NAS). Instead of fine-tuning language models or engineering complex graph encodings, the method simply represents each neural architecture as its raw PyTorch class definition text. An off-the-shelf LM (e.g., BERT) then extracts embeddings from this code, which are fed into a lightweight regression head to predict accuracy. The key insight: code already carries a strong inductive bias about the architecture's structure, making it highly informative.

Experiments on NAS-Bench-201 and einspace search spaces show COLE outperforms other text-based encodings (like ONNX-to-text) when using frozen LMs. When combined with the BANANAS algorithm for surrogate-assisted search, COLE drove a 34% reduction in the evaluation budget needed to reach within 1% of the fittest CIFAR-100 architecture (by test accuracy). Since any neural architecture can be written as code, COLE offers a general, fine-tuning-free foundation for NAS. The work appears at GECCO 2026, with code publicly available.