SEval-NAS: A Search-Agnostic Evaluation for Neural Architecture Search

A team of researchers has introduced SEval-NAS, a novel framework designed to solve a critical bottleneck in Neural Architecture Search (NAS). NAS automates the design of AI models but traditionally uses hardcoded evaluation procedures, making it difficult to assess new performance metrics, especially for hardware-aware objectives like latency and memory on edge devices. SEval-NAS breaks this constraint by providing a flexible, search-agnostic evaluation mechanism. It works by converting a neural network's architecture into a string representation, embedding that string into a vector, and then using that vector to predict various performance metrics. This decouples the evaluation logic from the search algorithm itself.

The researchers validated SEval-NAS on standard benchmarks NATS-Bench and HW-NAS-Bench, focusing on accuracy, latency, and memory. Results showed it was particularly effective at predicting hardware costs, with Kendall's τ correlation indicating stronger predictions for latency and memory than for accuracy. To prove its practicality, they integrated SEval-NAS into the FreeREA search algorithm to evaluate metrics not originally supported. The integration successfully ranked newly generated architectures without increasing search time and required only minimal code changes. This work, accepted for SAC26, provides an open-source tool that could significantly accelerate the development of efficient AI models for real-world, resource-constrained hardware.