SONAR: New metric to evaluate AI-generated code summaries

Researchers Simantika Bhattacharjee Dristi and Matthew B. Dwyer from the University of Nebraska-Lincoln have introduced SONAR, a groundbreaking reference-free framework designed to evaluate source code summaries for their utility to LLM-based tools and agents in software engineering tasks. Traditional evaluation methods rely on human-written reference summaries or subjective preferences, which often fail to capture what makes a summary useful for an AI agent performing downstream tasks like code generation or bug fixing.

SONAR shifts this paradigm by introducing a task-aware evaluation approach that measures summaries across four key dimensions: Correctness, Abstraction, Conciseness, and Fluency. The framework leverages a novel code regeneration technique, where a summary is used to regenerate the original code. The quality of this reconstruction serves as an empirical signal of the summary's effectiveness, eliminating the need for human or LLM-based subjective judgments. The researchers found that Correctness and Abstraction significantly correlate with LLM performance, with Correctness demonstrating up to 14x higher correlation than existing baselines. Surprisingly, Conciseness and Fluency—traits highly valued by human developers—showed minimal impact on LLM performance, underscoring the task- and consumer-dependent nature of useful summaries.