SAAG framework diagnoses agent failures to improve calling reliability

A new paper from Ritvik Garimella and colleagues at the University of South Carolina introduces SAAG (Structured Agent Assessment and Grounding), a diagnostic framework that breaks down AI agent function calling evaluation into three sequential stages: registry conformance (did the model pick a valid function?), structural completeness (did it provide all required arguments?), and argument grounding (are the argument values correct?). Unlike traditional single-score benchmarks that lump all failure types together, SAAG produces stage-specific diagnostics that help practitioners pinpoint exactly where and why an agent call fails.

Using a controlled benchmark derived from Glaive's function-calling dataset with registry sizes of 5, 10, and 15 agents, the team tested three local sub-4B-parameter models. SAAG's structured feedback consistently improved argument precision and reduced value hallucination relative to both single-pass inference and uninformative binary feedback. While end-to-end F1 gains were modest and model-dependent, the stage-decomposed approach proved necessary for understanding agent calling reliability across different model families and registry scales. The paper is accepted to the KnowFM workshop at ACL 2026.