SpecBench and Buddy: New AI Benchmark Targets Intent-to-Specification Gaps

Current AI agents are excellent at executing well-defined software plans, but they falter when user intent is ambiguous. SpecBench, introduced by Hao Wang and colleagues, provides a new evaluation framework to test how well agents can turn fuzzy preferences into precise, executable specifications. The authors identify two extreme behaviors in existing agents: some rush into implementation mode, overestimating their understanding of user intent; others exhaust their limited question budget by asking about every minor design choice. SpecBench exposes these weaknesses by scoring an agent's ability to ask the right questions and produce a specification that aligns with the user's true, often unstated, preferences.

To solve this, the team developed Buddy, a user-assistant agent inspired by classical morphological analysis. Buddy first decomposes user intent into a structured space of design dimensions (e.g., feature trade-offs, performance priorities) and candidate choices. It then creates simulated user personas to evaluate these choices—reducing the burden on the real user. After narrowing down the possibilities, Buddy engages the real user with targeted, high-value questions to resolve remaining ambiguities and finalize the specification. By emphasizing specification over execution, SpecBench and Buddy open a new frontier in AI agent design: meaningful user collaboration rather than just code generation.