Implicit Intelligence -- Evaluating Agents on What Users Don't Say

Researchers Ved Sirdeshmukh and Marc Wetter have introduced 'Implicit Intelligence,' a groundbreaking evaluation framework that exposes a fundamental weakness in current AI agents: their inability to understand what humans don't explicitly say. The paper argues that real-world requests are inherently underspecified, relying on shared context and unstated constraints that current benchmarks completely miss. The framework specifically tests whether agents can reason about implicit requirements spanning accessibility needs, privacy boundaries, catastrophic risks, and contextual constraints—moving beyond simple prompt-following to become genuine goal-fulfillers. This represents a paradigm shift in how we evaluate AI assistants.

The team paired their framework with 'Agent-as-a-World' (AaW), a novel harness where interactive environments are defined in human-readable YAML files and simulated by language models. These scenarios feature deceptively simple user requests that hide complex correct solutions, requiring agents to discover constraints through environmental exploration. When testing 16 frontier and open-weight models across 205 scenarios, the results were stark: even the best-performing model achieved only a 48.3% scenario pass rate. This reveals substantial room for improvement in bridging the gap between literal instruction-following and human-like contextual reasoning, suggesting current AI assistants are far from understanding the implicit dimensions of human communication.