Agentic Synthesis method cuts code churn 70% with counterexample-guided sketches

Coding agents often fix a single failing example but lose the underlying domain rule, causing repeated mistakes in later generations. Researchers propose agentic synthesis against counterexample-supplemented sketches, a repository-native approach where a human operator starts with a partial, code-shaped sketch and a coding agent generates the first implementation. When a concrete failure exposes missing or mistaken policy, the operator explicitly approves the corrected behavior and rule. The agent then revises the sketch and repairs or regenerates code and prompt surfaces solely for that counterexample. The system preserves full provenance, gates revisions through a selected regression set, and periodically tests whether the evolved sketch—not prompt history or accumulated examples—carries the learned policy.

In an open-world experiment with GPT-5.4-mini, the CatSynth implementation saw 8 of 14 frozen candidate cases become counterexamples. Rebuilding from the evolved sketch passed 19 of 21 withheld cases, compared with 15 of 21 when rebuilding from the initial sketch and replaying all accepted examples. Retaining code across counterexamples required 9 Developer calls and 719 lines of cumulative artifact churn, versus 15 calls and 2,394 lines for the replay-all approach—a 70% reduction in rework. While these results provide inspectable evidence that the evolved sketch carries reviewed policy and that retaining code reduces rework, the authors caution that the experiment uses only one model and one reveal order, so it does not establish general superiority beyond the encoded checks.