AGOPS auto-generates prompt guidelines, boosting LLM accuracy by up to 81.7%

A new paper from Cedric Richter, Salah Ghamizi, and Mike Papadakis tackles a critical weakness in LLM interactions: underspecified user queries. When prompts lack clear constraints, context, or evaluation criteria, models are forced to guess the user's intent—often dramatically reducing accuracy. The researchers measured drops up to 95.3% on tasks like mathematical reasoning, medical question answering, and coding, compared to well-specified prompts. Crucially, existing prompt optimization techniques (e.g., automatic prompt engineering or chain-of-thought tuning) failed to recover this performance loss because they assume a reasonably complete prompt to start with.

To solve this, the team introduces AGOPS (Automatic Guideline Optimization via Prompt Synthesis). The system treats writing guidelines as an optimization problem: it starts with a set of already-completed task examples (user queries with reference answers), uses a prompt LLM to propose candidate guidelines, a solver LLM to evaluate their effectiveness, and iteratively evolves the best guidelines. At inference time, these task-specific guidelines help users craft well-specified prompts from scratch. Across benchmarks, AGOPS recovered 15.5% to 81.7% of the lost performance, consistently outperforming baselines. The method is model-agnostic and works with any LLM, making it practical for production environments where vague user input is the norm.