Contrastive Reflection boosts LLM prompt accuracy by 9% on HotpotQA
A new iterative method uses contrastive examples to repair LLM agent prompts, boosting exact-match from 51.4% to 60.4%.
A team of researchers led by Derek Koh has released a paper on Contrastive Reflection, a framework for iteratively optimizing prompts in LLM-based information retrieval agents. The method treats prompt optimization as a debugging problem: rather than blind search, it uses structured traces from QA and grading agents to identify specific behaviors that failed. It then contrasts those failures with nearby successful examples and asks a Teacher LLM to propose a targeted edit. The edit is accepted only if validation performance improves, with optional regression checks.
In experiments on the HotpotQA dataset, a single contrastive repair improved held-out exact-match accuracy from 51.4% to 60.4% — a 9-point gain. Failure-only and random-evidence variants improved less and broke more previously correct examples. Compared to other modern optimizers, Contrastive Reflection reached 60.4% versus MIPROv2's 59.4% and GEPA's 57.0%. The paper emphasizes interpretability: the loop produces inspectable slices and rationales, making it easier for engineers to understand why a prompt change works. The work will appear at the Agent4IR workshop at KDD 2026.
- Contrastive Reflection uses a tree-based slice selector to find error-anchored behavioral slices and adds nearby successful examples for contrastive repair.
- One iteration improved HotpotQA exact-match accuracy from 51.4% to 60.4%, outperforming MIPROv2 (59.4%) and GEPA (57.0%).
- The framework accepts prompt edits only when validation performance improves, optionally with regression checks to avoid breaking working examples.
Why It Matters
Makes prompt optimization for LLM agents more systematic, interpretable, and less prone to regressions in real-world IR tasks.