MAGE framework reveals prompt optimization coupling effect boosting performance but amplifying variance

Researchers from the AI community (author Prateek Singh) have released a paper on arXiv titled "MAGE: Understanding Stability-Performance Trade-offs in Multi-component Prompt Optimization." The work introduces MAGE (Memory-Augmented Goal-directed Prompt Evolution), a controlled ablation framework that integrates episodic memory, multi-objective Pareto selection, and adaptive evaluation — not as a superior optimizer but as a platform to dissect how different components of iterative prompt optimization interact. The key discovery is the Prompt Optimization Coupling Effect (POCE): when multiple stochastic optimization signals operate within a closed reflective loop, they interact in ways that simultaneously improve performance and amplify variance, a behavior that cannot be predicted by analyzing components in isolation.

Three main findings emerge from experiments on GSM8K-Hard using gpt-4o-mini. First, failure-grounded reflection is essential — methods relying only on scores (OPRO) or abstract critique (Self-Refine) fail to improve prompts. Second, MAGE achieves 46.4% accuracy versus GEPA's 34.0% (+12.4%, P(MAGE>GEPA)=0.998, 5 seeds) with comparable variance (7.3% vs. 7.0%). Third, increasing candidate diversity reveals the clearest POCE signal: expanding the candidate pool from n=3 to n=5 improves mean accuracy by +21.6% while increasing variance by 3.7x. Validation on Llama 3.1 8B shows POCE is headroom-dependent — when the base model already achieves high accuracy, variance amplification disappears. Critically, in low-data regimes (Ntrain=30), well-designed fixed prompts outperform all reflective optimizers, indicating that scaffold choice dominates optimizer choice.