Cornell's RAP method predicts LLM prompt reliability from few examples

A new preprint from Cornell researchers (Zheng et al., arXiv:2605.21515) tackles a critical reliability gap in LLM-based systems: the inability to predict whether a prompt program that works on a few test cases will actually perform well in production. The paper formalizes performance prediction using a simple coin-flip model, treating each pass/fail execution as a Bernoulli random variable. The key insight comes from compiling empirical performance priors across diverse programs and tasks. For symbolic programs (e.g., Python code), the prior is sharply bimodal—programs are either highly correct or utterly broken—so a few passing tests can effectively certify overall performance. In stark contrast, prompt programs (LLM instructions) exhibit a far more diffuse prior, with many programs hovering near-correct but still failing on edge cases. This explains why current testing practices for LLM prompts are dangerously misleading: a handful of successes may mask systemic weaknesses that only appear at scale.

The team then proposes RAP (Retrieved Approximate Prior), a practical method that retrieves similar tasks and prompt programs from an existing corpus to construct a proxy prior for a new, untested program. This prior is then combined with the observed test outcomes to produce a calibrated performance prediction. The authors demonstrate that RAP achieves solid predictive accuracy across multiple domains, offering a principled way to estimate the trustworthiness of LLM-based systems before full deployment. For engineers and product teams relying on prompt-based tools, this work provides both a theoretical explanation for a common frustration and a concrete solution: RAP can flag unreliable prompts early, potentially saving significant debugging and deployment time.