CELEUS framework cuts LLM evaluation samples by 54-62% with certifiable guarantees
New method uses e-processes for anytime-valid confidence intervals in LLM testing
Deep Dive
A new paper introduces CELEUS, a certifiable framework for efficient LLM evaluation that leverages e-processes to build anytime-valid confidence intervals. It combines uncertainty-guided sampling and surrogate-assisted approximations to statistically guarantee that evaluation scores reflect true performance. CELEUS reaches target precision using 54–62% fewer evaluated samples than baselines while preserving anytime-valid coverage.
Key Points
- Combines uncertainty-guided sampling and surrogate-assisted approximations to minimize evaluated samples while preserving anytime-valid confidence intervals
- Achieves 54-62% reduction in samples needed to reach target precision compared to baselines in experiments
- Provides theoretical guarantees including near-parametric convergence rates and oracle-optimal sampling rule
Why It Matters
Enables certifiable and efficient LLM testing, saving compute while ensuring reliable, statistically valid performance guarantees.