SecureBio researcher's IRT study shows diminishing returns in LLM benchmark questions

A LessWrong post by bpomo (supported by SecureBio) introduces a method using Item Response Theory (IRT) to quantify diminishing returns when adding questions to LLM benchmarks. The core insight: if questions are correlated (e.g., many about the same subtopic), each new question provides less incremental information about a model's true ability (theta). The post uses Omni-MATH's 1000-item dataset to fit an IRT model and estimate latent capability scores for models like Claude 3.7 Sonnet. The Fisher information from the maximum likelihood estimator gives a bound on the variance of theta, showing that precision grows linearly only if questions are statistically independent. In practice, real benchmarks violate this, so the marginal information gain shrinks as the benchmark grows.

The author illustrates the concept with a zoology analogy: asking 100 bird questions gives a less precise ability estimate than 100 random animal questions. The appendix (partially assisted by Claude) provides the formal theory. While the method is limited by model fit error and the assumption of a one-dimensional ability construct, it offers a practical tool for benchmark designers. The post suggests potential applications: stopping benchmark expansion once new items add negligible information, thereby saving compute and improving evaluation efficiency. For AI safety and evaluations, this directly addresses the question of when a benchmark is "good enough."