AI Tests Are Maxing Out—Here's Why That's a Problem

Imagine a video game that's supposed to be hard, but the best players beat it perfectly every time. That's what's happening with AI tests. Benchmarks—standardized tests for AI—are becoming saturated, meaning top models score near 100%. For example, one popular coding test was solved 99.4% of the time by a leading AI right when it was released. When tests are that easy, they stop being useful for comparing models or spotting dangers.

This isn't just a nerdy inconvenience. In high-stakes areas like cybersecurity and bioweapons, we need to know exactly how capable AI is. Safety rules and regulations are built on thresholds: if a model can do X, it needs extra safeguards. But if tests are saturated, we might underestimate what a model can do. OpenAI, for instance, labeled its GPT-5 model as having 'High' biological capability even without definitive proof, just to be safe. That kind of uncertainty is risky.

Building harder tests is one solution, but it's slow and expensive. One recent update to a coding benchmark took 23 engineers and 1,900 hours, only to be nearly maxed out on release. With new AI models coming out every few months, constantly replacing tests isn't sustainable. Researchers suggest a smarter approach: analyze how models solve problems, not just whether they get the right answer. This could give us a richer picture of capability even when tests are easy.

However, this method has its own limits. It struggles when AI goes through 'paradigm shifts'—like when reasoning models suddenly started behaving differently. The key takeaway: we need evaluation methods that are robust to these shifts and can scale as AI improves. Otherwise, we'll be flying blind just when we need clear vision the most.