Claude Solves Only 1 in 3 College-Level Probability Proofs

Most AI math tests are small collections of competition questions — the kind bright teenagers solve in the International Math Olympiad. Those are impressive, but they don't look much like the math adults actually get paid to do. So a team of researchers built StochBench: 450 graduate-level probability problems, each written in Lean 4, a proof-checking language where a computer verifies every single logical step. The topics include Markov chains, random walks, queues, Brownian motion and stochastic calculus — the toolkit of real-world risk and randomness.

Why should you care? This is the math behind your daily life. It's how insurers price a policy, how hospitals decide how many beds to staff, how delivery companies route trucks, and how analysts model stock prices. If an AI can reliably work through these proofs, it becomes a genuinely useful assistant for actuaries, engineers and quantitative analysts — catching mistakes in reasoning that would otherwise take a human days to trace by hand. Because Lean checks the logic mechanically, there's no arguing about whether an answer is right.

The catch is the score. The best system tested — an AI agent built on Claude Opus 4.8 — solved 157 of 450 problems, a 34.9% success rate, given 15 minutes per problem. In other words, it fails roughly two out of three times on this material. That's a stress test showing where the limits are, not a product launch. It's also worth remembering that proving a mathematical statement in Lean only means the logic is airtight; it doesn't mean the model's assumptions match the real world.

So what's the takeaway? AI has become a solid junior assistant for hard quantitative work — good for checking, spotting gaps and grinding through routine steps — but it is not yet an authority on advanced probability. The benchmark is public, which means we'll be able to see exactly how quickly that 35% climbs, and which parts of applied math AI masters next.