The Number AI Tutors Use to Decide You've Mastered Math Is Broken

AI tutoring apps — the kind that quiz students on fractions or vocabulary and decide when they're ready for the next topic — all rely on one hidden setting: a mastery threshold. That's the score, usually somewhere between 50% and 99%, at which the software says "you've got this, move on." Teachers and schools rarely see it, but it quietly shapes how much practice every student gets.

A new study looked at six different AI tutoring models (programs that guess what a student knows based on their answers) and four public sets of student data. The researchers tested twelve thresholds from 0.50 to 0.99 and measured what happened afterward: did students actually do well on the next topic? How much extra practice did they get? Who got held back?

The key finding is that the same number does not mean the same thing across models. One older-style model barely changed its decisions as the bar moved. The newer AI models became far pickier — holding many more students back. That's partly because they're measuring different things: one estimates how well a student has learned, the other estimates the odds they'll get the next question right. A 0.80 in one system isn't a 0.80 in another.

Even more troubling: raising the bar didn't reliably shrink achievement gaps. Instead, it often blocked the students who needed help most. Students with stronger past performance advanced up to 3.26 times as often as those with weaker records. In half the tested classroom scenarios, models disagreed on the right cutoff by more than 0.10 — a big gap. The lesson: if a school switches tutoring software, the mastery setting should be re-tuned, not copied over.