The Way We Judge Captions Is Broken, Deaf Viewers Say

If you're deaf or hard of hearing, captions are how you watch TV. So a team of researchers did something simple but rarely done: they asked actual viewers how good the captions were, instead of trusting software scores. They surveyed 216 deaf and hard-of-hearing people across the U.S., who rated 70 clips recorded from live TV — a total of 4,832 individual judgments. Each clip was shown in four versions: regular TV captions with their typical 7-to-12-second delay, TV captions lined up with the audio, AI-generated captions lined up with the audio, and AI captions running about two seconds behind.

The first surprise: viewers rated AI captions and human TV captions about the same. That's a big deal, because AI captions are far cheaper and faster to produce, and broadcasters have long argued they're not good enough. The second surprise is the problem. The industry's standard quality scores — including Word Error Rate, which basically counts how many words the system got wrong — tracked viewer satisfaction reasonably well for TV captions. For AI captions, they barely tracked it at all. In other words, the same score can mean very different things depending on which technology made the captions. The tests aren't neutral referees.

Timing turned out to matter enormously. The 7-to-12-second delays typical of live TV captions significantly damaged the viewing experience, while a two-second delay on AI captions was far more tolerable. That suggests broadcasters could improve accessibility just by closing the lag, even without perfect word accuracy.

The takeaway: as AI captioning spreads across streaming, video calls, classrooms and live events, the scores used to approve or reject it may certify captions viewers find unusable — or reject ones they'd happily watch. The researchers argue quality standards should be validated against real deaf and hard-of-hearing viewers, not just error counts.