AI That Controls Your PC Still Fails Blind Users Half the Time

Imagine an AI that can see your computer screen, move the mouse, and click buttons for you — like a remote assistant. That's the promise of "computer-use agents." Now imagine being blind and relying on such a system to do everyday tasks like filling out a form or checking email. A new study tested whether these AI helpers are actually reliable enough for blind users. The answer? Not yet.

Researchers built a prototype called OLLA that works with screen readers (software that reads text aloud for blind users). Eight blind participants used it for three weeks across 12 common desktop applications. The team logged 1,258 commands and tested five different AI models. The best performer, GPT-5, completed tasks successfully only 52.5% of the time. That means nearly half of all commands failed — the AI clicked the wrong spot, misunderstood a menu, or lost track of what it was supposed to do.

Looking closer at the failures, the researchers found four repeating problems: the AI couldn't always "ground" its actions to what was actually on screen; it sometimes planned steps in the wrong order; it ignored constraints (like "don't delete this file"); and it didn't know when to stop. Beyond the technical issues, blind participants said they wanted more than just automation — they wanted control, feedback, and trust. One participant described it as "having a very distracted intern" helping out.

The study is a reality check. AI agents that control computers are exciting, but they're not ready to give blind users true independence. On the bright side, the detailed failure analysis gives developers a clear roadmap to improve. If AI can get past even 90% success while respecting user instructions, it could transform digital access for millions of people with visual impairments.