NeXUI benchmark stumps Gemini-3.5-Flash with 44% success rate

Modern web interfaces are increasingly difficult for blind and visually impaired users to navigate with screen readers, especially when pages update dynamically or hide structure behind visual layouts. While recent UI agents can take actions on these interfaces, they rarely explain what they're doing or allow users to stay in control. Most existing benchmarks measure only task completion, ignoring whether agents keep users informed. To address this, researcher Santosh Patapati introduces NeXUI, a benchmark that pairs realistic user goals with instrumented interface states, requiring agents to reason from both visual and structural information while producing clear, grounded explanations for each step.

In experiments, NeXUI exposed major gaps in current AI systems. Even state-of-the-art foundation models struggled: Google's Gemini-3.5-Flash achieved only a 44% success rate with poor explanation scores. The benchmark evaluates safety, efficiency, and task success while verifying that explanations are grounded in the actual interface state—ensuring agents don't hallucinate their reasoning. NeXUI is designed as a foundation for future research and development in assistive agents, providing a clearer way to study collaboration, user oversight, and accessibility in modern computing environments. The work was submitted to arXiv on July 1, 2026, and falls under Human-Computer Interaction and Artificial Intelligence.