Recommending Usability Improvements with Multimodal Large Language Models

A team led by Sebastian Lubos from TU Graz has developed a novel method that leverages multimodal large language models (MLLMs) to automate usability evaluation of application user interfaces. Their approach, accepted at FSE 2026, inputs screen recordings of user interactions along with limited application context into an MLLM. The model then identifies usability issues based on Nielsen's ten usability heuristics, provides explanations for each issue, and generates concrete improvement recommendations. To reduce manual prioritization effort, the system ranks all suggestions by severity, ensuring developers focus on the most impactful fixes first.

In a user study involving software engineers as evaluators, the team assessed the quality and practical usefulness of the highest-ranked recommendations. Results demonstrated that the MLLM-driven approach can produce low-effort, actionable usability insights that closely mirror those from expert evaluators. This makes it a promising complement to traditional methods, especially for small teams and organizations with limited access to usability experts. The authors envision future integration into development tools, enabling continuous, automated usability evaluation within standard software engineering workflows.