How effective are VLMs in assisting humans in inferring the quality of mental models from Multimodal short answers?

A team of researchers from IIT Bombay and other institutions has published a paper introducing MMGrader, a novel approach designed to tackle the complex challenge of assessing the quality of students' mental models in STEM education. Mental models—internal representations of how concepts connect—are critical indicators of deep understanding but are notoriously difficult to infer from student responses. MMGrader leverages Vision-Language Models (VLMs) to analyze multimodal answers (potentially combining text and diagrams) and uses concept graphs as an analytical framework to map and evaluate the structure of a student's conceptual knowledge. The goal is to move beyond simple grading to understanding how students integrate and apply ideas.

The researchers evaluated nine openly available VLMs on this reasoning-intensive task. Their findings reveal a significant performance gap: the best-performing models achieved only approximately 40% accuracy in inferring mental model quality, with a prediction error of 1.1 units on their scoring scale. While the AI's scoring distribution showed some alignment with human patterns, it falls far short of human-level performance. This underscores the current limitations of even state-of-the-art models in tasks requiring deep, contextual reasoning about conceptual understanding. However, the research lays a foundation for future development; with improved accuracy, such systems could become powerful assistants for educators, enabling efficient, whole-classroom assessment and data-driven pedagogical adjustments to target collective knowledge gaps.