Study: Sketching and critique reveal visualization skills multiple-choice tests miss

A new arXiv paper from researchers at University of Utah, Northwestern, and other institutions challenges the standard approach to measuring visualization literacy. Led by Zach Cutler, the team developed two web-based assessments that ask participants to either verbally critique a flawed visualization or sketch a chart from a given dataset and prompt. They then compared results against established multiple-choice tests CALVI and Mini-VLAT, administering all assessments to three groups with different experience levels: crowdworkers, students who had completed a visualization course, and visualization researchers.

The findings show a clear limitation of multiple-choice tests: they suffer from ceiling effects, meaning moderately skilled people score near the maximum, leaving little room to distinguish true experts. In contrast, the critique and sketching assessments captured distinct, higher-order skills such as applying external knowledge, identifying design flaws, and constructing effective charts. These qualitative methods differentiated between experienced individuals far better than multiple-choice alternatives. The authors acknowledge that administering and grading such assessments is more labor-intensive, but argue they are a valuable complement—especially when the goal is identifying top-tier visualization talent rather than just catching beginners.