Developing the PsyCogMetrics AI Lab to Evaluate Large Language Models and Advance Cognitive Science -- A Three-Cycle Action Design Science Study
A new cloud platform applies psychometric tests and cognitive science to evaluate AI models beyond simple benchmarks.
A team of researchers has introduced the PsyCogMetrics AI Lab, a novel cloud platform designed to bring scientific rigor to the evaluation of Large Language Models (LLMs) like GPT-4 and Claude. Developed through a three-cycle Action Design Science study, the platform addresses key limitations in current AI benchmarking, which often relies on narrow, task-specific metrics. Instead, PsyCogMetrics operationalizes established psychological and cognitive science theories—including Popperian falsifiability, Classical Test Theory, and Cognitive Load Theory—to create a more holistic and theoretically grounded assessment framework.
The platform's development followed a structured methodology: a Relevance Cycle identified stakeholder needs and current evaluation gaps, a Rigor Cycle drew on kernel theories to set design objectives, and a Design Cycle implemented these through iterative Build-Intervene-Evaluate loops. The resulting IT artifact, detailed in a paper accepted to HICSS 2026, provides an integrated environment where researchers can subject LLMs to tests that probe reasoning, learning, and cognitive load in ways analogous to human assessment. This bridges the gap between AI performance metrics and fundamental cognitive science, offering a new tool for interdisciplinary research.
By moving beyond simple accuracy scores, PsyCogMetrics allows for deeper investigation into how LLMs process information, where they fail, and how their 'cognitive' profiles compare to human intelligence. This standardized, cloud-based approach promises to make advanced LLM evaluation more accessible and reproducible for the broader research community at the intersection of AI, psychology, and behavioral science.
- The PsyCogMetrics AI Lab is a cloud-based platform that applies psychometric and cognitive science methods to evaluate LLMs.
- Its design is based on a three-cycle Action Design Science study incorporating theories like Classical Test Theory and Cognitive Load Theory.
- The platform aims to provide a more rigorous, standardized, and theoretically grounded alternative to current LLM benchmarking practices.
Why It Matters
Provides a scientific framework to move beyond simplistic AI benchmarks, enabling deeper understanding of model capabilities and failures.