Developing a Multi-Agent System to Generate Next Generation Science Assessments with Evidence-Centered Design

A team of researchers has published a paper demonstrating a novel Multi-Agent System (MAS) that automates the generation of complex, performance-based science assessments. The system integrates the rigorous Evidence-Centered Design (ECD) framework—a methodology that ensures assessment validity by modeling the learner, evidence, and tasks—with an ensemble of multiple large language models (LLMs) acting as specialized agents. This approach tackles a major bottleneck in modern education: the high cost and labor-intensive process of creating NGSS-aligned assessments, which require diverse expertise in content, pedagogy, and psychometrics.

The technical core of the system involves orchestrating LLMs with varying expertise to automate the multi-stage workflow traditionally performed by human teams. In a comparative study, the researchers examined the quality of AI-generated items against human-developed ones. Results showed the AI system produced items with overall comparable quality to human experts in terms of alignment with NGSS's three-dimensional standards (integrating core ideas, crosscutting concepts, and practices) and cognitive demand. A key finding was a divergent pattern: AI-generated items demonstrated a distinct, measurable strength in designing more inclusive content, likely due to training on vast, diverse datasets.

However, the AI system also exhibited limitations compared to humans, particularly in clarity, conciseness, and the ability to design effective multimodal components (like diagrams). Interestingly, both AI and human items shared weaknesses in 'evidence collectability'—how well student responses can be interpreted—and in aligning with student interests. The research concludes that integrating ECD into MAS offers a powerful path toward scalable, standards-aligned assessment design, but human expertise remains essential for refinement, particularly in areas requiring nuanced judgment and multimodal creativity. This represents a significant step toward AI-augmented, rather than fully automated, educational content creation.