From Domain Understanding to Design Readiness: a playbook for GenAI-supported learning in Software Engineering
A custom GPT-3.5 tutor helped 29 master's students achieve 98.9% accuracy on complex software design tasks.
A new academic study by Rafal Wlodarski demonstrates the potent role of generative AI as a specialized tutor in advanced software engineering education. In a two-week milestone within a master's course, 29 students used a customized ChatGPT (GPT-3.5) instance, which was grounded in a curated knowledge base of course materials. The AI's task was to tutor students in two complex, supporting knowledge areas: cryptocurrency-finance basics and Domain-Driven Design (DDD) modeling methods. The goal was to accelerate the rapid upskilling often required in such courses.
Researchers logged all 174 student-AI interactions and evaluated a 34.5% random sample (60 pairs) using a five-dimension rubric. The results were striking: the AI tutor delivered responses with 98.9% accuracy (with zero factual errors and only two minor inaccuracies) and 92.2% relevance. Pedagogical value was rated high at 89.4%, and cognitive load was generally appropriate (82.78%). However, the AI's supportiveness—its tone and follow-up structure—was a weak point, scoring only 37.78%. Despite this, students reported large gains in self-efficacy for both GenAI-assisted domain learning and DDD application after the intervention.
From this detailed analysis, Wlodarski distills a practical 'playbook' of 17 concrete teaching practices for educators. These span two key areas: prompt/configuration design (like setting expected answer granularity and constraining verbosity) and course/workflow design (such as curating guardrail examples and adding credit via simple quality rubrics). The study concludes that within this controlled context, GenAI-supported learning effectively complements traditional instruction for domain understanding and modeling tasks, while highlighting specific areas like conversational supportiveness for future improvement.
- Custom GPT-3.5 tutor achieved 98.9% accuracy and 92.2% relevance in teaching cryptocurrency finance and DDD to 29 students.
- Student self-efficacy saw 'large' gains, though AI supportiveness (tone/follow-up) was low at 37.78%.
- The study produces a playbook of 17 actionable teaching practices for integrating GenAI into technical curricula.
Why It Matters
Provides a validated blueprint for using AI tutors to efficiently train developers in complex, emerging domains like crypto and advanced software design.