To Layer or Not to Layer? Evaluating the Effects and Mechanisms of LLM-Generated Feedback on learning performance
LLM-generated layered feedback increased student effort but led to significantly worse learning performance.
A new study from researchers including Jie Cao and Kenneth R. Koedinger tackles a key question in AI-assisted education: does the structure of automated feedback matter? The team investigated whether "layered" feedback from Large Language Models (LLMs)—which sequences encouragement and hints before revealing an answer—is more effective than direct, "non-layered" feedback. In a controlled experiment with 199 participants, they measured the impact on learning performance, engagement, and user perceptions.
The results present a nuanced and somewhat counterintuitive finding. While layered feedback successfully elicited higher behavioral engagement and was perceived as more encouraging and supportive of learner independence, it also induced greater mental effort. Crucially, mediation analysis showed these positive affective perceptions were offset by a negative behavioral pathway: students receiving layered feedback required more attempts (≥3 submissions) on tasks. The net result was that layered feedback led to significantly worse learning outcomes compared to the simpler, non-layered approach. This study, available on arXiv, provides a critical evidence-based warning for the design of educational AI, highlighting that engagement and positive feelings do not automatically translate to better learning.
- Layered LLM feedback (scaffolded hints) was tested against direct feedback with 199 participants.
- It increased engagement and was seen as more encouraging but required greater mental effort.
- Despite positive perceptions, it resulted in significantly poorer learning outcomes due to increased task attempts.
Why It Matters
For EdTech and AI tool builders, it shows that designing for user engagement can inadvertently undermine core learning objectives.