LLM Chatbot Penny Reveals Two Learner Loops in 21K Interactions

A new study published on arXiv analyzes how Japanese EFL learners interact with 'Penny,' an LLM-powered writing chatbot, using Transition Network Analysis (TNA) to model temporal dynamics. Over 4,500 writing sessions and 21,000 chatbot interactions were examined, revealing two primary behavioral loops. The 'Revision Loop' involves learners receiving feedback and immediately correcting errors, a pattern more common among low-proficiency users. The 'Chat Loop' shows learners engaging in sustained dialogue with the chatbot after feedback, negotiating meaning and asking follow-up questions—a behavior more prevalent among high-proficiency users.

Crucially, the study demonstrates that AI-scaffolded writing is a non-linear, dialogic process heavily shaped by learner proficiency. High-proficiency students treat the chatbot as a conversational partner, using it to explore language nuances, while lower-proficiency students treat it as a corrective tool, cycling through repetitive error-fix patterns. The authors argue that current chatbot designs often treat all learners uniformly, missing opportunities to scaffold deeper cognitive engagement. They recommend differentiated interfaces—for example, prompting low-proficiency users to ask exploratory questions or providing high-proficiency users with more open-ended challenges. This research has direct implications for AI tutoring systems in language education and beyond.