Claude Sonnet 4 best at simulating student Java coding errors
LLMs generated errors indistinguishable from real students in blind tests.
A new arXiv paper from researchers at UC Irvine explores whether large language models (LLMs) can simulate realistic programming errors made by students — a critical need for building teachable agents, intelligent tutoring systems, and large-scale learning analytics. Using the CodeWorkout dataset, which contains over 74,000 unique Java submissions across 37 problems, the team evaluated five LLMs under three prompting strategies: Input-Output, Chain-of-Thought, and iterative Self-Refine. The key dimensions were diversity (range of distinct error patterns) and alignment (similarity to authentic student mistakes).
Quantitative results showed that while all models generated diverse errors, Claude Sonnet 4 emerged as the most balanced performer in aligning with human submissions. A blinded expert annotation study (N=401) confirmed that synthetic and authentic errors were functionally indistinguishable. Notably, higher-struggling-level problems triggered more diverse errors but those errors looked less student-like. The findings highlight trade-offs in using LLMs as proxies for human learners and offer design guidelines for integrating synthetic errors into educational AI tools.
- Evaluated 5 LLMs on 74,000+ Java submissions across 37 CodeWorkout problems.
- Claude Sonnet 4 achieved best balance of error diversity and alignment with real student errors.
- Blinded expert study (N=401) found synthetic errors functionally indistinguishable from authentic ones.
Why It Matters
Synthetic student errors could accelerate AI tutoring system development without waiting for real classroom data.