CodeBERT authorship detection fails on real student code, new study finds

A new arXiv paper from Serhii Yemets and Marek Horváth tests whether source code authorship attribution—identifying who wrote a program from style—actually works in educational settings. They fine-tuned CodeBERT, Microsoft's popular pre-trained transformer for code, on three data sources: Google Code Jam (GCJ) submissions from Kaggle, a curated GCJ archive, and coursework from a technical university. On competition data, the model produced impressive results: 92.6% Top-1 accuracy for 10 authors and 70.7% Top-1 (88.2% Top-10) for 1,000 authors. But the same pipeline collapsed on real student assignments—0.2% Top-1 on a closed assignment dataset of 690 authors and 0.06% Top-1 on open-ended assignments with 812 authors. Those numbers sit at or below random chance, meaning the model learns competition-specific patterns that do not transfer.

The performance gap persists across multiple model families, not just CodeBERT, according to a companion benchmark. The authors argue that dataset and task properties—like varying assignment constraints, student skill levels, and code similarity within cohorts—explain why competition-trained models fail in classrooms. They emphasize that GCJ-based benchmarks overestimate the practical usefulness of authorship attribution for plagiarism detection or academic integrity tools. The takeaway: AI-powered code attribution must be validated on target coursework contexts before deployment. For educators and edtech vendors, this is a critical warning that benchmark success on competitive programming does not guarantee real-world reliability, especially when stakes involve student discipline decisions.