DECODE dataset of 53.6K real code edits improves LLM training

A new dataset called DECODE (Developer Edits of Code Dataset) captures 53,609 real-world edits made by over 1,000 developers when modifying AI-generated code in Python, TypeScript, and JavaScript. Unlike Git commits that only show final working code, DECODE records fine-grained editing actions inside the IDE, offering a more realistic view of how developers fix or reject AI suggestions. The data reveals two key insights: most edits occur within the first 15 minutes after accepting an AI completion, and in 31% of edit trajectories, developers end up completely removing the AI-generated code. This suggests that current AI programming assistants often produce code that requires significant manual correction or outright rejection.

The paper also benchmarks LLMs on code edit prediction tasks. A fine-tuned 3B-parameter open-source model trained on DECODE significantly outperforms frontier models like GPT-4 and Claude 3.5 on this task, showing that realistic, high-quality editing data is more valuable than publicly available commit data for teaching models how to anticipate developer modifications. The authors argue that future AI programming assistants should incorporate developer-centric machine learning approaches—analyzing how humans actually edit code rather than just optimizing for final output. This work has implications for improving AI code generation tools, making them more adaptive to real-world development workflows.