LLM code edits fail at deleting code: study finds 52% recall

A new study titled *To Add Is Machine, To Delete Is Human* reveals that frontier large language models systematically avoid deleting code, even when the deletions are explicitly required by the task. Conducted by researchers from Queen's University, University of Waterloo, and others, the paper analyzes five leading models on the SWE-bench Verified leaderboard and finds that deletion recall tops out at 71.7%—even for tasks all models solve correctly. More critically, models retrieve the correct file for over 92% of required deletions but fail to cut the exact line in under 52% of cases. Instead, they often wrap the target code in a guard or fallback mechanism, a pattern dubbed *Guard-and-Go*, which allows tests to pass despite incomplete deletion.

The team then retrofitted 34 SWE-bench Verified tasks with tests that explicitly fail if targeted code remains. Under this stricter evaluation, four frontier models—spanning both closed and open-weight systems—saw their success rates plummet from 63.2% to 41.9%. To isolate deletion behavior, the researchers introduced CanItDelete, a benchmark of 200 real-world commit tasks where the entire edit is a deletion. Even without the confounding factor of code addition, the best-performing model still failed one in five tasks, and smaller open models performed as poorly as 18.0%. Prompt engineering showed limited improvement until the exact lines to delete were specified, but even then success only reached 80.5% because models either over-deleted or added extraneous code. A pilot post-training intervention, however, reduced deletion avoidance and improved broader code-editing performance, suggesting the behavior is undertrained rather than intractable.