DeepSWE benchmark exposes coding agents' reliance on memorized fixes

Current coding agent benchmarks like SWE-bench suffer from two major flaws: tasks are mined from publicly merged GitHub fixes that may appear in model training data, so high scores could reflect memorization rather than genuine problem-solving; and grading relies on tests written to validate a specific fix, which can fail correct alternative implementations or pass incomplete ones. These issues make it difficult to assess whether agents truly understand software engineering or are just recalling patterns.

To address this, researchers introduced DeepSWE, a benchmark of 113 original, long-horizon engineering tasks written from scratch across 91 active open-source repositories in five languages. The tasks are never contributed back upstream, ensuring reference solutions stay out of public training data. Each task is graded by a hand-written verifier that checks for the requested functionality, accepting any correct implementation. In validation, an independent LLM judge disagreed with DeepSWE's verifier only 1.4% of the time versus 32.4% for SWE-Bench Pro's inherited tests. Despite prompts being about half the length, tasks require 5.5x more code changes, and the benchmark spreads frontier agents across a wider performance band than existing leaderboards. The team has released the benchmark, verifiers, and full evaluation trajectories for public use.