Desc2Fix: New AI Framework Measures Semantic Drift in Bug Resolution
LLMs reveal a 1-2 point optimism bias when judging bug report-patch alignment across 2,857 triplets.
A new preprint from a team of 10 researchers introduces Desc2Fix, a framework that measures how behavioral signals from bug reports propagate through tests and into patches. By analyzing 2,857 report-test-patch triplets from Defects4J and SWT-Bench, the authors used GPT-4o and DeepSeek-Chat to extract structured behavioral anchors (reproduction steps, API cues, expected vs. actual behavior) and evaluate alignment with deterministic metrics (ROUGE, SBERT, CodeBERT, OpenAI embeddings) plus LLM-based judgments.
The study found that LLMs extract structured signals with up to 90% completeness and show strong cross-model consistency, but alignment is highly representation-sensitive: lexical similarity alone is insufficient. Across more than 182,000 alignment ratings, both models displayed systematic optimism relative to human judges (1-2 points on 5-point scales) and modest rank agreement. The authors argue that structured anchors combined with embedding-based proxies provide reproducible signals for ranking and filtering tests and candidate patches, transforming alignment into a controllable engineering signal for more reliable test generation, semantics-aware fault localization, and principled patch ranking.
- Desc2Fix framework uses GPT-4o and DeepSeek-Chat to extract structured behavioral signals from 2,857 bug report-test-patch triplets.
- LLMs show 90% completeness in extracting behavioral anchors but exhibit systematic optimism (1-2 points) vs. human raters across 182,000 alignment scores.
- Full diffs provide the most stable basis for report-patch correspondence, outperforming lexical similarity or structured summaries.
Why It Matters
Desc2Fix turns bug resolution into a quantifiable engineering signal, enabling automated test generation and smarter patch ranking.