REFORGE benchmark exposes LLM reverse engineering flaws: 87% to 66% accuracy drop

Large language models are increasingly being deployed in live reverse engineering and offensive security operations, yet existing benchmarks fail to account for a critical variable: compiler optimization. In their new paper, Koller and Schmidt present REFORGE, a provenance-tracked pipeline that constructs function-level ground truth from C source through compilation, DWARF extraction, alignment, and decompilation. The key innovation is an eight-gate confidence funnel that operationalizes alignment uncertainty into three tiers of reliability. On a controlled micro-benchmark, high-confidence yield dropped from 87.2% to 65.9% as compiler optimization increased, demonstrating that current evaluation practices systematically overstate model capability.

REFORGE also reveals that unpaired comparisons—comparing LLM predictions against ground truth from different optimization levels—introduce survivorship bias, making optimization-induced performance decay appear worse than it is. In a proof-of-concept evaluation of seven contemporary LLMs on function naming, the authors show that alignment uncertainty, not model ability, is the principal obstacle to fair benchmarking. The paper concludes that the reverse engineering community must adopt uncertainty-aware benchmarking practices to avoid overestimating LLM capabilities in critical security applications. Accepted at the 23rd International Conference on Applied Computing 2026, REFORGE provides a rigorous foundation for future evaluations.