SemLoc: Structured Grounding of Free-Form LLM Reasoning for Fault Localization

A team from UC Riverside and Meta has developed SemLoc, a novel AI framework that fundamentally changes how developers find elusive 'semantic bugs'—errors where code executes correctly but produces the wrong result. Traditional tools that rely on code coverage or execution traces fail here because passing and failing runs follow identical paths. While LLMs like GPT-4 can reason about intent, their outputs are stochastic and unverifiable. SemLoc solves this by converting an LLM's free-form reasoning about a program's purpose into a closed, structured intermediate representation. This representation binds each inferred semantic property (e.g., 'this variable should never be negative') to a specific, typed anchor in the code, making the reasoning traceable and checkable.

During execution, SemLoc instruments the program to build a 'semantic violation spectrum'—a matrix showing which inferred constraints fail on which tests. From this, it calculates suspiciousness scores for code regions, similar to how coverage-based fault localization works. A final counterfactual verification step prunes over-approximate constraints to isolate the primary causal violation. Evaluated on a new benchmark of 250 Python programs with single semantic faults (SemFault-250), SemLoc outperformed five existing coverage-, reduction-, and LLM-based techniques. It achieved a Top-1 accuracy of 42.8% (meaning the buggy line was its top suggestion nearly 43% of the time) and a Top-3 accuracy of 68%. Crucially, it reduced the amount of code a developer must manually inspect by 92.4%, down to just 7.6% of executable lines. The verification step provided an additional 12% accuracy gain, proving its value in filtering noise.