Are LLMs Reliable Code Reviewers? Systematic Overcorrection in Requirement Conformance Judgement

A new research paper from Haolin Jin and Huaming Chen, 'Are LLMs Reliable Code Reviewers? Systematic Overcorrection in Requirement Conformance Judgement,' exposes a critical flaw in how AI models like GPT-4 and Claude 3.5 perform code review tasks. The study demonstrates that these widely adopted LLMs systematically fail to correctly judge whether code implementations satisfy natural language requirements, frequently misclassifying perfectly correct code as defective or non-compliant. This failure undermines a core promise of AI coding assistants—reliable verification of code against specifications—and reveals a previously under-explored limitation in automated review pipelines.

The researchers found that more detailed prompting strategies, particularly those requiring the model to provide explanations and proposed corrections, actually increase misjudgment rates by up to 40%, highlighting a dangerous reliability gap. To address this, the team developed a 'Fix-guided Verification Filter' that treats the AI's proposed code fix as executable counterfactual evidence, then validates both the original and revised implementations using benchmark tests and specification-constrained augmented tests. This practical safeguard provides developers with a method to integrate LLM-based reviewers while maintaining accuracy, offering crucial guidance for building more robust automated development workflows that don't blindly trust AI judgment.