SolidCoder: Bridging the Mental-Reality Gap in LLM Code Generation through Concrete Execution

Researchers Woojin Lee and Jin-Xia Huang have introduced SolidCoder, a novel framework designed to solve a fundamental flaw in how large language models (LLMs) like GPT-4o generate code. They identified the 'Mental-Reality Gap,' where models internally hallucinate execution traces and confidently validate buggy code. This gap has two dimensions: the Specification Gap (overlooking edge cases) and the Verification Gap (hallucinating correct behavior). SolidCoder's S.O.L.I.D. architecture tackles this with a simple principle: don't imagine, execute.

The framework works by forcing the LLM to consider edge cases during the planning phase before writing any code. Then, instead of relying on the model's potentially flawed internal simulation, it runs the generated code in a sandboxed environment against property-based oracles for verification. This concrete execution catches errors that mental simulation misses. When tested with GPT-4o, SolidCoder achieved a 95.7% pass@1 score on HumanEval, a 77.0% score on CodeContests, and a 26.7% score on APPS, representing significant improvements over previous methods.

An ablation study revealed that the edge-case awareness component provided the largest individual performance gain. However, the execution grounding component was essential for catching a categorically different class of errors that specification improvements alone could not address. The researchers validated that these gains generalize to other models, including those fine-tuned with reinforcement learning, confirming that addressing both gap dimensions is crucial for robust code synthesis. The team has released their code and framework to facilitate further research in this area.