Code LLM alignment: pretrained start yields bigger gains, finetuned start risks regression

Large Language Model alignment typically relies on preference data to steer outputs toward quality standards, but little is known about its effectiveness for code generation tasks—especially how it handles both functional correctness and non-functional qualities like readability and maintainability. This paper by Uddin and Sivapiran (FSE 2026) tackles that gap by systematically comparing reward-free alignment techniques—Direct Preference Optimization (DPO) and Best-of-N Best-of-N (BoNBoN)—on five leading LLMs. For each training instance, they created preference pairs using the SelfCodeAlign pipeline, and ran alignment on both the pretrained and finetuned (instruction-tuned) version of each model.

The results reveal a clear trade-off. Starting from a pretrained base yields larger percentage improvements in the aligned variant compared to its pretrained baseline, but the pretrained variant itself is significantly less accurate than its finetuned counterpart. Conversely, aligning from a finetuned model produces much smaller performance gains and, in several cases, actually degrades code quality—especially on non-functional metrics like readability and maintainability as measured by the CODAL benchmark. The study used four functional benchmarks (HumanEval+, MBPP+, EvalPerf, EvoEval) to assess correctness. The takeaway: choosing the right starting point for alignment is critical—pretrained offers bigger leaps but starts lower, while finetuned risks regression.