AQLM beats QuIP#: Study reveals best quantization for code generation

As local inference frameworks like Ollama gain traction, developers increasingly run large code models on laptops and other resource-constrained hardware. Post-training quantization is essential to reduce memory footprint, but its impact on generated code quality has remained unclear. This paper by Afrin et al. empirically compares six quantization techniques (GPTQ, AWQ, QuIP#, AQLM, BitsAndBytes, GGUF) on two representative code model families (Qwen2.5-Coder and CodeLlama) using multilingual benchmarks (McEval and CoderEval) for Python and Java. The study goes beyond functional correctness (pass@1) to evaluate maintainability, reliability, security, and structural complexity. It also introduces a novel analysis of robustness under varying prompt complexity, characterized by Shannon entropy and token length.

The results show meaningful differences among quantization methods. AQLM stands out by consistently matching or even exceeding the full-precision baseline across all metrics, making it a top choice for deployment. Conversely, QuIP# exhibits the largest correctness degradation, particularly on complex prompts, suggesting its use should be avoided in production code generation scenarios. Security attributes remain stable across models, benchmarks, and programming languages, indicating that quantization does not introduce new vulnerabilities. Robustness to prompt complexity varies significantly, with some techniques handling high-entropy prompts better than others. These findings provide actionable guidance for selecting quantization strategies when deploying large code models on resource-constrained hardware, emphasizing the need to evaluate beyond functional correctness.