arXiv study: 4-bit quantization masks 2.5x failure spike in LLM agents

Researchers from Korea (Jang, Yang, Lim, and Park) ran a rigorous study on whether 4-bit post-training quantization truly is 'nearly lossless' for LLM agents. Using tau²-bench across eight cells — two open-weight model families in dense and MoE variants, two domains (telecom and retail) — they ran 456 episodes per cell at 16-, 8-, and 4-bit weights. On the standard task-reward metric, quantization looked free: no cell showed a score change that survived multiple-comparison correction, and equivalence testing bounded the largest damage within ±7.5 points.

The process tells a different story. Quantization amplified the failure the model already exhibits at full precision — tool-name hallucination in telecom, with the same trend in retail entity errors — by up to 2.5x in volume (+17.6 points per task), while creating essentially no new failure types (rank correlation ≥ 0.94, only 0.18% novel events). The benchmark's ten-error budget absorbed these extra failures, keeping scores flat. When the budget was shrunk to two errors, a 17-point score gap reappeared, exactly in the one cell where quantization had added error volume. A targeted error-repair prompt removed the damage precisely where it lived, confirming the mechanism. The authors suggest reporting per-channel error rates and shrinking-budget success alongside task reward, since both are already available from standard logs.