Small LLMs beat giants: Qwen Coder 3B hits 75.67% on new local benchmark
Fine-tuned sub-3B models gain up to 27 points, rivaling far larger AIs on structured tasks.
A new arXiv paper from Daniel Cersosimo (submitted May 5, 2026) challenges the assumption that frontier-size models are necessary for practical AI deployment. The study evaluates nine open-weight language models ranging from 135M to 3B parameters on a custom 1,085-example, 16-topic multiple-choice benchmark designed to test symbolic precision, constrained formatting, extraction, and short-horizon semantic decisions under a strict one-letter output protocol. The goal: determine whether small models can be selected, audited, and specialized under real-world hardware and governance constraints.
Results are striking. In base evaluation, Qwen Coder 3B leads at 75.67% strict accuracy, followed by Qwen2.5 1.5B at 67.10%, Qwen3.5 2B at 64.98%, and Granite 3.3 2B at 64.61%. Using a parameter-efficient fine-tuning pipeline with 4-bit NF4 quantization and DoRA/LoRA-style adapters on an NVIDIA L4-class budget (a common cloud GPU), adaptation on a 108-example held-out split delivers dramatic gains: Qwen Coder 3B improves by +26.85 points, SmolLM2 1.7B by +25.92, and even the tiny SmolLM2 135M gains +5.55. The paper concludes that a disciplined workflow of benchmark construction, cross-model evaluation, and low-cost specialization makes a subset of sub-3B models viable as local experts for structured niche workloads, democratizing AI for institutions with limited compute.
- Qwen Coder 3B achieved 75.67% strict accuracy on a new 1,085-example, 16-topic local deployment benchmark.
- Fine-tuning with 4-bit NF4 quantization and DoRA/LoRA adapters improved Qwen Coder 3B by +26.85 points on a held-out split.
- Even a 135M-parameter model gained +5.55 points, proving sub-3B models can be specialized for structured tasks under tight budgets.
Why It Matters
Small, fine-tuned models can replace expensive cloud AIs for specialized tasks, cutting costs and preserving data privacy.