Research reveals best open-weight AI models for intent classification
3B instruction-tuned models outperform several 7B base models in zero-shot intent classification
A new arXiv study systematically evaluated 41 open-weight language models across 15 model families, ranging from 135M to 9B parameters, for zero-shot intent classification. The evaluation covered eight English single-label intent-classification datasets, plus a ninth dataset (ATIS) reported as an auxiliary five-shot result. Key findings: instruction-tuned 3B models can outperform several evaluated 7B base models; differences among leading models on MASSIVE are statistically indistinguishable under pairwise McNemar tests; and widely used benchmarks like SNIPS have become saturated and no longer meaningfully discriminate among current open-weight models. The paper also analyzes confidence calibration, robustness to realistic input perturbations, statistical reliability of rankings, deployment efficiency, and benchmark saturation, offering practical guidance for selecting models under compute, latency, and robustness constraints.
- Evaluated 41 open-weight models (135M-9B parameters) across 8 intent-classification datasets
- 3B instruction-tuned models outperformed several 7B base models in zero-shot tasks
- Benchmarks like SNIPS are saturated, making them less useful for model discrimination
Why It Matters
Practical guidance for AI teams to optimize model selection for dialogue systems under real-world constraints