New study finds few-shot prompting benefits vary wildly by model type
Five LLMs tested; some thrive on examples, others degrade. Detailed results inside.
Researchers Ayush Dwivedi and Ashvi Soni systematically investigated how the number of few-shot examples (shots) affects classification performance across model scale, architecture, and output parsing. They tested five LLMs—Gemini Flash Lite, GPT-4o-mini, Llama 3.1 8B, Llama 3.3 70B, and Llama 4 Scout 17B—on the AG News four-class benchmark with shot counts of 0, 1, 2, 3, 5, and 8. Using macro-averaged F1 with 95% bootstrap confidence intervals, permutation tests, and Cohen's d effect sizes, they uncovered four distinct behavioral regimes. Importantly, they identified a systematic parsing artifact that artificially deflated Llama 3.3 70B's performance by up to 206%, leading to a corrected evaluation methodology.
The study found that the relationship between shot count and performance is neither monotonic nor universal. GPT-4o-mini and Gemini Flash Lite showed modest, statistically insignificant gains from additional examples. Llama 3.1 8B exhibited catastrophic zero-shot failure but recovered dramatically with just one example (d=10.98, p<0.0001). Llama 4 Scout degraded monotonically as shots increased. Llama 3.3 70B showed a U-shaped curve: 0-shot F1=0.907, 2-shot F1=0.635, 5-shot F1=0.785 (after parser correction). These results demonstrate that optimal shot count cannot be predicted from model scale alone, and that output parsing robustness is a critical confound in LLM evaluation.
- Four behavioral regimes identified: stable zero-shot, catastrophic zero-shot recovery, monotonic degradation, and U-shaped performance.
- Llama 3.1 8B showed 10.98 Cohen's d effect size from zero- to one-shot, the largest gain observed.
- A parsing artifact deflated Llama 3.3 70B's F1 by 206%, highlighting need for robust output parsing.
Why It Matters
Professionals must tailor few-shot counts per model and beware of parsing bugs skewing performance metrics.