After building 10+ production AI systems - the honest fine-tuning vs prompt engineering framework (with real thresholds)

An AI engineer with experience building over 10 production systems has shared a viral, practical framework for deciding between prompt engineering and fine-tuning, cutting through common tutorial advice. The framework provides clear, data-driven thresholds: prompt engineering is optimal for general-purpose tasks (like support or summarization), when training data changes frequently, when you have fewer than ~500 high-quality labeled examples, or when you need to ship fast and iterate based on real usage. Crucially, the author emphasizes that you should not fine-tune until you have measured your specific failure modes in production.

Fine-tuning becomes the right choice when you need absolute consistency in format or tone, when dealing with highly specialized domains (like regulatory or clinical documents), when you're hitting about 500,000 API calls per month and want to distill behavior into a cheaper model, or when you have a hard latency constraint. Most importantly, you should have at least 1,000 trusted, high-quality labeled examples derived from real production data, not synthetic generation. The author warns against the common mistake of deciding to fine-tune early with synthetic data based on assumptions, which almost always creates a model optimized for the wrong problems.

The recommended process is to always start with prompt engineering, ship it, and collect real failure cases. Only then should you identify the specific failing pattern and fine-tune on that mode using actual production data. A concrete example illustrates the impact: a client saved $18,000 per month by waiting three months for production data before fine-tuning GPT-3.5 for a classification task, achieving the same accuracy as GPT-4 at one-eighth the cost. Rushing to fine-tune with synthetic data would have created technical debt and a less effective model.