Post-training expert urges AI tinkerers to ditch benchmarks and fine-tune instead

The post challenges the common practice of simply downloading a model and sharing inference benchmarks after acquiring high-end hardware. The author, with 4 years of experience running a 'post-training-as-a-service' business, argues this is intellectually lazy and misses the real opportunity: fine-tuning models for production use cases. They describe real projects—identifying malicious consumer chats, tagging mouse movements for corporate espionage, and profiling sales leads—all done on a single 4090 server with supervised fine-tuning (SFT). The key takeaway is that post-training is a dark art: no tutorials exist, AI tools like Claude can't automate it, and the data mix (synthesis, transformation) is critical. Model choice matters too; Qwen models are crammed with knowledge but hard to tune, while Llama models absorb fine-tuning easily but lack base knowledge. The author emphasizes iteration speed via a low-power, massively-parallel stack (hint in the picture) to quickly find the best model.

Beyond SFT, the next frontier is Reinforcement Fine-Tuning (RFT), described as the 'wild west.' RFT requires a model doing fast inference/rollouts, a reward system (potentially spawning Docker containers to build and test code), and weight updates using PPO/GRPO/RLOO. This needs a specialized build-out that few are doing solo—the author is only starting. Post-training shops like Prime RL run in datacenters. The post ends with a call to action: consider post-training as a more intellectually rewarding and profitable path than just running benchmarks.