How to Fine-Tune a Reasoning Model? A Teacher-Student Cooperation Framework to Synthesize Student-Consistent SFT Data

A research team has introduced TESSY, a novel framework that solves a critical flaw in fine-tuning smaller AI reasoning models. The standard approach—using synthetic data from a stronger 'teacher' model like GPT-OSS-120B to train a 'student' like Qwen3-8B—often backfires, causing performance to drop. The paper identifies a 'stylistic divergence' between the teacher's complex outputs and the student's simpler distribution as the culprit. TESSY bridges this gap by having the teacher and student models collaborate, alternately generating tokens to create training data that retains the teacher's advanced reasoning but matches the student's style.

In code generation experiments, the results were stark. Fine-tuning Qwen3-8B on standard teacher-generated data led to significant performance regressions: a 3.25% drop on LiveCodeBench-Pro and a 10.02% drop on OJBench. In contrast, using TESSY's collaboratively synthesized data flipped these losses into substantial gains, achieving improvements of 11.25% and 6.68% on the same benchmarks. This demonstrates that data quality and stylistic alignment are more important than raw data quantity for effective supervised fine-tuning (SFT).

The TESSY framework represents a significant shift in how we think about knowledge distillation for reasoning models. It moves beyond simple imitation learning to a cooperative synthesis process, ensuring the training data is pedagogically appropriate for the student model's current capabilities. This method could unlock more efficient fine-tuning for a wide range of specialized, smaller models, making advanced reasoning more accessible without the performance penalty previously seen.