AuroSFT's adapter rollback beats full-model msft with 61.36% accuracy

AuroSFT, introduced by Yue Han and Ziniu Liu, rethinks multi-task supervised fine-tuning (SFT) by replacing full-model rollback with a compact adapter-state mechanism. Standard multi-task SFT, like msft, tracks overfitting via task-wise roll-out, exclusion, and rollback, but materializes scheduler state as full-model checkpoints—costly to store and deploy. AuroSFT instead freezes the pretrained backbone, injects lightweight adapters, and rolls back only adapter weights at task-wise peaks while continuing on the remaining active mixture. This slashes storage overhead and simplifies deployment. At layer level, AuroSFT uses an AuroRA-inspired adaptive nonlinear layer applied to low-rank weight factors, not sample representations, keeping updates linear and exactly mergeable into frozen projections.

Benchmarked under a retained-backbone protocol, AuroSFT achieves 61.36% average accuracy, beating msft's 59.85% reference row and posting higher accuracy on all five tested backbones. The method's compact adapter states make stage transitions cheap, and the exact mergeability means no inference-time overhead. Published on arXiv (2608.05250), the code is available via an anonymous repository. For practitioners juggling heterogeneous task mixtures, AuroSFT offers a practical, parameter-efficient alternative to full-model rollback, enabling faster experimentation, lower storage costs, and stronger generalization across diverse backbones.